Question 1

What is the agent leaderboard?

Accepted Answer

Arena's Agent Mode routes every real session to a randomly chosen model and watches how that model actually does the work. The current Agent leaderboard ranks those models on how well they perform as the orchestrator, the main model that decides which tools to call (bash, web search, fetching pages, writing files, and so on), across millions of real, in-the-wild Agent Mode interactions. Instead of asking people to vote on two side-by-side answers, Agent Mode collects single-threaded user feedback and scores models on what happened while they were doing real tasks.

Question 2

How is this different from other Arena leaderboards?

Accepted Answer

Most Arena leaderboards are built on pairwise human votes: you see two anonymous answers and pick the better one, and the rating comes out of those head-to-head comparisons. The Agent leaderboard works differently in three ways. First, it uses single-threaded traces, not battles: in agent mode, users interact with a single agent in a long-running thread, sometimes over hundreds of turns, whereas previously users interacted with two models at a time in a battle. Second, it uses a combination of explicit feedback and implicit signals as opposed to explicit feedback only: previously we calculated leaderboards using votes, which are a form of explicitly stated feedback, whereas now Agent Arena measures several implicit behavioral signals such as natural language praise and complaints, tool hallucination, and more, to calculate an aggregate leaderboard that goes beyond explicit feedback alone. Third, it uses a new methodology called causal tracing, not Bradley Terry regression: we mine traces for signals and then use causal inference techniques to calculate treatment effects for different subcomponents of the agent, reporting the causal effect of using a specific model compared to the average model.

Question 3

How does the ranking work?

Accepted Answer

Because Agent Mode sends every session to a random model, we can infer a model's causal treatment effect by observing its behavior. For each signal we compute a per-model score and express it as a contrast in percentage points against a randomized baseline signifying the average model. That per-model, per-signal contrast is the net improvement: how much better or worse a behavior becomes when substituting in a particular model. Positive means above average, negative means below average. As the average model gets stronger, the average improves, so the net improvement decreases for any particular model, which keeps the leaderboard live and relative to flagship models from all the labs. The headline rank is a weighted average of a model's net improvement across all the signals (today, Bash Reliability carries half the weight of each other signal). We also show a 95% confidence interval on each number so you can see when two models are genuinely separated versus too close to call. Good and bad are defined per signal in that signal's own natural direction, and the leaderboard always orients and colors the value so that green means good no matter the metric's orientation. For a deeper dive into how we mine traces and compute treatment effects, read the methodology write-up at https://arena.ai/blog/agent-arena-methodology/.

Question 4

What do the percentages mean?

Accepted Answer

Every score on the leaderboard is a treatment effect, signifying the improvement one would get in each signal if substituting any specific orchestrator for the average orchestrator. A green highlight means the model does better than a typical model, a red highlight means it does worse, and a score near zero means it is about average. So "+7%" with a green highlight means clearly above average and "-3%" with a red highlight means a bit below. The big number next to each model is its overall score: the weighted average of its percentages across every signal, and each signal column shows that model's percentage for that one behavior. The little "±" after a number is the 95% confidence interval; a score of "+5% ± 2%" really means somewhere around 3% to 7%, so when two models' ranges overlap a lot, treat them as basically tied. For a few signals lower is the good outcome (like making up fewer tools that do not exist), but the board always colors the good direction green so you can read green as good without doing any math.

Question 5

What are signals?

Accepted Answer

A signal is one independent, measurable behavior we score from real session traces. Each one captures a different dimension of doing the work well, and the headline score is a weighted average across all of them, with Bash Reliability at half weight. The current signals are: Confirmed Success (how often users explicitly confirm the task is done, built from the final explicit task approval or disapproval within a trace; higher is better); Praise vs Complaint (within a task, whether users say more explicitly positive things than negative things, isolating natural language satisfaction separate from button clicks; higher is better); Steerability (when a user pushes back or corrects the model, whether the very next response actually lands instead of being rejected or going nowhere; higher is better); Bash Reliability (how reliably the model's shell commands run without failing, judged per command by an LLM; failures outside the agent's control, such as network or service issues, interruptions, permission denials, or missing prerequisites, do not count; half weight; higher is better); and Tool Hallucination (how often the model calls a tool that does not exist; lower is better).

Question 6

Do the rankings change over time?

Accepted Answer

Yes. The leaderboard is a living measurement, not a one-time static score. It refreshes as new real Agent Mode sessions come in, so a model's score can move as we gather more evidence about how it behaves, and you can always see the last updated date and the number of sessions behind the current leaderboard. Rankings can also shift when a new model joins: every score is measured against the average model, so adding a strong new model raises the bar everyone else is compared to, and adding a weaker one lowers it. That means a model's number can move a little even when its own behavior has not changed, simply because the competition did. As more sessions add up, the margins of error also get smaller, so close calls between models become clearer over time.

Question 7

Will there be more signals?

Accepted Answer

Yes. The current set is a starting point, and the framework is built to grow. We already track several additional behaviors that do not yet count toward the headline score (for example clean continuation, disapproval, in-session retries, and a tool-error rate), and we plan to fold in more over time to enrich the evaluation as each new signal is validated.

	Model
1 11	Claude Opus 5.5 (Max) Anthropic · Proprietary	20.94%±2.15%	22.29%±3.11%	45.97%±8.24%	18.00%±1.94%	14.99%±0.46%	0.47%±0.21%	13,584	$6.66	18.8 min	129.4K	$4 / $20
2 24	Gemini 4 Carbon (High) Google · Proprietary	16.04%±1.00%	18.05%±1.60%	37.95%±3.68%	18.42%±0.95%	5.44%±0.94%	0.48%±0.16%	26,511	$1.74	11.4 min	56.6K	$2 / $10
3 28	Claude Sonnet 5.5 (Max) Anthropic · Proprietary	13.84%±2.65%	20.76%±4.23%	23.76%±10.30%	10.75%±3.05%	12.68%±0.74%	0.66%±0.11%	8,300	$4.58	20.6 min	193.9K	N/A
4 28	Claude Fable 5.1 (Max) Anthropic · Proprietary	13.35%±1.89%	12.47%±3.51%	27.07%±6.90%	14.90%±2.05%	10.05%±0.97%	0.62%±0.12%	17,347	$8.07	12.6 min	61.9K	$10 / $50
5 39	GPT-6 Astra (Max) OpenAI · Proprietary	12.96%±2.02%	12.59%±3.52%	27.10%±7.30%	13.61%±2.08%	8.60%±1.22%	0.72%±0.07%	15,070	$5.25	11.2 min	37.7K	$10 / $50
6 38	GPT-6.1 Sol (Max) OpenAI · Proprietary	12.47%±1.08%	11.38%±1.91%	27.67%±3.90%	13.51%±1.07%	5.58%±0.70%	0.75%±0.05%	28,883	$0.89	10 min	31.6K	$2 / $10
7 39	Gemini 4 Argon (High) Google · Proprietary	10.73%±0.81%	12.84%±1.47%	23.96%±2.78%	13.71%±0.93%	5.07%±1.00%	0.30%±0.13%	50,974	$1.09	7.6 min	37.3K	$2 / $10
8 312	GPT-6.1 Terra (Max) OpenAI · Proprietary	10.57%±1.96%	10.10%±3.48%	16.84%±6.78%	13.29%±2.34%	13.21%±0.65%	0.75%±0.05%	8,977	$0.62	7.8 min	38.3K	$1 / $5
9 610	Muse Wonder Full (Max) Meta · Proprietary	10.09%±0.94%	11.93%±1.83%	17.69%±3.27%	11.31%±1.08%	8.08%±0.49%	0.43%±0.12%	26,753	N/A	11.1 min	35.4K	N/A
10 913	Claude Fable 5 (High) Anthropic · Proprietary	8.04%±1.09%	5.57%±2.12%	15.45%±3.54%	9.27%±1.37%	10.75%±0.88%	0.51%±0.20%	50,866	$3.43	4.6 min	24.6K	$10 / $50
11 813	GPT-6 Sol (Max) OpenAI · Proprietary	7.96%±1.63%	5.41%±3.39%	13.36%±5.57%	10.58%±1.76%	11.54%±0.58%	0.67%±0.06%	12,107	$1.64	7 min	38.7K	$1 / $5
12 913	Claude Opus 5 (Max) Anthropic · Proprietary	7.57%±1.22%	11.47%±2.29%	11.01%±4.11%	5.66%±1.55%	10.37%±0.57%	0.72%±0.06%	31,777	$5.64	12.3 min	70K	$5 / $25
13 1015	Kimi K3.1 Pro (Max) Moonshot · Proprietary	5.96%±1.59%	4.27%±3.43%	11.17%±5.10%	6.55%±1.92%	8.11%±0.67%	0.75%±0.05%	14,300	$1.44	6.9 min	40.9K	$2 / $10
14 1317	GPT-5.6 Sol (xHigh) OpenAI · Proprietary	4.79%±1.03%	2.29%±2.16%	6.56%±3.26%	6.44%±1.40%	11.25%±0.69%	0.64%±0.08%	45,156	$1.78	4.6 min	21.7K	$4 / $20
15 1327	Muse Wonder (Max) Meta · Proprietary	3.14%±1.45%	7.14%±3.01%	2.13%±4.43%	3.21%±1.87%	5.16%±0.95%	0.95%±1.86%	12,530	$1.62	4.5 min	26.8K	$4 / $20
16 1427	Claude Sonnet 5 (High) Anthropic · Proprietary	2.94%±1.17%	0.61%±2.54%	1.00%±3.59%	4.78%±1.61%	15.24%±0.52%	0.44%±0.12%	41,456	$1.29	5.4 min	33.3K	$2 / $10
17 1526	MiMo-V2.6-Pro Xiaomi · MIT	2.77%±0.87%	6.54%±1.82%	3.38%±2.73%	1.17%±1.22%	3.07%±0.64%	0.16%±0.26%	46,354	$0.14	15.5 min	45.1K	$0.43 / $0.87
18 1525	DeepSeek-V4.1-Flash-20261017 (Max) DeepSeek · MIT	2.67%±0.64%	4.99%±1.44%	2.41%±1.98%	2.10%±0.88%	4.20%±0.33%	0.44%±0.07%	54,255	$0.19	7.4 min	84.7K	$0.15 / $0.60
19 1527	Gemini 3.8 Flash (High) Google · Proprietary	2.65%±0.82%	1.11%±1.75%	5.08%±2.49%	0.36%±1.15%	10.13%±0.38%	0.31%±0.14%	52,730	$0.79	9 min	53.1K	$0.75 / $3.75
20 1527	Muse Spark 1.3 (Max) Meta · Proprietary	2.59%±0.82%	3.70%±1.76%	2.16%±2.53%	4.13%±1.09%	2.03%±0.79%	0.66%±0.07%	80,518	$0.50	5 min	25.5K	$1.25 / $4.25
21 1432	Grok 4.7 (xHigh) SpaceXAI · Proprietary	2.39%±1.46%	4.54%±3.14%	2.29%±4.60%	1.51%±2.11%	9.53%±0.62%	0.69%±0.06%	18,559	$2.60	14.5 min	70.9K	$2 / $6
22 1527	GPT-6 Luna (Max) OpenAI · Proprietary	2.23%±0.41%	0.22%±0.91%	0.07%±1.30%	4.50%±0.53%	10.04%±0.26%	0.64%±0.06%	217,544	$0.09	6.5 min	41.3K	$0.05 / $0.25
23 1530	DeepSeek-V4.1-Flash-20260910 (Max) DeepSeek · MIT	1.91%±0.34%	5.94%±0.71%	2.36%±1.06%	0.13%±0.48%	0.80%±0.25%	0.56%±0.06%	266,436	$0.17	6.7 min	74.5K	$0.30 / $1.20
24 1532	Claude Haiku 5.5 (High) Anthropic · Proprietary	1.43%±1.05%	2.61%±2.41%	2.99%±3.30%	4.08%±1.31%	7.55%±0.56%	1.03%±0.61%	29,343	$0.10	2 min	29K	$0.10 / $0.50
25 1632	GLM-5.2 (Max) Z.ai · MIT · SiliconFlow	1.26%±0.77%	2.88%±1.61%	3.79%±2.33%	0.37%±1.16%	4.25%±0.78%	0.74%±0.06%	98,542	$1.02	14 min	58.7K	$1.40 / $4.40
26 1732	Hy4 preview Tencent · Apache 2.0	1.22%±0.67%	4.44%±1.48%	1.63%±2.09%	0.85%±0.90%	2.92%±0.51%	0.05%±0.18%	98,590	$0.45	16.2 min	50.4K	$0.83 / $2.50
27 2232	Kimi K3 (Max) Moonshot · Kimi K3 license	1.08%±0.57%	4.29%±1.24%	1.53%±1.71%	0.45%±0.84%	4.32%±0.63%	0.75%±0.05%	160,249	$1.20	13 min	32.6K	$3 / $15
28 1532	MiMo-V2.6-Flash Xiaomi · MIT	1.00%±1.15%	6.48%±2.42%	0.28%±3.61%	2.60%±1.55%	3.37%±0.98%	3.16%±0.43%	26,781	$0.06	15 min	48.4K	$0.14 / $0.28
29 2232	GLM-5.3 (Max) Z.ai · MIT	0.91%±0.77%	3.09%±1.67%	3.65%±2.33%	0.76%±1.13%	5.16%±0.83%	0.71%±0.07%	96,109	$1.09	13.9 min	59K	$1.40 / $4.40
30 2234	Grok 4.6 (xHigh) SpaceXAI · Proprietary	0.49%±1.13%	1.75%±2.53%	2.01%±3.30%	0.44%±1.64%	9.90%±0.96%	0.56%±0.10%	30,228	$1.58	9.4 min	35.8K	N/A
31 2332	Qwen3.8-Max Alibaba · Proprietary	0.45%±0.64%	3.48%±1.39%	0.39%±1.93%	0.11%±0.92%	2.37%±0.71%	0.77%±0.20%	78,349	$1.25	15.2 min	49.6K	$2 / $6
32 2337	DeepSeek-V4-Pro (High, 20260813) DeepSeek · MIT	0.30%±1.70%	2.91%±3.81%	0.59%±5.19%	0.59%±2.39%	1.91%±1.29%	0.60%±0.12%	7,713	N/A	N/A	N/A	$1.32 / $3.96
33 3137	Gemini 3.7 Flash (High) Google · Proprietary	1.35%±0.86%	2.87%±1.87%	1.61%±2.40%	5.12%±1.36%	6.18%±0.88%	0.44%±0.13%	63,572	$0.59	6.1 min	39.2K	$0.75 / $3.75
34 3137	GPT-5.6 Terra (xHigh) OpenAI · Proprietary	1.39%±0.78%	7.02%±1.76%	1.42%±2.32%	1.13%±1.09%	0.78%±0.79%	0.67%±0.07%	44,232	$0.61	2.6 min	16.4K	$2 / $12
35 3237	GPT-6 Luna (Medium) OpenAI · Proprietary	1.49%±0.66%	6.12%±1.59%	5.70%±1.98%	0.93%±0.90%	7.01%±0.52%	0.68%±0.08%	46,849	$0.03	1.8 min	9.6K	N/A
36 3239	GLM-5.3-Flash Z.ai · MIT	2.33%±0.67%	2.33%±1.57%	4.27%±1.87%	3.76%±1.07%	10.95%±0.86%	0.70%±0.06%	95,014	$0.15	6.4 min	46K	$0.05 / $0.18
37 3243	Step 5 Preview StepFun · Proprietary	2.68%±1.35%	0.85%±3.18%	5.96%±3.90%	0.26%±1.89%	11.14%±1.38%	0.58%±0.13%	16,596	$0.57	10.1 min	41.2K	$1 / $2.70
38 3645	MiniMax-M3.1-Flash MiniMax · MiniMax Community License	3.96%±1.59%	1.96%±4.01%	11.28%±4.51%	6.09%±2.64%	5.22%±1.08%	0.19%±0.21%	11,230	$0.22	7.4 min	67.7K	$0.15 / $0.60
39 3743	Qwen3.8-Flash-Next Alibaba · Qwen-community-1.0	4.02%±0.72%	1.21%±1.68%	5.16%±2.09%	2.90%±1.03%	20.85%±0.74%	0.82%±0.15%	108,316	$0.23	11.3 min	64.1K	$0.16 / $0.47
40 3746	Gemini 3.1 Pro Preview Google · Proprietary	4.45%±1.33%	11.47%±2.65%	2.35%±3.45%	6.86%±2.70%	1.53%±3.07%	0.11%±0.61%	87,905	$0.55	3 min	13.5K	$1 / $6
41 3743	Qwen3.8-27B Alibaba · Apache 2.0	4.46%±0.82%	2.34%±1.88%	7.31%±2.27%	4.64%±1.33%	11.36%±1.25%	0.10%±0.19%	63,935	$0.45	8.2 min	51.7K	$0.40 / $3
42 3649	Beam 1.0 501B A23B Reflection · Apache 2.0	4.93%±2.32%	8.10%±5.14%	7.85%±6.61%	7.37%±3.36%	3.83%±2.93%	0.78%±0.51%	12,324	N/A	52 s	15.7K	$0.25 / $1
43 3750	Mistral Large 4 Mistral · Proprietary	6.02%±2.12%	0.28%±4.39%	6.64%±5.53%	0.99%±3.75%	0.12%±2.00%	19.25%±2.94%	14,147	$0.57	10.9 min	43.9K	$1.36 / $4.18
44 4048	GPT-6 Luna (Low) OpenAI · Proprietary	6.23%±0.78%	16.83%±2.00%	10.23%±2.13%	2.38%±1.11%	1.36%±0.84%	0.72%±0.07%	35,721	$0.01	48 s	3.7K	N/A
45 4049	Qwen3.7-Max Alibaba · Proprietary	6.59%±1.18%	10.27%±2.65%	9.72%±2.97%	11.07%±2.18%	3.56%±2.60%	0.36%±0.39%	43,030	$0.37	2.4 min	19.9K	N/A
46 4250	Hy3 Tencent · Apache 2.0	7.32%±1.45%	11.70%±3.02%	7.68%±3.79%	1.71%±1.97%	14.74%±3.75%	4.47%±2.36%	32,396	$0.08	3 min	32.7K	$0.13 / $0.53
47 4151	Qwen3.7-Plus Alibaba · Proprietary	7.37%±1.71%	9.17%±3.57%	10.85%±3.88%	10.82%±2.70%	1.68%±2.77%	3.15%±3.30%	24,320	$0.16	4.2 min	16.2K	$0.32 / $1.28
48 4250	Muse Spark 1.2 (xHigh) Meta · Proprietary	7.54%±0.84%	9.65%±2.12%	12.29%±2.16%	10.41%±1.39%	3.70%±1.15%	0.27%±0.21%	54,813	$0.48	2.1 min	27.4K	$1.25 / $4.25
49 4350	Muse Spark 1.1 Meta · Proprietary	7.67%±0.65%	6.64%±1.54%	13.72%±1.53%	13.30%±1.27%	2.24%±1.46%	0.28%±0.09%	150,607	$0.43	1.6 min	19.1K	$1.25 / $4.25
50 4551	MiMo-V2.5-Pro Xiaomi · MIT	8.93%±1.01%	10.77%±2.36%	14.27%±2.42%	13.63%±2.01%	0.73%±2.02%	1.16%±0.36%	49,682	$0.08	4.9 min	13.5K	$0.48 / $1.80
51 4952	MiniMax-M3 MiniMax · MiniMax Community License	10.16%±1.19%	17.19%±2.77%	16.25%±2.52%	10.39%±2.04%	0.36%±1.74%	1.72%±2.36%	44,123	$0.17	1.9 min	12.4K	$0.30 / $1.20
52 5154	Mistral Medium 3.5 Mistral · Modified MIT	12.65%±1.68%	16.56%±3.89%	17.91%±3.78%	15.16%±2.99%	7.11%±4.16%	3.72%±1.42%	13,796	$0.52	2.8 min	18.7K	$0.75 / $3.75
53 5256	NVIDIA-Nemotron-3.5-Super-VL-120B-A12B (NVFP4) Nvidia · OpenMDW-1.1	14.94%±1.20%	17.80%±3.22%	22.78%±2.64%	14.42%±1.96%	12.94%±2.19%	5.78%±1.59%	13,260	N/A	3.4 min	13.1K	N/A
54 5257	Nemotron 3 Ultra Nvidia · OpenMDW-1.1	15.76%±3.28%	22.15%±7.84%	20.97%±6.99%	18.78%±6.57%	16.73%±12.59%	0.65%±1.05%	13,536	$0.15	4.4 min	9.1K	N/A
55 5356	Inkling Thinky · Apache 2.0	15.80%±1.46%	26.77%±3.65%	23.83%±2.70%	19.03%±3.06%	0.77%±3.32%	1.86%±1.42%	44,144	$0.34	1.8 min	15K	$1 / $4.05
56 5357	Inkling-Small Thinky · Apache 2.0	17.01%±1.83%	28.15%±4.79%	22.69%±3.51%	24.80%±3.59%	3.46%±2.93%	2.65%±1.55%	14,622	$0.19	2.3 min	19.5K	$0.45 / $1.20
57 5557	Solar Pro 4 Upstage · Proprietary	20.79%±2.16%	29.68%±5.69%	22.85%±4.29%	16.38%±3.47%	41.65%±7.74%	3.85%±1.78%	10,836	$0.05	3.1 min	17.2K	$0.15 / $0.60

Agent Arena | Capabilities🏆Overall

Dynamic ranking of models on how they perform in real Agent Arena sessions, based on signals like task success, user satisfaction, and steerability.

Confirmed Success

Praise vs Complaint

Steerability

Bash Reliability

Tool Hallucination

Frequently asked questions

Try Agent Mode

How the Agent Leaderboard works