After analyzing over 1.7+ million user sessions in Agent Mode, we identified the most prevalent categories of tasks users delegate to agents and measured the real cost of completing those tasks. Today, we’re introducing two complementary features to Agent Arena to help users gain a more nuanced understanding of how models stack up:
- Agent Costs: See what each model costs to complete a task, along with a Pareto Frontier view of the most cost-efficient models at every level of performance
- Agent Categories: Filter the leaderboard to understand how models rank in three categories: Code, Chat, and Work
Together, these transform the Agent Arena from a single ranking to a more personalized guide that helps you identify the right model at the right price point for your use case. Let’s take a deeper dive into each:
What a Task Costs
Representing costs in Agent Arena requires nuance because people use agents to do tasks. We opted for price per task as our primary measure for agent costs to represent how agents are used. We measure this using tasks completed by users in real Agent Mode sessions. With this update, we are also adding output tokens per task and cost per million tokens to the Agent leaderboard.
Task Boundaries
To measure cost fairly, we need a common unit of work that is comparable across models. We chose an individual task as the primary unit for costs. We segmented Agent Mode sessions into discrete task boundaries in order to compute the cost for each task.
For example: a single task may be a user asking an agent to rearrange a piece of sheet music for the violin. The task lasts four turns: the user goes back and forth with the agent on specifics before the agent produces a final arrangement, at which point the task is done and the user moves on to another task.
This task segmentation also allows us to measure the number of output tokens required per task, which helps measure the token-efficiency of a model, as models often differ in how many tokens they use to do a given task.
Measuring Task Cost
With a “task” boundary defined, we move to the actual cost calculation. A simple average of cost across all tasks in a session makes models that have been sampled in long-running sessions look expensive. This is because long-running sessions can skew costs higher through context carry-over. Longer sessions accumulate more context that is processed each turn, and as a result, later tasks in a session cost more regardless of the cost-efficiency of the model.
