Arena's Ranking Method

Today, we are announcing an update to our ranking methodology that makes our rankings both more interpretable and more accurate in how they reflect statistical uncertainty. We are grateful to our community members for helpful suggestions that have led to this decision.

Alongside each model score, we are now reporting the raw rank and the rank spread for the respective model.

The raw rank is simply the rank of the model's Arena score. There are no ties in the rank column: each model is assigned a unique rank. Models in the leaderboard are sorted based on the raw rank.

Ties are reflected in the rank spread, which is an interval whose lower and upper endpoints correspond to the best and worst rank a model could have, based on all models' confidence intervals (CIs). The scores and confidence intervals remain unchanged. Formally, the best rank a model M can have is 1 + #{models whose lower CI endpoint is greater than model M's upper CI endpoint} . Analogously, the worst rank a model M can have is 1 + #{models whose upper CI endpoint is greater than model M's lower CI endpoint} . (The "1+" ensures that ranks cannot be zero; the best possible rank is one.)

Intuitively, one can interpret the rank spread as follows. Suppose the true Arena score of model M is equal to the highest value included in its score's CI, and suppose that the true scores of all other models are the lowest values included in their respective CIs. This "optimistic" rank is the lower endpoint of model M's rank spread. The "pessimistic" rank is obtained analogously, by assuming model M's score is the lowest value included in its CI and all other models' scores are the highest values in their respective CIs. This is the upper endpoint of model M's rank spread.

Therefore, two models are tied if their rank spreads overlap. The raw ranking is the current best estimate of the true underlying ranking.