Arena Leaderboard Policy

1. If it is determined that the publicly released model differs from the pre-release version tested on Arena, Arena will remove the model from the leaderboard until the model can be re-evaluated under the requirements of this policy.

Evaluating publicly released models. Evaluating a public model consists of the following steps:

1. Add the model to Arena for testing and let the community know it was added. The model provider may choose a system prompt for configuration.
2. Accumulate enough votes until the model’s rating stabilizes (at least 1,000; typically more).
3. After the rating stabilizes, list the model on the leaderboard. If the votes were collected while the model was unreleased (see “Evaluating unreleased models” section), we will mark the model score as preliminary until enough fresh votes have been collected after the model’s public release.
4. The API of the launched model must be accessible for a minimum of 30 days post launch, or it will be removed from the leaderboard.

Note: Once testing of a public model begins, we are unable to pause or terminate the process. Testing will continue uninterrupted and the score will be released upon completion.

Evaluating unreleased models. We collaborate with model providers to bring their unreleased models to our community for preview testing.

Model providers can test an unreleased model with the model’s name anonymized. A model is considered “unreleased” if its weights are neither open nor available via a public API or service. Evaluating an unreleased model consists of the following steps:

1. Add the model to Arena with an anonymous label. Each anonymous model has its own unique label.
2. Keep testing the model until we accumulate enough votes for its rating to stabilize (at least 1,000; typically more) or until the model provider withdraws it.
3. Share the results privately with the model provider, once we accumulate enough votes.
4. Remove the model from Arena.

If a model is tested anonymously and is subsequently released publicly, we mark its score as preliminary until enough fresh votes have been collected after the model’s public release (see “Evaluating publicly released models”). Model providers are all allowed to test multiple variants of their models before making it public, subject to our system’s constraints.

Sampling policy. The policy with which we sample model pairs in a battle is based on several principles:

1. In every battle, at least one of the models is a publicly available model. At least 20% of battles will be between publicly available models only.
2. We reserve the right to deprecate models. This may happen, for example, because a model is no longer publicly accessible, there is a more recent model in the same series (e.g., gpt-4o-0513 vs gpt-4o-0806), or multiple model providers offer cheaper and strictly better models according to the overall Arena score. To ensure transparency, all models that have been retired from battle mode are recorded in a public list.
3. A publicly available model’s probability of being sampled is typically uniform, with adjustments for leaderboard integrity and user experience (such as increased sampling for new and/or leading public models). This is to ensure the best community experience as well as accurate evaluation for all public models. The regression for computing Arena scores uses reweighting, such that no matter how the sampling probabilities are set, the Arena scores remain unbiased.

Sharing data. We periodically share portions of our data with the community to support research and transparency. When we test unreleased models, we share conversation data with model providers to help them improve their models (see “Evaluating unreleased models”). Before sharing any data, we use tools (e.g. GCP’s Sensitive Data Protection API service) to remove personal and sensitive data.