Chatbot Arena - New models & Elo system update

Since OpenAI’s GPT-4 update in June, the community has been wondering whether there’s a performance change on the newer version of GPT-4. Some people find performance drop in certain domains (reference), but it’s still unclear what’s really going on. Previously we combined votes of the two versions into just GPT-4. As we transition from online Elo to the BT model (explained later in the post), we decide to separate out different versions of proprietary model APIs to better satisfy its assumptions on model staying static.

Surprisingly, we observe a significant difference between gpt-4-0314 and gpt-4-0613 (Rating 1201 vs 1152) based on Arena user preference. The GPT-4 API was automatically updated from 0314 to 0613 on June 27 and the 0314 version has since then been retired from Arena. Potential hypotheses:

1. Arena user distribution has shifted before/after July (e.g., prompt distribution, voting behaviors etc)
2. No comparison data for 0314 against newly added models after July may be unfair.
3. Arena users indeed prefer the 0314 version of GPT-4 than 0613.

To address this problem, we have brought up gpt-4-0314 online again to collect new votes, also directly comparing it against its newer 0613 version. At the time of writing we have collected 1,000 new votes for gpt-4-0314 and its performance is still robust from winrate over other models shown below. We’ll give more updates on this in the future.

Interestingly, gpt-3.5-turbo, which has been through a similar version change (0314 -> 0613), seems to be normal. As you can see, gpt-3.5-turbo-0613 has slightly higher rating than gpt-3.5-turbo-0314 (1112 vs 1106). However, we again observe a strange performance drop of the latest version gpt-3.5-turbo-1106 which has obtained over 5,000 votes. We hope to investigate this deeper by developing new tools to analyze user prompts and identify model strengths and weaknesses in different areas.

We adopted the Elo rating system for ranking models since the launch of the Arena. It has been useful to transform pairwise human preference to Elo ratings that serve as a predictor of winrate between models. Specifically, if player A has a rating of RARA​ and player B a rating of RBRB​, the probability of player A winning is

ELO rating has been used to rank chess players by the international community for over 60 years. Standard Elo rating systems assume a player’s performance changes overtime. So an online algorithm is needed to capture such dynamics, meaning recent games should weigh more than older games. Specifically, after each game, a player’s rating is updated according to the difference between predicted outcome and actual outcome.

This algorithm has two distinct features:

1. It can be computed asynchronously by players around the world.
2. It allows for players performance to change dynamically – it does not assume a fixed unknown value for the players rating.

This ability to adapt is determined by the parameter K which controls the magnitude of rating changes that can affect the overall result. A larger K essentially put more weight on the recent games, which may make sense for new players whose performance improves quickly. However as players become more senior and their performance “converges” then a smaller value of K is more appropriate. As a result, USCF adopted K based on the number of games and tournaments completed by the player (reference). That is, the Elo rating of a senior player changes slower than a new player.

When we launched the Arena, we noticed considerable variability in the ratings using the classic online algorithm. We tried to tune the K to be sufficiently stable while also allowing new models to move up quickly in the leaderboard. We ultimately decided to adopt a bootstrap-like technique to shuffle the data and sample Elo scores from 1000 permutations of the online plays. You can find the details in this notebook. This provided consistent stable scores and allowed us to incorporate new models quickly. This is also observed in a recent work by Cohere. However, we used the same samples to estimate confidence intervals which were therefore too wide (effectively CI’s for the original online Elo estimates).

In the context of LLM ranking, there are two important differences from the classic Elo chess ranking system. First, we have access to the entire history of all games for all models and so we don’t need a decentralized algorithm. Second, most models are static (we have access to the weights) and so we don’t expect their performance to change. However, it is worth noting that the hosted proprietary models may not be static and their behavior can change without notice. We try our best to pin specific model API versions if possible.

To improve the quality of our rankings and their confidence estimates, we are adopting another widely used rating system called the Bradley–Terry (BT) model. This model actually is the maximum likelihood (MLE) estimate of the underlying Elo model assuming a fixed but unknown pairwise win-rate. Similar to Elo rating, BT model is also based on pairwise comparison to derive ratings of players to estimate win rate between each other. The core difference between BT model vs the online Elo system is the assumption that player’s performance does not change (i.e., game order does not matter) and the computation takes place in a centralized fashion.

With the static performance assumption, the model ratings can be obtained by maximum likelihood estimation (MLE), i.e. maximizing the likelihood of the observed game outcomes given the model ratings. Code snippet below shows how to use MLE to compute the model ratings.

Similarly, we can also bootstrap the MLE Bradley-Terry scores to obtain the confidence intervals of model ratings. We observe that the mean rating by both methods are very similar and the rankings are almost the same.

More importantly, with the BT model, the bootstrap confidence intervals now better capture the variance of the model performance estimates. We observe clear improvement in the below figures. Newly added models with fewer votes have a wider range of confidence intervals than others.

Note that we extend BT model to consider ties by counting a tie as half a win and half a loss. Code to reproduce the calculation can be found at this notebook.

We’ve also conducted topic modeling on 50,000 user prompts to better understand how users interact with these models. Our approach utilized OpenAI embeddings text-embedding-ada-002 and K-means clustering, followed by GPT-4 to summarize the topics for each cluster, provided with the prompts close to the center. This analysis revealed a wide range of topics, from role-playing, story writing to programming advice. We show the topic distribution and a few examples below.

| Cluster ID | Arena User Prompt | | ---------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 1 | You are a Chief information Officer for a Biotechnology Manufacturing company and will act like one. Write a business need and objectives for a case study to Engage Info-Tech technical consulting services to conduct a comprehensive assessment of our current application development practices, including analyzing our development methodologies, tools, and frameworks. | | 2 | Write a short scene from a novel where a beautiful, wicked lamia coils around an unfortunate, quippy human adventurer. | | 3 | How should the balance be struck between freedom of speech and the ability to function in a world without continual distractions and distortions from misinformation? | | 4 | Can you give me a list of 5 suggestions on how to write software with fewer bugs? |

Moving forward, we aim to refine our methods to filter out low-quality prompts and improve categorization for a clearer understanding of model strengths and weaknesses in different areas.

We plan to ship real-time leaderboard update, diving deeper into user prompt analysis, and enhancing prompt moderation and categorization. Stay tuned for more insights as we continue to refine our approach to evaluating the evolving landscape of LLMs. Thanks for supporting us on this journey, and we look forward to sharing more updates soon!

• Chatbot Arena Demo
• Arena Elo Colab
• How Is ChatGPT’s Behavior Changing over Time?
• Bradley-Terry model lecture note, paper
• Elo Uncovered: Robustness and Best Practices in Language Model Evaluation

If you wish to see more models on Arena leaderboard, we invite you to contribute to FastChat or contact us to provide us with API access.