Chatbot Arena Leaderboard Updates (Week 2)

In this update, we have added 4 new yet strong players into the Arena, including three proprietary models and one open-source model. They are:

• OpenAI GPT-4
• OpenAI GPT-3.5-turbo
• Anthropic Claude-v1
• RWKV-4-Raven-14B

Table 1 displays the Elo ratings of all 13 models, which are based on the 13K voting data and calculations shared in this notebook. You can also try the voting demo.

If you want to see more models, please help us add them or contact us by giving us API access.

Thanks to the community’s help, we have gathered 13k anonymous votes. Looking at the rankings and data collected from this leaderboard update, we have a few interesting findings.

Gaps between proprietary and open-source models
We do observe a substantial gap between the three proprietary models and all other open-source models. In particular, GPT-4 is leading the board, achieving an Elo score of 1274. It is almost 200 scores higher than the best open-source alternative on this board – our Vicuna-13B. After dropping ties, GPT-4 wins 82% of the matches when it is against Vicuna-13B, and it even wins 79% of the matches when it is against its previous generation GPT-3.5-turbo.

However, it is important to note that these open-source models on the leaderboard generally have fewer parameters, in the range of 3B - 14B, than proprietary models. In fact, recent advancements in LLMs and data curation have allowed for significant improvements in performance with smaller models. Google’s latest PaLM 2 is a great example of this: knowing that PaLM 2 achieves even better performance than its previous generation using smaller model sizes, we remain very optimistic about the potential for open-source language models to catch up. Through our and this leaderboard effort, we hope to contribute a trusted evaluation platform for evaluating LLMs, and help advance this field and create better language models for everyone.However, among the three proprietary models, we do observe, based on our collected voting results, that Anthropic’s Claude model is preferred by our users over GPT-3.5-turbo, which is often discussed as its opponent. In fact, Claude is highly competitive even when competing against the most powerful model – OpenAI’s GPT-4. Looking at the win rate plots (Figure 3 below), among the 66 non-tied matches between GPT-4 and Claude, Claude indeed wins over GPT-4 in 32 (48%) matches. Great job Anthropic team!In this update, we have added RWKV-4-Raven-14B model into the Arena thanks to the community . Unlike all other models, RWKV model is an RNN instead of a transformer-based model; but it performs surprisingly well! It soon uptrends on the leaderboard and is positioned #6 on the overall leaderboard. It wins more than 50% of non-tied matches against all other open-source models except Vicuna. You are welcome to check out its to learn more about other features like memory saving and fast inference. Kudos to the RWKV developers.The Elo scores of existing models can go up and down depending on the results of the new games played. This is similar to the way the Elo scores of chess players vary over time (see ). Since the participation of the three strong proprietary models, the Chatbot Arena has never been more competitive than ever before! As a consequence, we observe the Elo scores of all open source models have decreased a bit. This is because open source models lose lots of pairwise matches when they are against the proprietary models.