How Many User Prompts are New?

For roughly 75% of the prompts collected each day, there is not a similar prompt submitted on a previous day. This indicates that roughly 75% of the prompts submitted each day are fresh.

If you look at the above analysis, the proportion of fresh prompts decreases as a function of t. This is expected, since as t grows, we are comparing new prompts with an ever-larger set of past prompts. For example, when t=1, there are no previos prompts, so of course, the proportion of unique prompts is 1/1=100%.

However, as t grows, this number stabilizes to around 70-80% fresh prompts at a similarity threshold of 0.7. This equilibrium represents the fraction of fresh prompts that we expect chatbot arena to generate in the long run.

Interestingly, we also see certain dates where prompt freshness is significantly lower than neighboring dates: we will get to why that is in the next section.

We find that many of the duplicates can be attributed to 3 things: “tester”prompts, hi/hello’s, and prompts asked multiple times by the same user back to back.

Hi’s and Hello’s. We see that 2.1% of our data is some variation of “hi” in various languages. As per our deduplication policy, these are deduplicated when calculating the final rankings.

Tester prompts. There are certain prompts that users have found to stump most LLM’s, like “how many r’s are in strawberry” or “what is bigger, 9.11 or 9.8”. When a new model comes out, these prompts are commonly asked to gauge performance, which can be a source of days with a low proportion of fresh prompts. For instance, the week of August 8th saw a large decrease in prompt freshness, which coincides with a release of an update to GPT-4o. Looking at the top prompts on those days we see many of these prompts are a version of “how many r’s are in strawberry”.

^ Strawberry and Nearest Neighbor Matches.

^ Most Common Prompts by Week (excluding "hi" prompts).

Repeated prompts by the same user. Many duplicate prompts are submitted by the same person on the same day. Comparing prompts to every prompt seen at an previous timestep (rather than a previous day or week) 65% of prompts at a given time have been previously seen. However, most of these duplicates occur on the same day, with 60% of these prompts submitted by the same user. This indicates that users are asking a prompt, voting, then starting a new battle with two new models and asking the same prompt. This is encouraging because the models used in each conversation vary, which helps maintain diversity in prompts across different model pairs and results in more consistent voting from the same user. Removing duplicate prompts submitted by the same user on the same day raises the percentage of unique prompts from 65% to 80%.

Lastly, we wanted to ensure that the prompts are not contained in commonly used benchmarks. Using the same similarity measure, we find a very low percentage of user prompts are seen in existing datasets, reducing the likelihood that models which overfit to these benchmarks are given an advantage in the arena.

The majority of our user prompts are fresh (i.e., 75%), and the data is not contaminated by existing benchmarks. A sample of the data with their nearest neighbors to the original prompt can be found in this space. We are excited to see how this data evolves over time!