High-Level Idea. The goal here is to understand the effect of style vs substance on the Arena Score. Consider models A and B. Model A is great at producing code, factual and unbiased answers, etc., but it outputs short and terse responses. Model B is not so great on substance (e.g., correctness), but it outputs great markdown, and gives long, detailed, flowery responses. Which is better, model A, or model B?
The answer is not one dimensional. Model A is better on substance, and Model B is better on style. Ideally, we would have a way of teasing apart this distinction: capturing how much of the model’s Arena Score is due to substance or style.
Our methodology is a first step towards this goal. We explicitly model style as an independent variable in our Bradley-Terry regression. For example, we added length as a feature—just like each model, the length difference has its own Arena Score! By doing this, we expect that the Arena Score of each model will reflect its strength, controlled for the effect of length.
Please read below for the technical details. We also controlled not just for length, but also a few other style features. As a first version, we propose controlling
- Answer token length
- Number of markdown headers
- Number of markdown bold elements
- Number of markdown lists
We publicly release our data with vote and style elements and code on google colab! You can try out experimenting with style control now. More improvements to come, and please reach out if you want to help contribute!
Background. To produce the results above, we controlled for the effect of style by adding extra “style features” into our Bradley-Terry regression. This is a standard technique in statistics, and has been recently used in LLM evaluations [1]. Additionally, there are studies suggesting potential bias for “pretty” and more detailed responses in humans [2, 3]. The idea is that, by including any confounding variables (e.g. response length) in the regression, we can attribute any increase in strength to the confounder, as opposed to the model. Then, the Bradley-Terry coefficient will be more reflective of the model’s intrinsic ability, as opposed to possible confounders. The definition of a confounder is to some extent up to our interpretation; as our style features, we use the (normalized) difference in response lengths, the number of markdown headers, and the number of lists.
More formally, consider vectors X1,…,Xn∈RM and Y1,…,Yn∈{0,1}, where n is the number of battles and M is the number of models.
For every i∈[n], We have that Xi,m=1 only if model m∈[M] is the model shown in the left-hand side in Chatbot Arena, and Xi,m=−1 only if it is shown on the right. That is, Xi is a vector with two nonzero elements. The outcome Yi takes the value Yi=1 if the left-hand model wins, and Yi=0 otherwise.
The standard method for computing the Arena Score (i.e., the Bradley-Terry coefficients, which we formerly called the Elo score) is to run a logistic regression of Yi onto Xi. That is, for every model m, we associate a scalar β^m that describes its strength, and the vector β^ is determined by solving the following logistic regression:
β^=argminβ∈RM1n∑i=1nBCELoss(sigmoid(Xi⊤β),Yi)
where BCELoss represents the binary cross-entropy loss. (In practice, we also reweight this objective to handle non-uniform model sampling, but let’s ignore that for now.)
