RedTeam Arena

People have been asking how we compute the leaderboard of players, models, and prompts. The idea is to treat every round of Bad Words as a 1v1 game between a player and a (prompt, model) combination, and calculate the corresponding Elo score. Doing this naively is sample-inefficient and would result in slow convergence, so we instead designed a new statistical method for this purpose (writeup coming!) and we’ll describe it below.

Observation model. Let T be the number of battles (“time-steps”), M be the number of models, P be the number of players, and R be the number of prompts. For each battle i∈[n], we have a player, a model, and a prompt, encoded as following:

• XiModel∈{0,1}M, a one-hot vector with 1 on the entry of the model sampled in battle i.
• XiPlayer∈{0,1}P, a one-hot vector with 1 on the entry of the player in battle i.
• XiPrompt∈{0,1}R, a one-hot vector with 1 on the entry of the prompt sampled in battle i.
• Yi∈{0,1}, a binary outcome taking the value 1 if the player won (or forfeited) and 0 otherwise.

We then compute the Extended Online Arena Score, with the feature Xi being the concatenation of XiModel, XiPlayer, and XiPrompt, and the label Yi being the outcome of battle i.

That’s it! After updating the model coefficients in this way, we report them in the tables in the RedTeam Arena.

RedTeam Arena is a community-driven project, and we’re eager to grow it further with your help! Whether through raising Github issues, creating PRs here, or providing feedback on Discord, we welcome all your contributions!