Testing XPA: Elo ratings, accuracy, and a hint of deflation
ChessDS recently shipped a new metric “XPA,” which aims to measure the points of “Expectation” a player gains or loses relative to a baseline of a strong GM (think: 2600 player). Expectation is in terms of the full point available in a chess game: if you are sure to win, your expectation may be 100%; if a draw is likely but you have some edge, your expectation might be 60%. A full explanation of how XPA works is in this previous blog post.
To validate that XPA works, and argue that it works better than accuracy (using the Lichess implementation), in this post we’ll take a look at how XPA and accuracy correlate with Elo ratings for top chessplayers.
Producing the data
For this exercise I consider all analyzed classical games in the dataset with Elo ratings I can find or closely approximate. This limits us mostly to post-1970, but still gives 94,447 games and 5,452 players to consider. Across this sample, I compute Lichess Accuracy% per game and XPA per move.
In the chart below, these measures are averaged per-player, and plotted against each player’s average Elo.
This chart is so interesting to me! Some features jump out:
- Both XPA / move and accuracy % are strongly correlated with Elo level. This is not surprising – playing the strongest moves is a good way to perform well!
- XPA / move discriminates Elo better than accuracy. Again not surprising to me, but it’s good to see! By subtracting out the expected error of a strong GM, as XPA does, we are left with a truer estimate of player performance, adjusting for how easy a given position is to play.
We can also pool across all players, which highlights the different construction of the two metrics. Accuracy uses a range from 92 to 96 on its 100-point scale to describe GM play, with an unclear baseline for comparison. XPA crosses the 0 line a bit past 2600 level, reflecting its design to show performance compared with this baseline.
Building on this foundation
With this insight in hand we can start to ask more interesting questions. For example, have ratings been stable over time? I took an early look here, looking at how the play of a ~2600 GM (players rated 2550–2650 at the time they played) has measured since 1970. And I do see some signal of deflation!
It appears that starting around 2005, the average “2600” GM started playing better, and it continues today. There are many interesting follow-ups here (and elsewhere) and I will make time for them soon!