Updating ChessDS with new models and XPAR ratings

Today ChessDS launches new model versions, on an expanded corpus, and an updated metric for measuring player performance move-by-move. I’m excited to tell you about the changes.

The recently shipped DSC neural models offer a powerful signal that improved existing ChessDS predictions. Today the site is live with the full corpus expanded (to over 650k games) and re-analyzed with updated models for move and result prediction. These models have improved by leveraging DSC outputs as features, as well as some other additions. Here’s a quick view of the improvements:

Game result prediction

Improvements are most modest here, but they are real. Adding the WDL signal from the neural DSC models, as well as using them to better understand position difficulty, leads to a measurable improvement in prediction accuracy. Here I show normalized entropy, where smaller is better, for the new ChessDS models and a host of other solutions, on my hold-out set of 35,460 GM-level classical games. This compares the mistakes made by the model (the “loss”) to the expected amount of mistakes a model would make if it predicted the baseline probabilities (for example, 25% white, 20% black, 55% draw). Thus a well-calibrated, but random model would score 1.0, while accurate models score lower:

Move prediction

Here the gains are bigger. The new neural models add a valuable signal and push performance higher than I thought was possible. On my GM set, the new model predicts the top move over 66% of the time, and improves on other measures as well. I’ll also add the new Toby 1.0 model does very well here and uses a similar method.

Performance measurement

This was the hard part. It turns out adding DSC features, even from the versions that don’t see game history, improves move prediction but weakens the XPA signal. This is complicated and maybe I will unpack in a future blog. The basic idea is, the models are so smart they start to understand that a player is strong, and predict they will make strong moves, which means the signal of what a baseline player would do is lost. And in practice it’s very hard to prevent that signal from leaking in.

Even without DSC, I was able to improve the XPA model, and also now use it to show an Elo-style “XPAR” rating. This rating is shown in two forms, which I’ll say more about soon:

• Career XPAR. This is the move-by-move performance measure built from a player’s entire career of games.
• Ranking XPAR. This is a brand-new player ranking, which indexes on play from the last two years, and rewards players who play more by considering rating uncertainty.

I’ve validated this in many ways; one straightforward test is to correlate player FIDE rating with XPA/XPAR. The new measure is a clear improvement on what I had before. An improvement that did not come easily!

I’ll be getting back to analysis soon powered by these new tools. And more to come about how those player rankings are produced, including a new page that will support viewing them.