Rules for annotations at ChessDS
Computers are much better than humans at playing chess. But one aspect of the game where they are arguably still behind is in annotating chess. Producing annotation symbols takes taste: sensing the magic behind a ! move; judging whether a mistake deserves ?, ??, or just ?!; the intuition of when a move is interesting enough for !?. It takes a human!
Of course, that hasn’t stopped chess sites and programs from producing these with machines. Count me in! It’s fun to try, and credible machine-produced annotations can scale immediately over massive databases, making them more useful and informative.
Annotations at ChessDS have been overhauled. The analyzed dataset (as of this writing over 500k games) is now fully annotated with new rules I’ll describe here.
Four cornerstones of ChessDS annotations
The rules below draw from four basic sources of information about a given chess move:
- Result probabilities. Produced by the main result prediction model, these are used to understand mistakes (which lose expectation) and strong moves (which gain expectation). They are also used in some cases to determine when a position is still contested. Forcing sequences are advanced to see their full impact at the end.
- Material offered. This is the stuff brilliant moves are made of! Material offered is computed with a Static Exchange Evaluation (SEE)-based algorithm (with some local modifications) which assesses how much material is offered by a given move.
- Position difficulty. Here we can draw on expected loss, which is calculated by integrating over the predicted moves and weighting them according to their move probability. It’s a component also of XPA.
- Move surprise. This is not based on the move prediction model, which tends to expect elite players to play strong moves the rest of us would find surprising. Instead it uses the policy model, which is a move predictor built without evals or any search at all. This approximates how “natural” each move is, without any concrete calculation.
The rules
Here are the ten rules now driving ChessDS annotations. They are evaluated in order as presented, and the first matching symbol is written:
These are hand-derived, based on a wide range of human-annotated and noteworthy games.
Descriptive stats
The best way to get a sense of these and see if they work is to go check out some famous games! A few good examples include Byrne – Fischer, 1956, Kasparov – Topalov, 1999, Morphy’s Opera game, 1858, Anderssen – Kieseritzky, 1851. But we do data science here, so let me also give you some figures. The following counts annotations by symbol per game with these rules, on our half-million classical corpus:
It’s also another interesting source of data for comparing eras. This needs to be done carefully; the mix of analyzed events is not matched across time. Here I exclude team and Swiss events, which are more common recently, and plot symbols per game across time:
These feel a lot better to me than the v0 versions the site launched with, but definitely not perfect. I’d love to hear what you think!