Chess Engine - Comparisons - Ratings

Ratings

Chess engine rating lists aim to provide statistically significant measures of relative engine strength. These lists play multiple games between engines on standard hardware platforms, so that processor differences are factored out. Some also standardize the opening books, in an attempt to measure the strength differences of the engines only. These lists not only provide a ranking, but also margins of error on the given ratings. Also rating lists typically play games continuously, publishing many updates per year, compared to tournaments which only take place annually.

There are a number of factors that vary among the chess engine rating lists:

• Time control. Longer time controls, such as 40 moves in 120 minutes, are better suited for determining tournament play strength, but also make testing more time-consuming.
• Hardware used. Faster hardware with more memory leads to stronger play.
• 64-bit (vs. 32-bit) hardware and operating systems favor bitboard-based programs
• Multiprocessor vs. single processor hardware.
• Ponder settings (speculative analysis while the opponent is thinking) aka Permanent Brain.
• Transposition table sizes.
• Opening book settings.
• All listed engines are 64-bit

These differences affect the results, and make direct comparisons between rating lists difficult.

• Note that the listings in the above table only count the best entry for a given engine.

These ratings, although calculated by using the Elo system (or similar rating methods), have no direct relation to FIDE Elo ratings or to other chess federation ratings of human players. Except for some man versus machine games which the SSDF had organized many years ago (which were far from today's level), there is no calibration between any of these rating lists and player pools. Hence, the results which matter are the ranks and the differences between the ratings, not the absolute level of the numbers. Also, each list calibrates their Elo via a different method. Therefore no Elo comparisons can be made between the lists. Nevertheless, in view of recent man versus machine matches, it is generally undisputed that top computer chess engines should be rated at least in the range of top human performances, and probably significantly higher.

Missing from many rating lists are IPPOLIT and its derivatives (e.g. Fire ). Although very strong and open source, there are allegations from commercial software interests that they were derived from disassembled binary of Rybka. Due to the controversy, all these engines have been blacklisted from many tournaments and rating lists. Although Rybka has been accused of being based on Fruit, it is not blacklisted from computer chess tournaments or rating lists. In June 2011, the ICGA claimed Rybka was derived from Fruit and Crafty and Rybka has been banned from the International Computer Games Association World Computer Chess Championship, and its previous victories (2007, 2008, 2009, and 2010) has been revoked. The ICGA was criticized for this decision by Dr. Søren Riis, a longstanding supporter of the Rybka program. All rating lists continue to include Rybka.