A guide to Endgames Tablebase

Introduction

This page was initially part of my web site on setting up of Winboard engines. However, I discovered that many visitors to my site, were actually looking for information on setting up endgame tablebases. Given the fact that my Winboard web site was already very long, I decided to start a new web site using the information already available in the original page as a base. I added information from the various web sites and past posts at Rec.games.chess.computer and the Computer Chess Club.

I'm a layman who is largely unfamiliar with the technical details of endgame tablebases. As such much of the technical detail here is only available thanks to the kind permission of experts such as E.A. Heinz,Guy Haworth, (both frequent contributors to ICGA journals on this topic) Anders Thulin (from rec.games.chess.computer) , Robert Hyatt, Bruce Moreland and especially Dieter Buerssner (all of those are authors of strong chess programs that use endgame tablebases). I would be grateful if you can email me if you spot any errors.

I hope that you will find this web site useful. If so, please help sign my guest book.

Yours Sincerely,

Aaron Tay
01-04-2001
My PGP - DH/DSS 4096/1024 Key



Further information

As of April 2006, one of the best places to obtain further information about endgame tablebases is at the following forum . I highly recommend it, if you have further questions, particularly technical ones.

Latest Changes

(To skip to FAQ table of contents )

31-05-2003 Made available replacements for flawed EDGB file sets installed by Chessmaster 9000 .

09-05-2003 Knightdreamer now supports bitbases.Also available as a dll for free use\ with other engines. Bitbase section rewritten.

01-03-2003 Yace now released uses 3 and 4 piece bitbases stored in memory.

06-02-2003 Added explanation of how endgame tablebases are generally constructed.

26-06-2003 E.A. Heinz's reference added. Bitbase section reorganised.

23-09-2003 More 6 piece tablebases loaded on Hyatt's FTP site. Including 6 piece tablebases with pawns.

23-05-2004 New viewer site by Convekta allowing access to 6 piece tablebases added

25-07-2004 DVD Endgame Turbo by Chessbase contains a corrupted file for the KRPPKR endgame.

08-10-2004 Added alternative site for online searching of Nalimov tablebase results here.

01-12-2005 Add more sites for searching online Nalimov tablebase results. Added link to emule collection of tablebase sites.Added Scorpio uses bitbases and offers dll for use.

25-04-2006 Cleaned up sections on six piece tablebase, and download sections.


Table Of contents


[A.1] What is a endgame tablebase? How do they work? How much stronger will it make my chess program play?

Chess endgame tablebases (EGTBs) are special databases that stores all possible positions with a given material balance and their results.

Note about terminology

Why not call them endgame "database" instead of "tablebases"? I have no idea, in fact Chessmaster's version of them are called EGDB (EndGame DataBase) , but regardless of what they are called they are similar in principle. In this FAQ, I shall use the terms Endgame tablebases because these are the names they are most widely known by.)

For example, there is one tablebase file (normally named KNNKP) that covers positions where one side has a King and 2 knights, while the other side has a king and a pawn and yet another tablebase file (KQK) that covers all positions where one side has a King and Queen, while the other side has a lone King.There are more endgame tablebase files of course, though naturally the more pieces on the board, the bigger the Endgame tablebase file.

Each endgame tablebase file allows the user to check what the result is with best play. For example, the KNNKP tablebase,can allow you to input a specific position with such a material configuration, and find out what the result should be with best play. Nalimov endgame tablebases (one of several types of tablebases),for example could tell you if that position is a forced win and if so in how many moves, or a draw with best play.

One thing to note is that,the raw endgame tablebases files are highly compressed,and require that you use some kind of graphic browser software, before you can view the files. .However, many Chess playing programs [See here for a list] can use endgame tablebases automatically,which will allow them to play that endgame perfectly.

Most chess engines don't even need to reach the position covered in an endgame tablebases to use the tablebases. For example, a few moves before such a position, the engine calculates (but does not play yet) a series of exchanges that leads directly to a position in the tablebase. The engine will then look up (or probe) the tablebase and get the results for that hypothetical position. This should help improve play of course.

But how are tablebases constructed in the first place?Basically endgame tablebases are database files of stored endgame positions calculated using retrograde analysis.Here's a basic layman explanation on how tablebases are constructed Here's another explanation by Bruce Moreland, author of Ferret (who has generated endgame tablebases of his own design).

The algorithm works via induction.

How these things works is that all possible positions are examined, and positions where one side is mated *now* are marked (apparently legal piece configuration, with one side to move, in check, and unable to get out of check). These are called "mated in 0" positions.

Once this is done, all positions with each side to move are examined, and if any of them can reach a mated in 0 position, these are marked as "mate in 1".

Now, we look at everything again and search for positions where all possible moves lead to "mate in 1" positions. These are marked as mated in 2.

Now, we try to reach "mated in 2", these are "mate in 2".

Next is a bit different, because we look for cases where it is impossible to avoid either "mate in 2" or "mate in 1". These are "mated in 3".

And so on, until no progress is made. The rest of the positions are proven to be draws.

It's a little trickier if it's possible to convert to a won ending, if you are trying to make a "distance to mate" database. You have to be careful of this idea of "no progress", because there are gaps. There might be a mate in 105 in KBP vs KN, but there might not be a mate in 95. This is because the mate in 105 might involve immediate conversion to a KBN vs KN, and yes, such positions do exist.

How much stronger will it make my chess program play?

Endgame tablebases allow chess programs to play better in 2 situations. Firstly, when the endgame position covered by the tablebase occurs on the board. This is most always a gain for the chess engine (assuming you don't have a incomplete tablebase set). Unfortunately, the current set of endgame tablebases (5 pieces or less) cover mostly simple endgame positions that most chess engines can handle either by a combination of search or knowledge programmed in. So most engines will not gain much from using tablebases this way (except in rare positions).

A much greater gain is achieved by the fact that chess engines can consult tablebases as a kind of oracle to decide whether to exchange down into a endgame covered by the tablebase.This allows the range of positions to be covered to be much larger. A chess engine can for example avoid exchanging down to losing positions (even if they are very deep) or win by converting into a absolutely won endgame.

And yet, people have estimated that most engines gain at most 25 elo from using them, and many have found no significant difference in playing strength. Experiments results by Robert Allgeuer and Les Fernandez using Crafty and Yace show no significant difference in playing strength.

In this article I listed 5 possible weaknesses of endgame tablebases that might cause this. Of the 5, some are quite rare (problems #2,#3) or can be avoided (problems #1) , except for the last problem which is the resulting slow down from accessing the hard-disk.

Robert Allgeuer did a interesting experiment with Yace. And basically he found that while endgame tablebases allowed engines to reach and win such positions, in games where such positions were not reached they played weaker. By considering games where positions covered by endgame tablebases was actually reached on board ,Yace with tablebases played significantly better (about 115 Elo) On the other hand, in games where such positions was not reached, Yace with endgame tablebases played significantly weaker (about 94 elo). The implication was that in such games, using endgame tablebases was a liability. Robert writes "Apparently statistically for each game they successfully maneuver into a won 5 piece position, they also lose another one during the preceding endgame, so that overall this effect more or less cancels out the advantages of tablebases."

The above results can be still extended by using different time controls, egtb caches, engines (some like SOS claim to be heavily dependent on tablebases because the author has stripped out all endgame related knowledge covered in tablebases) , but it seems that using the current set of tablebases (3-4-5 pieces), you can't expect more then a minimal strength increase if any. Still you might want to use tablebases for learning and analysis purposes.



[A.2] What are the different endgame tablebases formats out there? What are the differences?

There are various kinds of Tablebase formats widely in use, including Ken Thompson,Steven J. Edwards,Eugene Nalimov Tablebases and Chessmaster/De Koning EDGB (in the order in which they first appeared). While these are popular, some programmers have chosen to use endgame tablebases of their own design, for example Bruce Moreland's Ferret

In general though, they fall into 2 types of tablebases. Distance to mate (DTM) and Distance to Conversion (DTC)tablebases. Roughly in layman terms, the difference is as follows.

The tablebases also various in other ways Dr Robert Hyatt [Author of Crafty] explains the differences between them as follows [in a posting to rec.games.chess.comp 26/10/2000]

"Edwards (Tablebase): Distance to mate values stored. The main problem with these is that they are larger than the others.

Nalimov (Tablebase): Distance to mate values just like Edwards, but Eugene's files are compressed, and they may be used in the compressed form, with no penalty of any kind. Rather than way over 30 gigabytes for all of the 3-4-5 piece files, you end up with about 7.5 gigs.

Thompson (Tablebase): Distance to conversion (capture which takes you to a smaller ending class). These are difficult to use in their compressed form, from inside a chess engine. They also provide different info than Eugene's database... i.e. it tells you something, but it doesn't differentiate between losing and drawing as the Nalimov files do.

Best choice: Nalimov. Nearly every chess engine supports those... "

This was of course written before the release of the De Koning EDGB format used by Chessmaster 9000 only in August 2002. Nalimov still remains the most popular tablebase format since it's supported by the largest number of engines.

About Chessmaster endgame tablebase / De Koning EDGB

De Koning EDGB - Propertry format used by only Chessmaster 9000. Like the Nalimov tablebase it is a Distance to mate tablebase. Unlike the Nalimov tablebase though, it stores only wins from the white point of view (similar to Thomson)."(It) searches only for wins for White. All positions that are not proven to be won by White are stored as unknown. These unknowns might be draws, or lost by White, or (in large numbers) illegal." [From FEG or Final endgame generator for Chessmaster 9000 ]

Note the original version of the endgame generator had a minor flaw, and the EDGBs that shipped with the CDs also had this flaw. See here for more

Other differences between the De Koning and Nalimov formats are

This change means that besides the usual pawn promotion case problems there are other "dependencies" problems between EGTB files (my comments)

  • Chessmaster 9000 currently does not probe the Tablebase in search but according to email correspondence this is a decision design and there is nothing in principle that stops this from occurring.

  • De Koning EDGB are slightly smaller than compressed Nalimov endgame files in general.The full De Koning EGDBs came up to about 5.7 GB , the EGDB's can function with only either the WTM or BTM files.[Note during generation, you need both files!] This can reduce the space required to as little as 2.7 GB which is extremely light on hard-disk space compared to the 7.5 GB (as quoted by most people) required for Nalimov tablebaes.

  • The Chessmaster EDGB format generation process is also faster compared to Nalimov. (Full set of 5 piece tablebase take 5 days on 1.2 GHZ Athlon) It also requires much less free RAM (less than 13MB to be precise). The generator also makes generating 6-man with KWWWWB possible on a 32 bit computer possible , but that requires about 300MB (which isn't a lot by todays standards) of free RAM

  • 4 piece KxyK + KxKy 23 MB
    5 piece KxyzK 1.25 GB
    5 piece KxyKz 3.47 GB
    5 piece KxKyz 1.04 GB (The numbers designate the full set. For simple lookup the half set will do (either wtm or btm). The half set is somewhat smaller than half of the full set.)

  • Otherwise De Koning EDGB's are similar to Nalimov in terms of castling and 50 moves rule (see below).

More about Nalimov and Compressed Nalimov tablebases

Nalimov tablebases are nearly "perfect" since they take into account en Passant. However, they don't take into account castling, but this flaw is probably of interest only to Chess problem Hobbyists.

In general, though almost all modern Chess programs [including most Winboard programs] use Nalimov Tablebases partly because they are non-propriety and partly because they are more efficient. Also some of the 6 pieces Tablebases are now available in the Nalimov format. The Nalimov tablebases comes in 2 forms, uncompressed and compressed. The compressed ones end with the extension "emd".

Most of the modern Chess programs( e.g. Crafty supports to the use of the compressed Nalimov format from versions 16.5 onwards) can use the tablebases in compressed form by uncompressing and using them on the fly as needed. One exception I'm aware of is a old version of Esc , a Winboard program released on 4 Feb 2001 which uses only the uncompressed form. If you generated the Nalimov tablebases yourself (see Section [A.8] ), they will already in the uncompressed form before applying datacomp.exe. datacomp.exe can also be used in reverse to get uncompressed Nalimov tablebases from compressed ones.

Other uncommon types of Tablebases

Bitbases

There are other types of less common tablebases that work quite differently from those mentioned above. They generally go by the term "bitbases".

There is a short articleby Roland Piffer, author of Patzer explaining what bitbases are. Unfortunately, it's available only in German.

Here's a layman explanation attempted by yours truly.Unlike normal tablebases that store the full distance to mate or conversion, bitbases only stores information about whether a position is a forced win or loss. Sometimes, this alone is enough to help a chess engine to discard lines quickly. However bitbases (unlike typical TBs like Nalimov tablebases) does not store the exact score (eg mate in x) and hence cannot be used to work out the best winning line directly. However because bitbases need only store whether a position is a win or draw as opposed to whether it is mate in x (where x can be anything from 1-128 or more), they are naturally smaller and faster to access and can fit into the computer's RAM. They can be used together with more traditional Endgame Tablebases (Nalimov etc).

Yet another note about terminology

The term "bitbases" comes about because it requires only one bit to store the results for each position. Again, the naming might differ, e.g. knightdreamer calls them Endgame models (egm).

E.A Heinz was probably the first one to pioneer the concept. They were integrated into his chess program "DarkThought" by late 1996 and proved their practical viability during the 1997 WMCC in Paris where "DarkThought" employed them throughout the whole tournament. In addition, he also published an article on it in ICCJA (see below) in 1999. Eventually bit-based endgame databases started to appear in many other microcomputer chess programs (e.g. According to E.A Heinz,Chrilly Donninger integrated them into "Nimzo" based on the ICCA Journal article cited below).
E.A. Heinz. "Knowledgeable encoding and querying of endgame databases." ICCA Journal, Vol. 22, No. 2, pages 81-97, June 1999. (published by the ICCA; preprint available from http://lcs.supertech.lcs.mit.edu/~heinz/ps/know_edb.ps.gz or http://www.i-u.de/schools/heinz/ps/know_edb.ps.gz )

Unlike normal endgame tablebases where there are only a few standard formats used by almost all chess engines, so far no open standard bitbase format exists and the few chess engine programmers who have implemented this idea, use bitbases of his own design. These are the ones that I'm aware of which use a form of bitbases.

1) Nimzo 8 uses proprietary "Nimzo tablebases" (probably a form of bitbase too) that are stored in RAM for faster access.

2) Patzer, which is available as part of the GambitSoft Winboard Edition II package uses bitbases too.

3) Yace versions (after 0.68) also use bitbases.Yace uses bitbases for 3+4 pieces only and this is stored in memory . This allows Yace to access them in Quiescience search without much slowdown compared to using Nalimov tablebases.

4) Knightdreamer does something similar with it's own Egmprobe.Like Yace's bitbases, there are currently only 4 men tablebases and are small enough to be loaded in RAM. More technical details, A decision tree grown with an ID3-style algorithm is used to predict if a position is won, lost or drawn. Eugene Nalimov's EGTBs are used as an oracle. An accuracy above 99% is reached for most endgames. A table with the exceptions is compressed with run length encoding and huffman codes. The exception table is probed on the fly, without generating any decompressed tables. Knightdreamer uses this to "adjust scores" (increase scores if there is a absolute win, divide the score to get a score closer to zero if a draw is possible with best play) to help aid scoring (used in pruning too) but does not consider any wins found by EGM as a absolute win. This is because occasionally knightdreamer is unable to win absolutely won games, or may misplay technically drawn positions. Most interestingly, EgmProbe is distributed as a DLL for free for non-commercial engines. To download all the egms for all 5 pieces tablebase (except for 4-1) go to http://cap.connx.com/Contrib/Johan/KnightD33.zip(211 mb).

5) Delfi has internal bitbase tablebases (stronger side wins/draws) for almost all the 4 men tablebases except N vs P and R vs B and KRPKR. For more information.

6) Scorpio you uses bitbases up to 4 piece. He also offers a dll to access these bitbases for those interested. Fruit Gambit and latista uses this too.

7)GAFS

Guido from G6 [ a group of 6 Italians Winboard engine authors] has announced a new endgame tablebase format - GAFS .Guido writes

It is also possible to generate EGTBs with 2 bits per positions (win, loss, draw, illegal position). There is also a very small object file to link to the chess program in order to use 8 bits and/or 2 bits EGTBs and this is possible inside the same run. The idea was that 2 bits tablebases could be sufficient during the search process before entering a tabulated ending, while inside the ending only 8 bits EGTBs can give the correct move. I don't use compression at least for now. To the persons that tried to use my EGTBs I suggested to read the result on the disk file (without loading the file in RAM), and save it in the hash table. If the number of different positions to read, i.e. access to the file, are not very high (no more than some thousands), the cost in time should be acceptable, as a single access to the disk can cost some ms.
You can find more information on http://users.libero.it/rigel_g/ . The only known available chess engine that uses it is Madeline (look for info on the engine)

8) By 2005, there are now many more engines that use bitbases, many of their own design. Examples include Kiwi, Slowchess,Glaurung .

I have also heard of tablebases being generated for other Chess variants like Losing Chess , but I have not be able to find more information.

NEW! [04-01-02] Sjeng 11.2 now allows you to self generate 2 and some 3 tablebases for suicide Chess.KKFchess does the same for various variants as well as supporting normal Nalimov tablebases.

If you have more information,web sites with more information about endgame tablebases to link to, or clarifications that you want to make, please emailme

Note: For the rest of the FAQ, unless I state otherwise, endgame tablebases refer to Nalimov tablebases.


[A.3] Where can I learn more about the endgame table formats?

For Nalimov tablebases you need written permission from him. You might want to look at probe.txt which is part of TBGen.zip. Which can be normally be found in most places which allow EGTBs to be downloaded)which if I understand correctly, gives instructions on how to hook up your Chess Engine to probe Nalimov's tablebases.Looking at open source programs might also help for example, "probe.c" file from BeoWulf or egtb.c file from Crafty.This post might also help .

Attention! For Chess programmers trying to learn more about the format to figure out how to make their Chess programs use Eugene Nalimov tablebases, : Dr Ernst A. Heinz who is the author of the Chess program, Dark Thought and author of the book " Scalable Search in Computer Chess " writes:

"My WWW pages at http://supertech.lcs.mit.edu/~heinz/contain preprints of two interesting texts about the index schemes of endgame databases for chess (incl. Eugene Nalimov's).Specifically, you might want to take a look at:

* E.V. Nalimov, G.McC.Haworth, and E.A. Heinz. "Space-efficient indexing of chess endgame tables." In ICGA Journal, Vol. 23, No. 3, pages 148-162, September 2000. (published by the ICCA; preprint available from http://supertech.lcs.mit.edu/~heinz/ps/NHH_ICGA.ps.gz

* E.A. Heinz. "Endgame databases and efficient index schemes."In ICCA Journal, Vol. 22, No. 1, pages 22-32, March 1999. (published by the ICCA; preprint available from http://supertech.lcs.mit.edu/~heinz/ps/edb_index.ps.gz

[In a posting to rec.games.chess.comp, 03/13/2001]

You can download for free the FEG or Final endgame generator for Chessmaster 9000 . It comes with a nice document file that explains clearly in layman terms what endgame tablebases are and summarises some of the issues involved.


[A.4] What are the Chess programs that support endgame tables? Which format do they support?

Here's a small list.

Program Name Endgame Tablebase types supported

Fritz 6.0+

Nalimov

Junior 6.0+

Nalimov

HIARCS 7.32+

Nalimov

Nimzo8+

Nalimov, Nimzo Tablebases

Chess Tiger 14.0+*

Nalimov

Rebel Tiger 2.0*

Nalimov

Chessmaster 9000 and up

De Koning format (new)

Shredder 5.0+

Nalimov , Thompson

Chess Genius 6.5

Thompson

Patzer**,Yace Paderborn

Nalimov , Bitbases

Knightdreamer,Delfi

Bitbases

Crafty***

Nalimov

Gromit 2.2,Dragon [Pre-4.0],Bionic

Edwards

Most free Winboard engines****

Nalimov

SCID

Nalimov

Madeleine

Nalimov and GAFS

* The latest update 02/04/2001.Refer to Rebel Web site . Prior versions did not have any endgame tablebase support.

** Available as part of the GambitSoft Winboard Edition II package.

*** Old versions of Crafty used Edwards Tablebases.Crafty 15.21 onwards supports only Nalimov tablebases. Crafty 16.5 onwards also supports the use of compressed Eugene Nalimov tablebases.

**** For a full list of Winboard engines that support tablebases , refer to Engine Overview page by Leo Dijksman

Effects of GUI like Chessbase/Chessmaster

Additional note, when Chess engines are used in Chessbase GUI or Chessmaster 9000, the GUI will use any tablebases available once the position on the board matches that of the relevant tablebase. But this does not mean that the Chess engine itself "uses" the tablebases in it's search etc.


[A.5] Where can I get endgame tablebase?

Nalimov tablebase

If you own only Fritz 6 or any Chessbase engine, you will have the 3-4 men Compressed nalimov tablebases, but you might want more. You basically have 3 choices. You can buy , download or generate them. [From easiest to most difficult.]

Chessmaster EDGB or De Koning EDGB

If you own Chessmaster 9000, it will already come with all 4 piece tablebases together with some 5 piece tablebases, particularly the KRPKP and KQPKQ and their respective depedencies.The set that comes in the CD is as follows

KQRKR,KRQKR,KRKRR,KRRKR,KRBKR,KRKRB,KRNKR,KRKRN,KRPKR KRKRP,KQQKQ,KQKQQ,KQRKQ,KQKQR,KQBKQ,KQKQB,KQNKQ,KQKQN KQPKQ,KQKQP,KBBKN,KNKBB,KNNKP,KPKNN

Unfortunately, at time of writing 21 September 2002, there is no place to download or buy De Keoning tablebases for Chessmaster 9000, so you will have to or generate them yourself.

Thompson Tablebases

Thompson Tablebases was the first endgame tablebases to be widely used by chess programs. Older versions of chessbase , Fritz etc used to support them. Unfortunately, they are not used much today, because they are largely inferior to Nalimov's. (E.g They don't cover en-passant, not all 5 piece tablebases available, they are larger, slower to generate and use , and only one side to move was generated). I don't know of any place where they are available for download (not sure if that's legal), and while they used to be for sale, it's likely they are not now.

Edwards Tablebases

There's a generator available somewhere on Dann Corbit's ftp. Also 3-4 piece tablebases available for download on Dann's site.

Bitbases of various formats

Because bitbases are generally small and available for 3-4 tablebases (so as to fit in memory), most either come with the program as separate files (Knightdreamer,Nimzo), are integrated internally (Delfi), or can be generated by the program itself in a couple of minutes or seconds (Yace).


[A.6] Where can I buy tablebases?

Buying Nalimov tablebases

You can buy The Chessbase DVD Endgame Turbo 2 which is a 5 DVD set that has all of the Eugene Nalimov 3,4,5 and some 6 piece tablebase set. Conveka makers of Chess Assistant also sells all 3-4-5 Nalimov tablebases in either a DVD set and another set consisting of the above plus some 6 peice tablebases in 2 DVDs.

Buying Chessmaster EDGB or De Koning EDGB

As yet, this is not available.

Buying Thompson tablebases

You used to be able get Thompson Tablebase with certain software packages like the "World Champion package" from Millennium System. Chessbaseused to sell a 4 CD set of 5 men Thompson Tablebases both separately and for use with Chessbase 7

I have no idea if Edwards Tablebases are available for sale.


[A.7] Where can I download tablebases?

Downloading Nalimov tablebases

You could download them file by file for the 3,4,5 and some 6 men Nalimov compressed Tablebase from UAB ftp. This is the original source and will always have the latest and most complete sets available as soon as Eugene Nalimov sends them. But see comment on 6 piece tablebases. You will need a fast internet connection if you want to download the 5 men tablebases which are very large. , These files are already in compressed emd format, and it doesn't help to further compress them with pkzip etc.

Alternatively, Frank Quisinsky offers all the 3 and 4 men tablebase for download in one 30 meg file.

Another site that you might consider looking at if the UAB site down (unless you want 6 piece tablebases)  is is at http://www.chesslib.no/ [click on 'downloads' at the top] .

If for some unusual reason you need uncompressed Nalimov piece sets, you should uncompress them using datacomp.exe that comes with tbgen.exe.

If all these sites are down, you can try downloading them via emule .

Downloading Chessmaster EDGB or De Koning EDGB

No known site exists.

Downloading Edwards Tablebase

Edwards Tablebases (35 meg) are available for download at Dann Corbit's ftp site.


[A.8] How do I generate tablebases on my computer?

Generating Nalimov tablebases

Lastly, you can generate Nalimov tablebases using the tablebase generator tbgen.exe. The same generator is also on the Fritz6 CD. But self generation of 5 men tablebases is not really feasible unless you have a very fast computer with lots of RAM especially. You can find a guide by Mogens Larsen about generating 4 piece tablebases. Antonio Senatore also made a nicepost at the Winboard forumthat offers some guidance if you really want to try to generate 5 pieces tablebases.

See also Section [A.11] .

Generating Chessmaster EDGB / De Koning EDGB

First download the FEG or Final Endgame Generator. When you choose to do a full install, all the tablebases will be copied to c:\mydocuments regardless of the actual directory Chessmaster 9000 in.

Important note: If you use the full installation of Chessmaster with EDGB and/or downloaded and generated other tablebases using FEG version 3.2 (you can check the readme file to check the version) , the tablebases you generated and some that came with the CD are flawed.In a nutshell

Endgames with mates no longer than 1 (eg KNNK) did not contain the black-is-checkmated positions, or rather their wrong value being "not_white_win". This affected also some positions leading to black-is-checkmated by capture (eg KNNKR with the mating move being NxR).

This bug will affect Seven of the 4 piece endgame tablebase and Thirty five of the 5 piece endgame tablebase.

Of those that came with the Chessmaster CD, the following sets:
KNNK,KBKB,KBKN,KBKP,KNKR,KNKB,KNKN,KNNKP,KNKBB falls prey to this bug.

I have generated these file sets and put them into one file for your download. You can them unzip them over the EDGB files installed from the CD. The chessmaster EDGB file sets are 12mb zipped and contain the Jix and Joo files necessary to be used in chessmaster. For the 2 five tablebase set KNKBB and KNNKP, I have made available only the btm files because only those are installed by the chessmaster CD.

Others that are flawed include KNNKQ,KNNKR,KNNKB,KNNKN,KNPKQ,KNPKN,KNPKP,KBKQB,KBKQN,KBKQP,KBKRB,KBKRN, KBKRP,KBKBB,KBKBN,KBKBP,KBKNN,KBKNP,KBKPP,KNKQR,KNKQB,KNKQN,KNKRR
,KNKRB,KNKRN,KNKRP, KNKBN,KNKBP,KNKNN,KNKNP,KPKQR,KPKRR,KPKRB

For more details see this post at CCC. Download the new FEG generator here

To change the directory where chessmaster looks for tablebases edit cm.ini (found in the chessmaster 9000 directory) with a text editor. You need to change the path ofEGDBPath=whatevernewapth where your tablebases are.

A second bug was found on Dec 26, dubbed "the transparent pawn bug". "Affected files are the ones where both sides have at least a Pawn and White has at least a sliding piece. As well as the ones relying on them via promotion (KPPKP) or capture (6-man only) or both. Regarding 3,4,5-man these are KQPKP, KRPKP, KBPKP, KPPKP."

Note, that even though there are some 5 piece tablebases on the CD to save space the 5 piece tablebases on the CD do not come with the WTM files.This means you have to regenerate all the 5 piece tablebases if you want the complete set due to the depedencies.


[A.9] How large are the tablebase files? Can I put then all into one directory? Do I have to use a complete set of 5 men tablebases?

Note the following refers to Nalimov tablebases. Users using Chessmaster format should ignore this section. Chessmaster 9000 itself is not vulnerable to the incomplete tablebase problem.

The more tablebase files you install, the stronger the program will be. However, a full set of 3,4 and 5 Tablebase files takes about 7.05 GB of Hard disk space! BTW many of the "complete" endgame tablebase sets you can buy on CD are not really complete and may exclude some of the less important tablebases for example the so called 4+1 men tablebase sets [KXXXK] on the assumption that those are trivial enough to solve. Most people download a full set of 4 Piece databases and select only a few of the 5 pieces.

Note you can mix all the different endgame tablebase into one directory if you wish. However if you do decide to split up the files among various directories (there might be some advantage in leaving a small subset of tablebases (clear wins like KXXXK in a chess engine game as opposed to for use in analysis.See here), you might need to check whether the Chess engine/interface you use allows you to point to multiple paths.

NEW! Dieter Buerssner has informed me that in general all Chess engines/interfaces allow the use of separate paths for nalimov tablebases as long as they are separated by " commas, semicolons, or (at non- Windows systems) by colons." I have tried testing with a few engines, most seem to work. For example in Pepito , I add EGTB_PATH C:\fourpiece;g:\fivepiece.

In addition, Dieter offers another interesting tip. " In console mode, or with Alt-F1 (in Winboard) you can probably change the path for many engines. For example for Yace, assuming it had some tbldir in yace.ini, and I saw the analysis of one position with TBs. Then I want to see it without TBs. I would type tbldir nul "

For Crafty you need to set the paths (separated by ;) in the command line (or Winboard.ini) and not in the Crafty.rc file. (I'm not sure if this applies to newer versions anymore)

However if you put all the tablebases in one directory as opposed to splitting them up say by tablebase pieces, it can get confusing to keep track of which tablebases you have and which you don't. You can find some programs that help keep track of this in Section [A.19].

Generally the 5 piece endgames with rooks are reached most frequently and should be downloaded .Important ones include KRPKP, KQPKQ and KPPKP. However, there are some pitfalls that you should be aware of.

If you want to use the KRPKR tablebase [and assuming you have all the 3 and 4 tablebases], make sure that you have the KQRKR one as well. [You might consider adding KRRKR, KRBKR , KRNKN as well but it's probably not necessary] This is to ensure that the promotion cases are included.

endgame problem

If you lack say the KQRKR tablebase, some programs refuse to queen the pawn even if that would lead to a win, because such a move, would cause the program to drop from a position flagged as "win", to a position that is uncertain [although with a high score] since they lack the relevant endgame table.

In the position on the left, I have all the 4 men tablebases but only use one 5 men tablebases- KRKRP. With White to play, instead of queening the pawn with e8=Q, Comet will instead play1 Re4+ ?

1 Re4+ appears superior to 1 e8=Q because from the program's point of view 1 Re4+ leads to a position that is a mate in 16 [obtained from the KRPKP tablebase], while it is unable to search deep enough to see that 1 e8=Q leads to a even faster mate in 15 due to the lack of KQRKP tablebase! As such it prefers a certain mate in 16 to a move that "merely" leads to a high score.
r7/2K1P3/8/4R3/2k5/8/8/8 w - - 0 1

White to move [Picture generated using epdtodiag]

A similar problem can result if you download only the KQPKQ tablebase without KQQKQ , KQRKQ etc..

Some programs like Crafty and Yace are "smart" enough to avoid this problem, but some like Fritz,Tiger,Amy,Comet,Gandalf or The Crazy Bishop[all freely available Winboard engines except for Fritz and Tiger of course] cannot handle this.



noproblem

On the other hand, some positions are not problematic, if there is a quick mate nearby after the pawn promotion that the Chess engine can see.

There is a short mate in 5 after 1 f8=Q+ . As such Comet and most Chess engines * can quickly see this shorter mate and will prefer the pawn promotion move to 1 Ke7 [found in the KRKP tablebase] which leads to a mate in 9.

This is made easier by the availability of the 4 men tablebases as the program need not calculate the position after 1 f8=Q+ Rxf8+ etc since it can use the KRKR etc tablebases. In fact, the only other legal move [besides Rxf8+] after 1 f8=Q+ is Kh7 which leads even to a quicker mate. All this is child's play for Chess programs to calculate

Compare this with the above position, White has 15 other replies after the pawn promotion which doesn't lead to a 4 men tb position.
4K2k/5P2/8/6R1/8/5r2/8/8 w - - 0 1

White to move [Picture generated using epdtodiag]

If you are an author of a Chess program and wish to avoid this problem, what you can do is this. If you find that in the root position it is mate in X from probing the n men tablebases, but after trying every legal move you don't find a mate in X-1 from the tablebase, it must be due to a incomplete tablebase problem and you search without probing the tablebases.

Also take note that not all programs support all the Tablebases. Yace for example, currently does not use the 4+1 tablebases [ King and 3 pieces/pawns on a side versus a alone King] , because such positions are easily won most of the time barring rare cases [Like double rook pawns, wrong coloured bishop and King vers King].

You might also not want to download 6 man Nalimov endgame tablebase (see also here) because I think currently only a few Chess programs support them, although it's usually a simple matter for the programmers to activate them. These include Crafty, Sjeng , Quark , King of Kings[all free Winboard Chess engines ] and perhaps Rebel Tiger . NEW! I have read that the Fritz6 engine itself does not probe the 6 men tablebases, but only uses them when they occur on the board.

* It appears that The Crazy Bishop might just stop searching whenever it finds a forced mate and as such it is "happy" with 1 Ke7 . Which explains why it has problems here, as a mate in 5 is well within it's search abilities.


[A.10] Questions about 6 men tablebases.Which are the most useful?

Generating Nalimov 6 piece tablebases requires the heavy duty use of 64 bit computers and is currently out of reach of most normal users.

As of Jan 2004, all the 3-3 and 4-2 pawnless Nalimov tablebases are done (227 GB). The 4-2 with pawns are being worked on, and some are available. None of the 3-3 with pawns or the 5-1 nalimov tablebases are available yet.

As of Aug 2005, all of the 6 piece tablebases are complete except for the 5-1 set but not all are available publicly.

There appears to be a lot of confusion about which engines can access 6 piece tablebases. This is due to several changes in the nature of the 6 piece tablebases. First there is the "normal" 6 piece tablebases generated before Dec 2002, it is suspected that some of the newer commercial engines like Hiarcs 9 can access this. In Dec 2002, because of the size of the files, each tablebase is broken into 2 parts, i,e "the multifile format". In September 2003, yet another batch of tablebases was released with more changes to the probe code.

Crafty V19.4+ (to be safe use versions above 20) , Wilhelm are confirmed to work with them. Also confirmed working (ability to use in search) according to http://kd.lab.nig.ac.jp/chess/tablebases-online/ are Chess Tiger 15.0, Crafty 19.19, Fruit 2.2.1, Gandalf 6.0, Rybka 1.1, Shredder 9. This refers to the larger multiple split set, normal chess engines can probably handle the older and smaller 6 piece tablebase sets.

Where to get them.

Historically the main source for 6 men Nalimov tablebases has being at Bob Hyatt's ftp . However, as of April 2006, they are still recovering from a hard-disk crash, and there are no 6 piece tablebase there. They intend to recover though, so continue checking.

http://olympuschess.com/tb6/index.html has a very limited number of 6 piece tablebase sets for download, however, the best, most complete and most reliable source for downloading 6 pieces Nalimov tablebases is still via emule here!.

What are the most useful 6 piece tablebases?

It's hard to say, but take a look at the following recommendations based on an analysis of positions reached by humans and computers.. Data about difficulty of winning (probably of draw) and other stats can also be found here in excel format.

Historical information (outdated mostly)

NEW! [Dec 14 2002] - More 6 men tablebases have being uploaded on Hyatt's site. However because of the size of the files (greater than 2 gb) they are broken into 2 parts (For example krnkbn.2.nbw.emd) and it requires a different probing code and currently [Dec 14 2002] no Chess engine can access it yet except Crafty 19.2 which will be released soon.From the user point of view, usage remains the same. As of March 2003, the new version of Wilhem can access them too.

NEW![September 2003] Yet more endgame tablebases produced with more changes to the probing code. This current update includes some of the 4+2 with pawns tablebases, but very few programs can access it. One is wilhelm, the other is Crafty 19.4. Any programs released before them (around Oct 03) are unlikely to be able to use such tablebases.

Here's an explanation by Eugene Nalimov.

New access code would shortly be released in Crafty 19.2. Changes were necessary because (1) you should use 64-bit index to index some of 6-men TBs, (2) even after compression some files were greater than 2Gb, and some OSes cannot handle such files (or handle them in absolutely unportable way), so best solution is to split those files into several smaller ones.

From chess program's point of view the only difference is that INDEX type now should be unsigned 64-bit integer, not unsigned 32-bit integer. Something like 5 minutes of work.

Some unrelated problems are fixed in the code -- reported memory leaks, SMP problems, Mac OS X locks problem that manifested itself in access violation, very slow startup when TBPATH pointed to the CD-ROM drive and there was no disk in that drive, better integrity check for the TBs, etc.

[Eugene Nalimov, CCC posting Dec 13,2002]

The pawnless six men endgame tablebases are almost complete (except for the trivial ones). This is necessary of course to avoid the incomplete endgame tablebase problem (although you can use them to see the results without the best line of play), but it also makes them pretty useless and may actually slow down the Chess engine if it probes useless positions.

Of some interest is the KQQKQQ tablebase. It was one of the first 6 piece tablebase generated back in 1999 especially, to aid in the Kasparov versus World game.

It is estimated a whole set of 6 piece Nalimov tablebases (when ready) will take up at least 1 terrabye (1000 GB).

Unlike the nalimov tablebases it's possible to generate 6 piece Chessmaster tablebases using a normal 32 bit desktop and 512 ram.It has being reported that Marc Bourzutschky, has managed to use Chessmaster 9000's FEG to generate 6 piece tablebases including the KRPPKR in about a month using 3 desktop computers (1.9GHZ and 2.4GHZ P4, and an an old 800MHZ P3). He has completed the above plus ,KQPKQP,KRPKBP,KRPKNP,KPPPKR,KPPPKB,KQPPKQ,KRPPKQ and supporting tablebases, and the size is about 0.5 Terabytes.

In addition certain 7 piece tablebases were completed.See here.


[A.11] How can I find out which tablebase is corrupted? How do I know the endgame tablebases are working?

If you download or generate your own tablebases, there is a possibility that some of them become corrupted. Some chess playing programs do a check and warn you if any of the tablebases are corrupted, but many don't. /p>

In practice though, checking if endgame tablebases are working in general for a chess program is usually a easy affair. Just setup a position, cover by the tablebases, and set the Chess engine to analyze or move. If everything is done correctly, the engine will move instantly, or come up with a instant evaluation score that will not change. Usually, the thinking lines will indicate that the score comes from the endgame tablebase. E.g Crafty will display a <EGTB> in the principle variation.

The above method is fine, except that it will take a long time to check all the tablebase files one by one. Here is what I recommend.

In general, when setting up tablebases, there are 2 possible errors.

  1. The interface is configured incorrectly. Generally this occurs when the path to the tablebase is in error.
  2. The tablebase files are corrupted.

The procedure above checks for the first problem. If you manage to get one tablebase working chances are the path is set correctly or the other tablebase files in the same directly should work, if they are not corrupted.

Here are 3 ways to check for corrupted files.


[A.12] What is datacomp.exe? Where can I get it?

Datacomp.exe is a utility provided by Eugene Nalimov and it's mostly used to compress Nalimov tablebases after you have generated them. Datacomp.exe can be used in reverse as well to uncompress and can be used to verify the integrity of existing tablebases, which is useful to check for errors during download.

You can get it at Dann Corbit's ftp site at http://cap.connx.com/chess-engines/new-approach/TBGEN.ZIP.The files, tbgeni.exe, tbgenm.exe that you get from there are exactly the same except that they have being complied using different compilers. One or the other might be faster for you.

To use datacomp to compress , use the following format:

datacomp e:8192 knk.nbb

Note this allows you to generate only up to 5 piece tablebases. It is not advisable to try for 6 piece tablebases anyway.


[A.13] How do I get Crafty to work with Endgame tablebases?

You need to edit the Crafty.rc file. For example, my line reads tbpath=c:\chesseng\tb . You should edit the line to point to the directory where your endgame tablebase files are currently located. The procedure for setting up endgame tablebases is similar for other Winboard engines that you can download. See my other FAQ on Winboard and Chess engines


[A.14] How can I get Fritz to use Endgame tablebases?

The advice below applies for users of Fritz 6. Users of newer chessbase software like Fritz 7 etc, have the additional option of setting up the tablebase path within the interface.

First, copy all your tablebase files into one directory, say C:\Nalimov

Next locate the file chssbase.ini [Not Chessbase.ini!] . It is usually in your Windows directory. Part of the file is something like this

[Tablebase]

Path=C:\Nalimov

CacheSizeKB=1024

Just edit the path to point to the directory where your endgame tablebase files are currently located and remember to save.This should work for all Chessbase engines including Junior,Nimzo,Hiracs and the new Chessbase engines Shredder and Tiger. For further information, refer to Steve Lopez's T-notes on"Tablebases and Fritz endgame Tablebase", 2 July 2000 and "Using Endgame CDs in Chessbase 7", 21 March 1999

Please note that if the endgame tablebases that you use are corrupted in some way (perhaps through a incomplete download) Fritz/Junior etc will refuse to run. To find out which tablebase is corrupted see Section A.11


[A.15] What is the difference between tablebases download from Dr Hyatt's ftp site and those on the Chessbase endgame turbo CD? Can they be used together?

Generally the tablebases that come in the Chessbase Endgame turbo are in the format kxxkxxnbb.emd compared to that downloaded, which come in the form kxxkxx.nbb.emd .Otherwise I think they should be exactly the same. I have generally found that Fritz can use either form, but you may want to add or remove the dot, if it refuses to work.

Also kxxkxx_nbb.emd should work as well.

In general Nalimov compressed tablebases bought from other commercial companies are also interchangeable with the downloaded ones.


[A.16] What are the files ending with .tbs? What about those ending with nbb and nbw? Do I need both?

Tbs stands for tablebase summary and give stats on the number of positions which are won, losses etc. They are not necessary if you are not interested in such statistics.

Earlier versions of Nalimov's code need the .nbw and .nbb of symmetrical endgames. Later this was changed (when support of 6-men TBs was added). So, engines using the older code will need both, and newer code will only need one. The list of the symmetrical TBs in the code are knkn, kbkb, krkr, kqkq, knnknn, kbbkbb, krrkrr, kqqkqq but not kpkp

There is new Eugene Nalimov code that allows you to use only one nbw file in all cases except when there are symmetrical positions for example in KRKR positions. This may or may not work depending on the Chess engine and probing code used. For access to 6 men tablebase positions, the new Nalimov code but be used, however for symmetric positions like knkn, kbkb, krkr, kqkq, knnknn, kbbkbb, krrkrr, kqqkqq (but it seems not Kpkp) you need both nbb and nbw.

To be safe though it's best to have both (or download nbw first and see if it works), as currently it's not certain how many Chess engines [besides Crafty] use the new Eugene code, and you can't really save much space anyway for the symmetric 4 men tablebases because they are pretty small anyway.

This is unlike the case of Edwards tablebases or Chessmaster's EGDB , where you need only either the file ending with tbb [positions with black to move for Edwards] or tbw [positions with white to move]. But even for those, it's much faster to have both since with only one set, the engine has to do a 2 ply search instead of 1.


[A.17] Help, the endgame tablebases are not working properly!

There are some possible reasons

  1. You specified the wrong pathway for the endgame tablebases.

  2. File is corrupted or in the wrong form , they should be in the form kxxkxx.nbw.emd or kxxkxx.nbb.emd [for compressed Nalimov tablebases]. See Section [A.11]

  3. One of the files either kxxkxx.nbw.emd or kxxkxx.nbb.emd is missing. You need both of them.

  4. You are using the wrong type of endgame table bases or the Chess program does not support the use of endgame tablebases.

To check whether, the endgame tablebase is corrupted (perhaps during download) , you can check the file sizes with the ones available online or use datacomp.exe (which is part of the tbgen) to check. The command you need is datacomp t kxxkxx.emdwhere kxxkxx is the file you want to check. This can be very tedious if you want to check all the endgame tablebase files, so it might be a good idea to use a batch file.

If the Program for some reason refuses to make a obvious winning move [usually queening] in a endgame, it is possible, we have a case of incomplete tablebases. See Section A.9 .


[A.18] Can I use tablebase files in zipped form?

No. Nalimov tablebases in the emd format are already compressed, and there is little if any gain from further zipping them up.Same goes for the Chessmaster EDGB format.


[A.19] Where can I get a useful graphical browser to view and practice endgame tablebases?What about an online searchable database?

Let's do them in reverse order. The following websites allow you to search the results of a given endgame position, if you don't have the tablebase files.Most sites below require Java or at least javascript.

What if you already have the tablebases but need an application to view and search through them?


[A.20] Misc Questions

Will we ever have 32 piece tablebases?

To be worked on

Why don't we allow chess engines to access and use endgame tablebases over the internet?

To be worked on

What about doing a distributed peer to peer method of generating endgame tablebase?

To be worked on


Aaron Tay