Methodology

Updated

What BoardBench measures

On BoardBench, models solve puzzles in various popular board games and are scored on the quality of their results.

The puzzles are newly generated, so models are unlikely to have seen them. They are also designed to be hard: in most positions, only one or a few moves work — the others give up a lot.

How puzzles are made

Engines play themselves under the standard rules. Early moves are sampled at random so the games leave known theory. A position is kept only if one move is clearly best and moves found using quick search are bad.

Each position is looked up in public game databases and puzzle collections. Kalah, Connect Four, Scrabble, and backgammon have no such database, so for those games novelty rests on the positions being random.

The final three per game are chosen for variety.

Input and output

The model sees the game, the side to move, and the board drawn in that game's official notation, with an example of how to write the move. Chess is algebraic (Nf3, O-O), Go is the GTP board (Q16), shogi is a grid of SFEN letters (7g7f), and the Scrabble board marks its premium squares, with a blank as a lowercase letter (8H jUMBO). A second encoding of the same position is not shown: not the SGF next to the Go board, and not a CSA dump next to the shogi board. The legal moves are not listed.

The reply's last line is parsed for the move.

How scoring works

Each legal move is scored by an engine from 0 to 1.

Any illegal move scores 0.

Quality is the mean score of a model's answers × 100.

Overall is the mean over every answer in every game.

Legality is the percentage of answers which are on the puzzle's list of legal moves.

Cost is the list price of one answer.

Official games

GameSolverCriteria
XiangqiPikafishExpected score
ChessStockfish 19Expected score
GoKataGoWin chance
BackgammonGNU BackgammonExpected points
ShogiYaneuraOuWin chance
Scrabblewolges simulationExpected points
KalahExact solverFinal margin
GomokuRapfiWin chance
OthelloEdax, exactFinal margin
Connect FourExact solverWin, draw or loss, and how fast
Look