Othello

8×8 Othello at 20 empties. Edax solves every legal move exactly by final disc difference; the answer is a cell. See Methodology.

Quality is the mean oracle score of the moves played, × 100. The best move scores 100; an illegal move or no move scores 0. Protocol boardbench-0.2.

Quality vs cost

8 of 8 models
  • Anthropic
  • OpenAI
  • DeepSeek
  • Z.ai
  • Google
1020304050607080$0.001$0.01$0.1$1Claude Haiku 5.5 High · 2.7% · <$0.01 · Pareto frontierClaude Opus 5.5 High · 0.4% · $0.04DeepSeek V4.1 Flash High · 34.5% · $0.04Gemini 4 Argon High · 55.1% · $0.23GLM-5.3 Flash High · 37.8% · $0.02GPT-6 Astra High · 55.1% · $0.62GPT-6 Luna High · 55.1% · <$0.01 · Pareto frontierGPT-6.1 Sol High · 55.1% · $0.10GPT-6 Luna HighGPT-6.1 Sol HighGemini 4 Argon HighGPT-6 Astra HighGLM-5.3 Flash HighDeepSeek V4.1 Flash HighClaude Haiku 5.5 HighClaude Opus 5.5 High
  • Pareto frontier: nothing plotted is both cheaper and better
  • Anthropic
  • OpenAI
  • DeepSeek
  • Z.ai
  • Google

Horizontal axis is logarithmic.

Table

ModelQualityLegalityCostn
Gemini 4 ArgonHigh55.1100.0$0.233
AstraHigh55.1100.0$0.623
LunaHigh55.1100.0<$0.013
SolHigh55.1100.0$0.103
GLM-5.3 FlashHigh37.8100.0$0.023
DeepSeek V4.1 FlashHigh34.5100.0$0.043
Haiku 5.5High2.766.7<$0.013
Opus 5.5High0.4100.0$0.043

Headline is the mean move score × 100. n is answers: puzzles × samples.

Look