Go
19×19 Go, Chinese rules, komi 7.5, given as SGF. KataGo scores every legal move and pass by win rate; the answer is a GTP coordinate. See Methodology.
Fewer puzzles than the other models, so results are noisy.
Quality is the mean oracle score of the moves played, × 100. The best move scores 100; an illegal move or no move scores 0. Protocol boardbench-0.2.
Quality vs cost
4 of 4 models
- Anthropic
- OpenAI
- Pareto frontier: nothing plotted is both cheaper and better
- Anthropic
- OpenAI
Horizontal axis is logarithmic.
Table
| Model | Quality | Legality | Cost | n |
|---|---|---|---|---|
| Luna highMedium | 44.5 | 100.0 | <$0.01 | 9 |
| Sonnet 5.5 (fewer samples)Medium | 38.7 | 100.0 | $0.01 | 3 |
| Sol (fewer samples)Medium | 34.4 | 100.0 | $0.02 | 3 |
| Haiku 5.5Medium | 33.4 | 100.0 | <$0.01 | 9 |
Headline is the mean move score × 100. n is answers: puzzles × samples.