Engine strengths

You simulate different strengths by changing the opponent policy, not the rules and not the scoring oracle. The same engine family can sit at several rungs if you degrade how it chooses, in a way you can name, freeze, and reproduce from the match seed.

Separate three jobs

If you weaken the oracle when you weaken the table, exact_ideal_pct stops meaning the same thing across matches.

Four honest ways to weaken an engine

1. Shorter sight (search weaker)
Same evaluation, less look-ahead: depth, node budget, or time. Play is still purposeful, just myopic. This is the usual chess “level 1–20” story and the natural BoardBench reduced profile (Splendor already has a node budget; shrinking it is a strength knob).

Effect: consistently weaker, not human-like. A shallow engine blunders the same kind of long tactics every game.

2. Softer choice (Boltzmann / mix over ranked moves)
Score legal actions with the engine (or with one-ply equity), then sample:

\[
P(a) \propto \exp(\lambda \cdot v(a))
\]

High \(\lambda\): almost always plateau. Low \(\lambda\): more second-best and third-best. This is how self-play engines get “variety” without becoming uniform random.

Prefer this over uniform-among-legal. Uniform is a different agent; Boltzmann is “the same player, less decisive.”

3. Occasional slip (epsilon, regret-gated)
With probability \(1-p\), play an ideal move. With probability \(p\), play a worse legal move, gated on regret so you skip forced moves and skip dumping the king for no reason.

This is the seats-post “sometimes slip”: variance in quality, not a permanently dumber eval. Humans are closer to this than to a permanently 4-ply search.

4. Less knowledge
Drop tablebases, belief-set width (Scotland Yard detectives), or fancy leaf features. The engine is still maximizing, with a worse model of the position. Use this when the skill you care about is information / evaluation, not just depth.

Handicaps (odds, extra luck, worse start) change the game, not the player. Fine for casual play; they muddy a benchmark that wants comparable positions.

How to turn that into named strengths

Do not expose raw \(\lambda\) and \(p\) on the public protocol. Freeze a few profiles:

• strong — official opponent; full search; \(\lambda \to \infty\) or greedy plateau
• reduced — same eval, documented shallower budget
• club / strong_slip — full (or reduced) search + small \(p\), or a moderate \(\lambda\)
• random — uniform legal; floor / canary only

Calibrate rungs against the same engine, not against the LLM field:

• self-play: strong vs reduced should separate cleanly
• vs random: capture / win-rate floor (already a Scotland Yard canary)
• optional: mean regret of the weak policy vs the oracle, so “club” has a number, not a vibe

Once calibrated, freeze. Retuning so models look better is a protocol change.

Constraints that matter at a real table

Determinism. Mix, slips, and Boltzmann draws hash from match seed + ply + seat + the information that seat is allowed. Same seed ⇒ same weak player. Shared seeds across models still line up until the first different action.

Information. Weaken only with what that seat sees. Detectives must not slip using Mr. X’s hidden station. The oracle may still continue from the true state (privilege); the opponent must not.

Ideal set, not one string. When not slipping, pick from the equity plateau (tied best moves), not a single arbitrary engine_act string. Weakness is then “left the plateau,” which matches how you already score the model.

Don’t score the furniture. Weak engines are not leaderboard rows. Outcome vs a club engine is diagnostic; quality of the model seat is still vs strong oracle.

What to use when

Search-weaker and slip compose: a club player can be short-sighted and occasionally leave the plateau. Keep random out of that stack so you always know whether you degraded skill or replaced the agent.

That is the whole method: one classical engine, several frozen policies on how greedily it uses that engine, oracle untouched.