Jide Oyelayo
CV

Work / 1.5 Catan engine, search bots and learning agents

Case study 1.5 · ML

Catan engine, search bots and learning agents

A C++20 engine for four-player Catan with eleven learning agents and two search bots, measured against a hand-written bot. Search beat it; learning did not, and the best learner drew level with it only in a tournament.

Date
Nov 2026
Role
Author
Stack
C++20, Python, PyTorch, pybind11, neat-python, GoogleTest
Code
Not public yet: the repository is private

Status: published. Certainty: Every measured number on this page is printed in the saved outputs of the study’s notebook, report/report.ipynb, or read from the code, at CatanAIPlayground commit b673106 of 19 November 2026; the write-up, report/REPORT.md, agrees wherever it gives the same values, except in four places, which are left out. Nothing was re-run for this page. The repository is private, so readers cannot check the source yet, and the study continues. The agents were measured against bots only, never against people.

Summary

CatanAIPlayground asks which approach plays the four-player board game Catan best. It is a C++20 rules engine bound to Python with pybind11, eleven learning agents in PyTorch and neat-python, two search bots, a hand-written heuristic bot, a browser viewer and a report. The project is not affiliated with the game’s publisher.

Search beat the heuristic bot; learning did not. With every trained seed on the same 416 deals, one seat against three copies of the heuristic bot, where an equal player wins 25%, the tree search bot won 36.3% [31.7, 41.1] and the lookahead bot 35.1% [30.8, 39.7]. The heuristic bot itself scored 25.7% [21.6, 30.0]. Neither search bot is trained, and both lean on the heuristic bot’s knowledge: without its opening and the hand-written position score they win almost nothing.

The best learner, PPO with a network shaped like the board, won 21.4% [17.8, 25.5], and drew level with the heuristic bot in a nine-entry tournament (29.8% against 30.9%). Plain PPO won 7.9%. The network-guided search agent did not learn in the budget it had, and an earlier study of five learners found none that beat the heuristic bot. The numbers are as of 19 November 2026; the study continues.

Context and constraints

Players build on a board of resource tiles that produce on dice rolls, trade with the bank and each other, and race to 10 victory points; hands and development cards are hidden.

Operating limits
LimitValue
RulesThe full four-player base game, including the robber, all five development cards, ports, bank trades, longest road and largest army; a draw after 500 turns
Player tradesStructured so agents can make them: the player to move offers a one-for-one, two-for-one or one-for-two swap, the others answer in turn, the first to accept takes it; at most three offers a turn
What an agent sees1,022 numbers from its own seat’s view, with seats relative to it so one network can play any seat; one of 314 actions, illegal ones masked
Hidden cardsThe search bots never see them: before searching they re-deal the other hands and the development deck at random, keeping every public fact. The write-up calls this an approximation
Training budget100,000 games per run and three seeds per agent; the clone had 10,000 games and the network-guided search agent 8,000. The long runs used one seed
MachineOne machine, CPU only, shared by overlapping runs and heavily loaded by another project, so recorded training times are too high and serve only to compare agents
Tests57 GoogleTest cases, a pytest suite and fuzz tests that check the rule invariants while bots play; the write-up says they pass; not re-run for this page

My role

The repository and the study are mine: the engine, bindings, agents, experiment script, notebook and write-up, committed between 17 and 19 November 2026. The repository is private and the code is not public yet, so it is not linked; this page quotes its saved results.

Architecture

engine/ holds the rules, the observation encoding, the random and heuristic bots, both search bots and a batched environment. The environment steps many games at once and plays every bot seat inside its step, so Python sees only a learner’s decisions. Each game slot gets the same deals for a given seed whatever was played before, which makes evaluations paired. bindings/ is one pybind11 module. catan_ai/ has the agents, training, tournament, benchmark, diagnostics, replay export and a local play server; viewer/index.html replays games.

In training the learner holds two seats, with a heuristic bot and a random bot in the others. The reward is 1 for a win plus 0.05 for each victory point gained, minus 0.05 for each lost.

The first study’s learners are plain policy gradient with a value baseline, PPO, PPO with an LSTM, double DQN and NEAT, which scores each legal move from 30 hand-made features. The second adds six agents built on ideas from agents for other games: PPO that leaves the robber, discards and trade answers to the heuristic rules (the hybrid); PPO against its own past checkpoints (the league); PPO with a network that passes messages between tiles, corners and edges (the board network); a supervised copy of the lookahead bot (the clone), and PPO started from it (clone-then-PPO); and tree search guided by a network trained on its own search results, at 48 simulations a move (the network-guided search agent).

The heuristic bot takes the best production spots, builds cities before settlements before roads, and trades only towards its next build. The lookahead bot tries every legal move, plays out the rest of its turn to depth 3 on 8 re-dealt copies of the game and scores the result with a hand-written value function. The tree search bot runs 800 simulations a move, each ending in two turns of heuristic play scored by the same function.

Key decisions

One yardstick. Every entry is judged in one seat against three heuristic bots, where an equal player wins 25%. Random bots are too easy: every first-study learner but DQN beat them in 90% of evaluation games within 10,000 training games; DQN took 75,000.

Every seed on the same deals. The report’s third part re-measures every trained seed on the same 416 deals, sampled and greedy, with 95% intervals that resample deals and seeds together, and differences against PPO taken deal by deal. The benchmark records the code commit and a fingerprint of each of its 33 model files.

Size the hand-written help. Switches in the engine turn off the heuristic opening, the heuristic first move, heuristic playouts and the position score one at a time, so the search bots’ debt to hand-written knowledge is measured rather than assumed.

Leave CFR out. The poker method was not tried: it is built for two-player games with short hands, which Catan is not.

Evaluation and results

equal player, 25%Tree search bot36.3Lookahead bot35.1Heuristic bot25.7Board network21.4Hybrid14.0Clone, then PPO13.8PPO7.9League6.8Recurrent PPO1.1Clone0.8DQN0.6010203040win rate, %
Fig. 1 Win rate against three heuristic bots, same 416 deals, every seed Each entry in one seat against three heuristic bots on the same 416 deals, playing as trained (the policy-gradient learners sample; DQN makes 2% random moves), with every training seed pooled; 95% intervals resample deals and seeds together. Drawn here from the benchmark table in report/report.ipynb at commit b673106. The plain policy-gradient agent is left out because the write-up and the notebook give different values for it. NEAT and the network-guided search agent were measured greedy only (9.1% and 9.7%) and are not drawn.

Search beats the heuristic bot; no learner does. The tree search and lookahead bots gain 28.4 and 27.2 points over PPO. Among learners, the board network gains 13.5 points [9.0, 18.1] and the hybrid 6.1 [2.0, 10.0]; clone-then-PPO is also clear of PPO, and the league is not (−1.1 [−4.5, 1.9]). The board network’s seeds range from 19.0% to 24.0%, and greedy play changes little for it (22.4%) or PPO (8.4%). The clone is the exception: 0.8% sampled, 6.1% greedy.

The board network’s weakness is probably not its opening: with the heuristic bot’s opening it scores 19.9%, within noise of its 21.4%. In its main phase a settlement was possible at 1.1% of its decisions, against 1.8% for PPO and 4.4% for the heuristic bot, and it built one 96% of the time it could. It rarely reaches a position where it can expand. It is also costly: about 28 microseconds per decision against 1.8 for PPO’s flat network.

equal player, 25%Tree search bot56.8Lookahead bot46.2Heuristic bot30.9Board network29.8Clone, then PPO15.5Hybrid12.5PPO12.0League11.8NEAT9.70102030405060win rate, %
Fig. 2 Second tournament: every table of four from nine entries Win rate of the seed-0 model of six learners, the heuristic bot and both search bots over 8,064 games, 3,584 per entry, none drawn, with ±1.96 standard errors as the notebook computes them; they treat games as independent, so the write-up calls such intervals slightly too narrow. Drawn here from the tournament table in report/report.ipynb at commit b673106.

The search bots’ strength is mostly hand-written. One seat against three heuristic bots, 400 games per variant:

VariantLookaheadTree search
Full37.8%38.0%
Random opening instead of the heuristic one3.5%6.8%
Position scored by points only2.0%16.8%
Heuristic move not tried first–35.2%
Random playouts instead of heuristic ones–29.0%
No hand-written help at all0.0%0.2%

So “search beats the heuristic bot” means search on top of that bot’s opening and a good hand-written score. No variant tested lets search stand on its own.

Training longer did not reach parity. One seed each, averaged over the checkpoints in each stretch, PPO went from 7.0% in its first 100,000 games to 17.2% between 750,000 and a million, and the hybrid from 11.0% to 18.3%. The board network’s long run is unfinished: 23.3% between 100,000 and 250,000 games and 25.2% after that, as far as it had got.

The network-guided search agent does not learn at this scale. Its search does not clearly improve on the network it is given. The repaired agent’s network alone wins 9.4%, and with 48, 200 and 800 simulations 12.5%, 11.7% and 9.4%; the clone’s network alone 7.8%, and with search 7.8%, 3.9% and 5.5%. The write-up reads these small samples as no gain, so the agent trains on search results no better than its own policy: its three seeds ended 8,000 games at 7.9% to 10.5%, having started at 7.0% to 9.0%. The write-up’s guess, which it has not tested, is that a tree over the 60 trade-offer actions is too wide for these budgets.

Method what is measured, and how

Question
Which approach beats a hand-written heuristic bot at four-player Catan, and by how much?
Workload
Every trained seed of each learner, and each bot, in one seat against three heuristic bots on the same 416 deals, sampled and greedy; a round-robin of nine seed-0 entries over 8,064 games; 400 games per search-bot variant
Controls
Paired deals for every entry; all seeds pooled; the benchmark records the code commit and each model file’s fingerprint; search bots re-deal hidden cards instead of reading them
Environment
One shared, loaded machine, CPU only
Reported
Win rate with a 95% interval resampling deals and seeds (benchmark), ±1.96 standard errors treating games as independent (tournament), and paired differences against PPO in points
Reproduce
scripts/run_experiments.sh 100000, then uv run python -m catan_ai.benchmark and the notebook; not possible for readers while the repository is private
Not claimed
That any agent plays well against people or another program, that learning beats the heuristic bot, that the search bots win by search alone, or any result at a larger budget.

What went wrong and limits

The network-guided search agent’s first version collapsed: its network wins 0.0% at every search budget tested. The repair stopped the collapse but not the stall above. The clone agrees with its teacher’s move on 49.1% of positions from the teacher’s games and 44.5% from its own, and wins under 1% when it samples. In two single-seed league runs the final model beats its untrained first snapshot in every game and wins 44% and 47% against the newest, so the league improves against its past selves, but not against the heuristic bot.

The write-up and the notebook disagree in four places: the network-guided search agent’s row in the second study’s learner table, the plain policy-gradient agent’s benchmark win rate, clone-then-PPO’s paired gain over PPO, and one of the clone’s diagnostics. Those values are left off this page.

The limits stand. Three seeds per agent, one for the long runs, and seed-0 models only in the tournaments; differences of two or three points between learners are within noise. Budgets are small, and PPO was still creeping up at a million games. NEAT reads hand-made features, so it is not a like-for-like comparison of learning methods. No human games have been recorded, there is no external bot to compare with, and everything ran on one loaded machine.

Next

From the write-up: finish the board network’s million-game run, train it longer and try it as the network inside the search; run the tournaments with every seed; and measure the agents against people and another program.