Jide Oyelayo
CV

Work / 1.4 Poker engine and self-play agents

Case study 1.4 · ML · Quant

Poker engine and self-play agents

A C++20 Texas hold’em engine bound to Python, on which Deep Q-Network agents learn by self-play; the hand evaluator is exhaustively tested, and the agents do not yet beat fixed baselines.

Date
Jan 2026
Role
Author
Stack
C++20, Python, PyTorch, pybind11

Status: draft. Certainty: The evaluator, betting, throughput and win-rate figures were measured on one shared Apple M1 Pro with the tests and benchmarks in pull request #1, after 2,000 training hands per seed, 2–4% of the README’s budget. The README’s figures (Figs. 1 and 2) are reported in the repository; not re-run here, and the logging behind them was biased.

Summary

This project is a simplified Texas hold’em engine written in C++20 and a set of Deep Q-Network (DQN) agents, written in Python with PyTorch, that learn to play it against one another with no poker strategy given. The engine is bound to Python with pybind11, so each agent controls one seat inside the C++ simulation. After training, symbolic regression writes the agents’ valuation of starting hands as short formulas. The code is in PokerRLPlayground.

The repository’s main branch has one commit, no tests and no recorded runs. Pull request #1 adds tests and benchmarks. The hand evaluator is correct over all 133,784,560 seven-card hands. The engine had a betting-round bug, now fixed, and the training log was biased in ways that affect the README’s figures. At 2,000 training hands per seed, no agent beats a fixed baseline reliably. The page stays a draft because the README’s figures have not been regenerated.

Context and constraints

Poker hides information. In this engine each agent sees only its own two hole cards and the public state: community cards, pot, amount to call, stacks and street. The README states that no opponent’s private information is exposed. The reward is the only teacher.

The limits below come from the README and the code; the measured rows are from pull request #1.

Operating limits
LimitValue
GameSimplified Texas hold’em: a shuffled 52-card deck, two private cards each, the usual betting rounds, blinds rotating one seat per hand. The small blind is posted by the seat after the big blind and acts last on every street; heads-up, the big blind acts first. This is not standard hold’em
ActionsFold, check, call, all-in, and raises to the current bet plus one big blind, half the pot or the full pot. After any raise the one-big-blind size is below the minimum raise, so it is replaced by another size or action
Seats and settingsOne agent per seat; the README’s example seats four models with stacks of 10,000, blinds 1/2 and MAX_HANDS = 10
Training cost0.62–0.82 CPU-seconds per hand for the four models on CPU (seeds 1–3), so about 5–7 CPU-hours for 30,000 hands (extrapolated); the README reports several hundred
Hand-evaluator correctnessCorrect: no disagreement with an independent reference over all 133,784,560 seven-card and 2,598,960 five-card hands; category counts match the published tables
Engine throughput42,374 hands/s [40,554–42,650] through the C++ step() API with every seat calling, 12,950 [12,883–13,194] through the Python binding; median of 7 runs [min–max], 4 seats, one thread, -O3
Win rate against fixed baselinesNo agent beats one reliably (table below)
Seeds and training logsRecorded for the runs on this page; a rerun of seed 2 matched the first for 114 hands, until a 2-second wall-clock timeout. The README’s figures still have none

My role

The repository is mine: I wrote the engine, the bindings, the agents and the analysis, and published them in a single commit on 29 January 2026. The tests, benchmarks and fixes on this page are in pull request #1. The preflop tables in theoretical_data/ (for two to nine players) do not cite a source, so their origin is to confirm.

Architecture

The repository has three parts: the C++20 engine and its pybind11 bindings in PokerEngine/; the agent, trainer and self-play loop in ReinforcementLearningEnvironment/; and four Q-networks in Models/, a multi-layer perceptron, an LSTM, a GRU and a Transformer encoder. main.ipynb drives training and symbolic_regression.ipynb the analysis.

One hand is one episode. The state is a fixed-length vector built from the acting player’s view, and the action is one of seven: fold, check, call, three raise sizes and all-in. Illegal actions are masked. The reward is zero during the hand and, at the end, the player’s profit or loss in big blinds. Each seat is its own agent and always plays the current versions of the others. Every hand is logged as one row per player, with the starting hand, the profit in big blinds and whether the player folded; the README computes its metrics from these logs.

Key decisions

A C++ engine with Python learners. The README gives performance and determinism as the reasons for the C++ engine. It was not deterministic, since the deck was seeded from the system clock; pull request #1 adds a seed. Nor is the pybind11 crossing the significant overhead the README lists: the engine, binding and state encoding take about 0.55 ms of a training hand of roughly 300 ms, and the networks take the rest.

Three raise sizes. Raises are limited to three sizes to keep the state–action space small and to steady learning. The cost, which the README states, is realism: the agents cannot express finer bet sizing.

A sparse reward in big blinds. Rewarding only the end-of-hand profit aims the agents at expected value over time. The cost is credit assignment: the reward of a player who did not act last is stored with no state, and training drops it, so only the player whose action ends a hand is ever trained on a non-zero reward. Pull request #1 corrects the README’s claim that every player gets a learning signal, not the code; the agents measured here were trained with this fault.

Small networks of four kinds. The README keeps the models small, since larger ones can overfit to the current opponents and simulation is already expensive. Comparing four families is meant to show which results do not depend on the architecture.

Symbolic regression for interpretation. A regression over four features (the two ranks, whether the hand is a pair, whether it is suited) searches for short formulas for each hand’s value, as a readable check against poker theory.

Evaluation and results

The measurements are in bench/RESULTS.md, with raw output in bench/raw/. The agents were trained together with the repository’s own loop, as main.ipynb sets it up. Each greedy agent then played three copies of a baseline bot, with stacks reset every hand, on duplicate deals: each deal is replayed with the agent in every seat, so an agent that plays exactly like the baseline scores 0.

Agentvs randomvs callvs tight
MLP−4,224 [−19,417, 10,969]0 [0, 0]−46,064 [−140,905, 48,776]
LSTM12,695 [−2,094, 27,483]4 [−17, 25]−17,873 [−26,974, −8,772]
GRU14,310 [−9,792, 38,412]−21 [−68, 27]−29,081 [−119,653, 61,491]
Transformer0 [−38,329, 38,329]5 [−15, 24]−30,795 [−114,058, 52,468]
Random bot (control)3,150 [−13,441, 19,741]2,878 [−15,862, 21,618]−31,501 [−37,351, −25,651]

Big blinds per 100 hands, mean of three seeds with a 95% t-interval over seeds (per-seed table). The random bot picks any legal action; the calling bot calls or checks; the tight bot raises the pot with TT+, AK and AQ, calls with other pairs, two broadway cards or suited aces, and after the flop continues only with a pair.

No trained agent beats any baseline reliably. Every interval against the random and calling bots contains zero. Every agent loses to the tight bot in every seed; only the LSTM’s interval excludes zero, and it is a loss. Against the calling bot, 7 of the 12 seed–agent pairs score exactly 0, and seed 3’s GRU folded every hand. Single seeds beat the random bot, but none holds across seeds. The values run to tens of thousands because stacks are 5,000 big blinds deep and most policies collapse towards moving all-in: late in training, 6 of the 10 seats still being dealt in opened all-in in 53–100% of their hands.

The rest of this section is reported in the repository; not re-run here. The README compares the learned values of the 169 starting hands with a theoretical preflop table: at 50,000 hands pocket queens rank above pocket aces, and by 100,000 hands pocket aces have the highest value (Fig. 1). The value of each tracked hand swings early in training and flattens later (Fig. 2).

The training log behind those figures was biased. Folded players were never logged, so each starting hand’s value was averaged only over the hands where the player did not fold; in the new logs, 8.6–48.3% of rows are folds. Stacks were read after the blinds, so each hand’s results summed to +1.5 big blinds instead of zero. Pull request #1 fixes the logging; the figures have not been regenerated.

Heatmap of mean expected value in big blinds per hand for every pair of card ranks, 2 to ace on both axes. The pocket-aces cell is the brightest, near 5,000; pocket kings and pocket queens come next, near 4,000.
Fig. 1 Expected value by starting hand after 100,000 hands Mean expected value in big blinds per hand for each pair of card ranks, from images/expected_value_of_pocketcards_100000_iter.png in the repository. Reported in the repository; not re-run here. The logging behind it dropped folded hands and the blinds.
Line chart of cumulative average big blinds per hand against hands played, up to 30,000, for seven starting hands. The lines swing widely in the first few thousand hands, then flatten: AA highest near 5,000, KK and QQ near 4,000, TT near 2,000, KQo and AKs between 1,500 and 2,000, and 72o below zero near −1,000.
Fig. 2 Cumulative average expected value for seven starting hands Cumulative average big blinds per hand over training for AA, KK, QQ, TT, AKs, KQo and 72o, from images/ev_curves_by_pocket_cum_avg.png in the repository. Its axis runs to 30,000 hands; the README does not say which run produced it. Reported in the repository; not re-run here.

Method what is measured, and how

Question
Do agents trained only by self-play learn a ranking of starting hands that agrees with preflop theory, and do they beat fixed baseline bots?
Workload
2,000 self-play training hands per seed, then 1,000 duplicate deals (4,000 hands) per agent and baseline at a four-seat table, stacks 10,000, blinds 1/2
Controls
Seeds 1–3 fix Python, NumPy, torch and the deals; training repeats only until the first wall-clock timeout
Environment
Apple M1 Pro, macOS 27.0.1, shared; Apple clang 21.0.0, -O3; Python 3.14.6, torch 2.9.1 on CPU
Reported
Big blinds per 100 hands, mean of 3 seeds with a 95% t-interval over seeds
Reproduce
sh bench/run_benchmarks.sh winrate (42 minutes); the README’s figures: not reproducible
Not claimed
That any agent beats a baseline, a Nash equilibrium, low exploitability or robustness against unseen opponents (the README’s own exclusions), or that the README’s figures were re-run.

What went wrong and limits

The engine had a betting-round bug. A street closed only when the action returned to one marker seat with all bets matched; if that seat had folded or was all-in, the street never closed. On the old engine, depending on the mix of policies, between 38 and 2,480 of 5,000 test hands did not finish within the training loop’s 2,000-step cap, 22 reached a player with no legal action and 12 destroyed chips. The training loop silently aborted and reset on each stuck hand. Pull request #1 fixes the bug, and its test passes all 124 checks.

The README’s results are not reproducible from the repository. Its main branch has no tests, a single commit, and no committed seeds or training logs, and the committed training call sets MAX_HANDS = 10. The README says the results are reproducible across architectures and training runs without recording any runs. PySR, which the analysis imports, is not in requirements.txt. Even with seeds, training repeats only until the first 2-second timeout.

Some faults remain. Only the player whose action ends a hand is trained on its reward. The legal-action mask marks all three raise sizes legal whenever any raise is legal, so a chosen size can be swapped silently. A busted agent sits out until the tournament resets; in seed 1 the last 500 hands were heads-up.

Random exploration stops after 80 hands (max(0, 80 − hands) in Agent.py). The README’s best values, near 5,000 big blinds per hand, are a whole starting stack, which matches the all-in play measured above. Fig. 2 does not match the README’s 50,000-hand result of queens above aces: in it, aces lead queens from the first few thousand hands.

Next

Regenerate the README’s figures with the corrected engine and logging, from one command with recorded seeds. Fix the reward for players who did not act last, remove the wall-clock timeout, then train well beyond 2,000 hands and measure win rate again. The draft label goes when the figures on this page come from runs that can be reproduced.