GomokuBench

GomokuBench is a lightweight benchmark for testing frontier LLMs against a search-powered Gomoku engine in a setting that is simple, adversarial, reproducible, and easy to inspect move by move. It also supports dual mode, where two LLMs play Gomoku against each other as Black and White with separate prompts, responses, and reasoning logs.

YouTube Demo

It is built for AI companies, model builders, and researchers who want a fast way to answer a practical question:

Can a general-purpose language model consistently beat a classical search algorithm in a fully specified board game?

Until 2026.4.26, no LLM in this benchmark has beaten the built-in AlphaBeta search engine. If you find one, please share it with us.

GomokuBench is designed to be easy to plug into model APIs, easy to run from the command line, and easy to audit after the game ends. Every move can be replayed, every prompt is explicit, and every result is saved as structured JSON.

Author: Homer Quan
GitHub: homerquan/GomokuBench

Why This Benchmark

• Simple game, hard reasoning: Gomoku has clear rules and no hidden information, so failures are easier to interpret.
• Search vs LLM: benchmark a deterministic AlphaBeta engine against modern chat models under the same rules.
• Fast iteration: add a new model with a small JSON config and start benchmarking right away.
• Useful outputs: save per-game logs, final boards, and aggregate win/loss results for later analysis.
• Good for demos and research: use it for model evals, prompting experiments, tool-use studies, and public scoreboards.

Current Leaderboard

As of 2026.9.20, these are the latest results from python -m gomoku report on the benchmarks/ directory.

Model	Wins	Losses	Draws	Avg Moves (Win)	Avg Moves (Loss)	Score
Gemini 3.8 Flash	0	10	0	0	14.5	2.9
GPT-5.5 (OpenRouter)	0	10	0	0	9.2	1.84
Nemotron 3.5 Lightning (Docker)	0	10	0	0	9.0	1.8
Nemotron 3 Super	0	10	0	0	8.0	1.6
Muse Glimmer 30B NVFP4 + DFlash on Spark	0	10	0	0	7.8	1.56
Bonsai 2 27B (llama.cpp Local)	0	10	0	0	7.1	1.42
Nemotron-Labs-Diffusion-14B	0	10	0	0	7.1	1.42
GPT-OSS 120B (Free)	0	10	0	0	7.0	1.4
Mistral Small 2603 (OpenRouter)	0	10	0	0	4.8	0.96
Mistral Medium 3.5 128B (NVIDIA, Fast)	0	10	0	0	4.7	0.94

Quick Start

Install from PyPI:

pip install gomokubench

If your Python install does not put console scripts on PATH, use the module form:

python -m gomoku play
python -m gomoku benchmark --model nemotron-3-super -r 10

Play against the engine:

gomoku play

Benchmark an LLM:

gomoku benchmark --model nemotron-3-super -r 10

Benchmark with a custom model config file:

gomoku benchmark --model-file ./my-model.json -r 10

Run two LLMs against each other:

gomoku dual --BLACK-LLM-FILE ./black-model.json --WHITE-LLM-FILE ./white-model.json -r 10

You can use the same model config for both sides; GomokuBench keeps their game prompts and reasoning logs separate. When -r is greater than 1, the first player alternates by round for fairer play: White starts round 1, Black starts round 2, and so on. Dual mode does not use --ai-level because no search engine is involved.

If you have spare credit to burn, try:

gomoku dual --WHITE-LLM-FILE models/claude-opus-4.7.json --BLACK-LLM-FILE models/gpt-5.5-pro.json -r 10 -v

Requirements

• Python 3
• numpy

Install

Or install from source:

pip install .

The package installs a gomoku console command. Some native Python installations place that command in a user scripts directory that is not on PATH, so gomoku may not be found immediately after pip install.

You can always run GomokuBench through Python instead:

python -m gomoku play
python -m gomoku benchmark --model nemotron-3-super -r 10

If python -m gomoku works but gomoku does not, add Python's scripts directory to PATH. On macOS and Linux this is often ~/.local/bin; on Windows it is often %APPDATA%\Python\Python3x\Scripts.

Play

gomoku play

Optional flags:

• --player black|white
• --ai-first
• --ai-level easy|standard|hard

The CLI always uses a 19x19 board.

AI levels:

• easy: shallower search with a small amount of move randomness
• standard: the default benchmark setting
• hard: deeper search

Moves use x,y with 1-based coordinates, for example 10,10.

Benchmark

Run an LLM against the built-in alpha-beta AI:

gomoku benchmark --model nemotron-3-super -r 10

You can also point directly at a custom model config JSON file:

gomoku benchmark --model-file ./my-model.json -r 10

To give the LLM a better chance, benchmark against a weaker engine:

gomoku benchmark --model nemotron-3-super -r 10 --ai-level easy

To watch the rounds play out in the console while benchmarking:

gomoku benchmark --model nemotron-3-super -r 10 -v

What this does:

• Loads the model config from models/nemotron-3-super.json
• Or, with --model-file, loads the model config from the JSON path you provide
• Runs 10 rounds total
• Uses balanced starts: 5 rounds with the AI moving first and 5 rounds with the LLM moving first
• Always uses a 19x19 board
• Uses the selected AI level, defaulting to standard
• -v prints each round, move, and board state in the console
• Saves the benchmark report to benchmarks/nemotron-3-super.json

The benchmark report is saved as JSON and includes the summary plus per-game move logs and final boards.

What Gets Saved

Each benchmark run saves a JSON report in benchmarks/ with:

• model name and provider
• board size and AI level
• total wins, losses, and draws
• which side moved first in each round
• full move logs
• final board states

That makes GomokuBench useful both as a quick CLI demo and as a small research harness for repeatable comparisons across model versions and providers.

Adding Models

This repo now includes a few example model configs in the models/ folder.

You can add another model by creating a new JSON config that uses an OpenAI-compatible chat completions API format.

See the models folder on GitHub for example config files.

API Keys and Environment Variables

For models requiring authentication, set the necessary API key in your environment or in the provider options.

If you are using OpenRouter, you will need an OPENROUTER_API_KEY. You can obtain your API key by signing up at https://openrouter.ai.

Once you have your key, set it as an environment variable in your terminal:

export OPENROUTER_API_KEY=your_actual_api_key_here

Alternatively, you can add it to a .env file in the project root:

OPENROUTER_API_KEY=your_actual_api_key_here

The benchmark will look for this environment variable when running models configured to use it. Provider configs can also use a hardcoded apiKey or api_key value under options. If both apiKeyEnv and apiKey are present, the environment variable wins and the hardcoded key is used as a fallback.

Model configs can set rate_limit_rpm on an individual model to pace API calls. If it is omitted, GomokuBench defaults to 50 requests per minute. Streaming OpenAI-compatible responses are supported by setting "stream": true in extra_body. API call timeouts can be set with timeout_seconds or timeout; if omitted, GomokuBench defaults to 120 seconds.

In general, to add a model:

• add a new config file under models/
• or keep it anywhere and pass it with --model-file /path/to/model.json
• point it at an OpenAI-compatible baseURL
• set the remote model name
• add any required API key env var to .env or export it directly in your terminal
• optionally set rate_limit_rpm if the provider has a lower request-per-minute limit
• optionally set timeout_seconds if the provider needs more time to respond

Examples in this repo include Ollama-compatible, Hugging Face Router, and OpenRouter model configs.