Skip to content

Keep completed teacher searches across interrupted training-data jobs - #9

Draft
arhancanli wants to merge 1 commit into
engine/paired-training-20261005from
engine/resumable-teacher-20261005
Draft

arhancanli wants to merge 1 commit into
engine/paired-training-20261005from
engine/resumable-teacher-20261005

Conversation

@arhancanli

@arhancanli arhancanli commented Oct 5, 2026 •

Copy link
Copy Markdown
Owner

A teacher-labeling run previously kept all results in memory until its entire sample finished. A process interruption could discard completed searches and require another full run. This adds a separate corpus runner that captures eligible fresh record bytes and labels them in bounded chunks without resampling on recovery. It does not change the live trainer, playing engine or promotion rules.

The manifest freezes sampled offsets and bytes, reference executable/options/node ceiling, source state, sampling seed and labeling code. Each completed chunk records paired original/WDL targets and source provenance, validates legal reference PVs and identical non-score bytes, and is atomically committed before updating progress. Recovery verifies completed artifact/report hashes and also handles a crash after chunk commit but before the progress write. Interrupted partial evidence is retained. A corpus lock prevents simultaneous writers; CPU bounds and the configured free-space floor apply before new work. Labeling runs only within a caller-reserved supervisor slot.

The default expanded study samples 65,536 eligible records with distinct source offsets; duplicate board/turn groups are counted and grouped by the paired trainer. This remains a conditional irreversible-move sample because legacy records omit other state. WDL targets and completed-iteration bounds retain their existing provenance rules. The final corpus contains paired binaries, audit rows and a validation manifest accepted by the paired trainer.

Validation: all 69 Python controller/teacher/pairing/corpus tests pass with ResourceWarning treated as an error, including 10 new regressions for interruptions, disk pause, checkpoint recovery, data/report corruption, input changes and concurrent writers. Syntax and diff checks pass. A real 512-record, 18-core Stockfish pilot passed separate-process recovery, independent native decoding and paired trainer feature/outcome checks. The first completed chunk was retained byte for byte. A frozen expanded sample contains 65,536 fresh records at distinct offsets, representing 65,533 board/turn groups; 7,680 labels in 15 completed chunks are preserved. The expanded run stopped at the configured 8 GiB free-space floor, and the regular engine service resumed. The remaining 57,856 labels are pending sustained disk headroom; no completed chunk was discarded or resampled. Exact-commit CI passed at 32e82ec, including release Rust tests, full perft/SPRT calibration, WASM and all 69 Python tests. No model training or strength gain is claimed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant