Repository navigation
Keep completed teacher searches across interrupted training-data jobs - #9
Draft
arhancanli wants to merge 1 commit into
Draft
arhancanli wants to merge 1 commit into
arhancanli wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A teacher-labeling run previously kept all results in memory until its entire sample finished. A process interruption could discard completed searches and require another full run. This adds a separate corpus runner that captures eligible fresh record bytes and labels them in bounded chunks without resampling on recovery. It does not change the live trainer, playing engine or promotion rules.
The manifest freezes sampled offsets and bytes, reference executable/options/node ceiling, source state, sampling seed and labeling code. Each completed chunk records paired original/WDL targets and source provenance, validates legal reference PVs and identical non-score bytes, and is atomically committed before updating progress. Recovery verifies completed artifact/report hashes and also handles a crash after chunk commit but before the progress write. Interrupted partial evidence is retained. A corpus lock prevents simultaneous writers; CPU bounds and the configured free-space floor apply before new work. Labeling runs only within a caller-reserved supervisor slot.
The default expanded study samples 65,536 eligible records with distinct source offsets; duplicate board/turn groups are counted and grouped by the paired trainer. This remains a conditional irreversible-move sample because legacy records omit other state. WDL targets and completed-iteration bounds retain their existing provenance rules. The final corpus contains paired binaries, audit rows and a validation manifest accepted by the paired trainer.
Validation: all 69 Python controller/teacher/pairing/corpus tests pass with ResourceWarning treated as an error, including 10 new regressions for interruptions, disk pause, checkpoint recovery, data/report corruption, input changes and concurrent writers. Syntax and diff checks pass. A real 512-record, 18-core Stockfish pilot passed separate-process recovery, independent native decoding and paired trainer feature/outcome checks. The first completed chunk was retained byte for byte. A frozen expanded sample contains 65,536 fresh records at distinct offsets, representing 65,533 board/turn groups; 7,680 labels in 15 completed chunks are preserved. The expanded run stopped at the configured 8 GiB free-space floor, and the regular engine service resumed. The remaining 57,856 labels are pending sustained disk headroom; no completed chunk was discarded or resampled. Exact-commit CI passed at 32e82ec, including release Rust tests, full perft/SPRT calibration, WASM and all 69 Python tests. No model training or strength gain is claimed.