v0.8.0 — inline script evaluation, derived memory budget, full mining candidates - #17
Merged
Conversation
…h client fetch_blocks took &[(u32, String)] and cloned every id into its batch queue, so the caller's list and the queue's copies were both live for all of phase 2 — ~167 MiB plus a ~153 MiB duplicate at 1.82M headers. It now takes the vec by value and moves the strings. maybe_refresh built a fresh reqwest::Client every 30s, each owning a connection pool; ~480 over a 4-hour header phase. Built once and held. Also documents addons/ as main-session-owned in CLAUDE.md — neither addon has per-crate scaffolding, so the dispatch rule never applied to them — and adds facts/fastsync.md, which surfaced two defects while being written: an ingest body limit that is documented but not implemented, and a resume path that silently drops block coverage below the restart point.
…emory Two JVM-parity fields on /info. maxPeerHeight was surfaced by a real consumer: the ergolumen.net node map computes sync progress as headersHeight / (maxPeerHeight || headersHeight), so omitting it made the map report 100% synced regardless of actual progress. It is Option<u32>, omitted rather than zero before any SyncInfo arrives — a zero would assert the network is at height 0. Replaces /debug/memory's chainHeaderEstimateBytes, which multiplied the header count by a hardcoded AVG_HEADER_BYTES = 800 sizing a Vec<Header> that chain/ retired in Phase 3. It reported 1.48 GB for a structure that no longer existed — more than the entire process RSS — and was quoted as the leading suspect during a live memory investigation. The estimate now lives in chain/ beside the fields it models, exposed through ChainAccess::memory_estimate: chainIndexBytes (the unbounded by_id map, real hashbrown bucket accounting), plus bounded header and score cache occupancy. The API layer transports these and computes nothing.
jemalloc defaults background_thread to false on Linux, so an arena's dirty pages decay only when a thread next touches that arena. With 4 x ncpu arenas (128 on a 32-core host) quiet arenas hold their pages indefinitely. Measured during a genesis resync 2026-08-11: resident-minus-active reached 1021 MB, a third of the process, while jemalloc's own `allocated` stayed flat — memory correctly freed by the program and never handed back to the OS. narenas is deliberately left at its default. It would also reduce per-arena retention, and changing both at once makes the verification run unattributable.
Without this feature redb's cache_stats() returns zeros, and /debug/memory would confidently report an empty page cache for what heap profiling showed to be 89% of header-phase growth. Declared in store/ and state/ as well as the workspace root: `cargo test -p enr-store` does not build the root crate, so a root-only declaration would not be unified into a standalone crate build and the accessors would silently report 0 under test.
cache_mb becomes the total across both databases, split by cache_store_pct (1-99, validated at startup). RedbModifierStore::new gains an explicit cache size instead of inheriting redb's 1 GiB default. /debug/memory gains storeCacheBytes, stateCacheBytes and storeCacheEvictions, all Option and omitted rather than zeroed when unavailable. Evictions are included because a rising count is the only signal distinguishing an undersized cache from a comfortable one.
… budget set_cache_size splits its argument 90% read / 10% write, but the in-place resize path (set_read_cache_limit) reaches only the read half — max_write_buffer_bytes is fixed at open() and redb has no setter for it. So after an at-tip resize the process still holds a write buffer sized from the cold-sync cache_mb, and stateCacheBytes reports a figure the current limit does not bound. A full drop-and-reopen moves both halves; only the in-place path is partial. Found by the state/ session while implementing cache_bytes_used, verified against patches/redb/src/db.rs:1177. Not fixed — bounding the write half at runtime needs a redb patch.
…ug/memory cache_mb becomes the TOTAL across modifiers.redb and state.redb, split by the new cache_store_pct (default 50, validated 1-99 at startup). modifiers.redb previously took redb's built-in 1 GiB default because RedbModifierStore::new called bare Database::create — a gigabyte nobody chose, invisible to /debug/memory, and 89% of header-phase heap growth in profiling. synced_cache_mb is likewise a total; the same split applies to the at-tip resize. /debug/memory gains storeCacheBytes, stateCacheBytes and storeCacheEvictions. The 158 MB the modifier store holds at tip was previously unattributed. Three things the crate sessions found that changed the work: - The at-tip in-place resize moves only 90% of the budget. set_cache_size splits 90/10 read/write and set_read_cache_limit reaches only the read half; the write buffer is fixed at open(). Documented in facts/state.md, not fixed (needs a redb patch). - used_bytes() reports the live working set, not the ceiling — identical at 8 MiB and 1 GiB under the same load. Only cache_evictions responds to the configured size, so storeCacheBytes must not be read as proof the budget binds. Documented in facts/store.md. - UtxoAccess is implemented by an adapter holding SwappableReader, not the storage, so the accessor had to land on SnapshotReader to be reachable. Maintenance paths take a small explicit cache rather than the operator's budget: --reset-scores-migration deletes one key, state_inspect reads a few.
…rumentation facts/sync.md already recorded "No backpressure" and a per-item cost of ~25KB/~410KB. It never stated that the COUNT is unbounded, which is where the memory exposure actually lives. evals_in_flight is written only by += 1 at dispatch, -= 1 at drain, and two resets. During catch-up `blocking = sweep_size <= 1` is false, so a 192-block sweep dispatches up to 192 evals and never waits. If verification is slower than state application the queue grows across sweeps without limit. Consistent with a field OOM on 2026-08-12 (anon-rss 10.62 GiB, file-rss 2.4 MB, 4-thread host): 10.62 GiB / 410 KB is roughly 27,000 queued evals. Invisible on high-core hosts where Rayon keeps pace, which is why our own profiling never surfaced it. Specifies a catch-up progress record (evals_in_flight, both watermarks, eval_lag, jemalloc allocated) so the depth is observable. Diagnostic only — bounding the queue is a separate design question.
…ght source The sync/ session caught a defect in my contract: I specified state_applied_height as "existing watermark", pointing at the struct field. That field is a cache reconciled only AFTER the sweep loop, so mid-sweep it stays frozen at the pre-sweep tip while script_verified_height climbs past it. A record built from it would peg eval_lag at 0 by saturation for the whole sweep — instrumentation that runs, logs, and reports a healthy system while the queue grows. Corrected to the validator's validated_height(). Also bumps the journal-events contract to 1.5.0. It declared 1.3.0 while already documenting validation_rollback_failed as "Since: 1.4" — drift found independently by an operator. api/src/lib.rs::JOURNAL_EVENTS_VERSION is still "1.3" and remains owed.
…esult Diagnosed by the sync/ session using the instrumentation added hours earlier. `verified` in drain_eval_results is a function-local BTreeSet dropped on return, so a result whose predecessor is still running is drained, found non-contiguous, and DISCARDED. The channel never resends it and the contiguous frontier can never pass that hole. Live trigger: block 3522 is 3 txs / 17 inputs, the first non-coinbase-only block in its region; with two rayon threads the trivial eval for 3523 overtook it and was dropped. Frozen at 3522 through 190,000+ blocks. Scripts ARE still evaluated and failures still roll back — the Err branch does not depend on contiguity — so this is bookkeeping, not a verification skip. Corrects two things I wrote today: the contract's "advances in-order as eval results arrive" (false — it advances only while nothing overtakes), and the journal-events claim that eval_lag is the field that climbs if the backlog hypothesis holds. eval_lag read 187,711 while evals_in_flight was 1 and jemalloc allocated was flat. Trust evals_in_flight only until this is fixed.
…ozen watermark Adds a catch-up-only INFO record every 5s: evals_in_flight, the applied tip, script_verified_height, eval_lag and jemalloc allocated. New private module sync/src/eval_backlog.rs (gate + emitter) with the emit itself under test. Motivated by an operator OOM on 2026-08-12: anon-rss 10.62 GiB on a 4-thread host during catch-up. evals_in_flight has no cap and the sweep-end drain never blocks during catch-up, so a slow verifier can accumulate DeferredEvals — each holding a block's parsed txs plus every input and data-input box — without limit. Nothing exposed the depth. The record immediately found a different bug, now characterized in tests but NOT fixed: `verified` in drain_eval_results is a function-local BTreeSet dropped on return, so an eval result arriving before its predecessor is drained, found non-contiguous, and discarded. The channel never resends it and script_verified_height can never pass the hole. Live: block 3522 (3 txs, 17 inputs) was overtaken by the coinbase-only 3523 under RAYON_NUM_THREADS=2; the watermark froze there for 190k+ blocks. Scripts are still evaluated for every block and failures still roll back — the Err branch does not depend on contiguity — so this is bookkeeping, not a verification skip, and startup already self-heals the persisted value. Consequence for readers: trust evals_in_flight, NOT eval_lag. Measured eval_lag=187711 while evals_in_flight=1 and jemalloc allocated was flat.
…accounting Bytes-in-flight bound with a count guardrail, gated at dispatch, releasing at half of each enabled bound. The budget is deliberately disjoint from flush_heap_threshold_mb: flushing redb frees dirty pages, not queued evals. Corrects the per-DeferredEval size figures, which were 6-8x low — they counted payload and missed that ergo-lib's Transaction materialises every output twice. The 10.62 GiB OOM divides to ~3,000 queued evals, not ~27,000; the measured quantity is the anon-rss, so quote that instead. journal-events to 1.6.0: eval_bytes_in_flight on the backlog record, plus deferred_eval_gate_engaged and eval_frontier_hole. Records one open decision: whether the frontier may advance over heights whose scripts were never evaluated. Checkpoint skips and startup gaps are the same question and are answered together, not separately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sync/ needs to bound its eval queue by memory, and cannot weigh a DeferredEval itself — it never sees the boxes. Both construction sites now go through a shared DeferredEval::new that derives approx_heap_bytes from serialized sizes already on hand, so digest and UTXO mode cannot drift. The field is private and read through an accessor. A pub field would let an out-of-crate struct literal set it to zero, and a zero silently disables the caller's bound — the invariant is load-bearing enough that it should not rest on nobody writing a literal. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rmark The queue had no cap. A 4-thread host was OOM-killed mid-catch-up at 10.62 GiB of anonymous RSS. Dispatch now gates on eval_backlog_max_mb (256) and eval_backlog_max_blocks (256), draining to half of each enabled bound; a disabled bound gets a low-water mark of never, not of zero. Three bugs in the same pipeline, all of which the bound depended on: - The reorder buffer was drain-local, so a result that overtook its predecessor was dropped and the frontier could never pass the hole. It was wedged at 3522 for 190,000 blocks. Hoisted to a field, paired with an eval_generation stamp so a pre-rollback result is retired for accounting but cannot reach the frontier. - handle_eval_failure zeroed evals_in_flight while its rayon tasks were still running and still holding their heap, which is exactly the undercount that opens the gate early. It bumps the generation instead, and only where the validator actually moved. - Reorg invalidation was unconditional; it is now conditional on rolled_back. The eval channel moves from crossbeam to tokio mpsc so the drain awaits rather than blocking a worker. block_in_place panics on current_thread, which every test uses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…kers An api/ audit found both events added today disagree with what this document specifies, and found why nothing caught it. The gate marker began with the whole of deferred_eval_backlog's marker, so a prefix parser matched both events and then failed looking for eval_lag. It also carried a U+2014 inside the matched portion while this document specified a hyphen. Renamed to "eval dispatch gate engaged". Both field lists move to what the code emits rather than the reverse: naming which bound tripped and the eval that did not fit beats restating static config the operator already has. Adds two format rules — no marker may be a prefix of another, and the matched portion is ASCII — since both were broken the day the convention was written down. Also bumps JOURNAL_EVENTS_VERSION 1.3 -> 1.6, which had drifted two versions behind, and openapi.yaml 0.6.5 -> 0.7.11, six behind. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A node configured with a checkpoint above its start height had a permanent frontier hole: heights at or below the checkpoint never dispatch an eval, so the results that would fill the gap are never produced. Resolved by flooring the watermark at checkpoint_height everywhere it is set — not only at startup, since a reorg to a fork point below the checkpoint would re-open it. The old invariant claimed heights below the watermark were "fully validated (state + scripts)". Outside sync/ the watermark has exactly one consumer, src/bridge.rs, which persists it; it is on no endpoint and read by no other crate. Its three real jobs — rollback target, restart resume point, eval_lag — all want "no further script work is owed below here", which the floor gives. The stronger claim was not required by anything and could not be honoured by a checkpointed node. Startup gap handling deliberately does NOT get the same treatment and stays open. A checkpoint is a feature the operator configured; a startup gap is an accident of shutdown timing with nothing to gate on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The journal-event tests pinned inline mirrors: each called info! with its
own copy of the marker and fields, then asserted the output contained that
copy. They tested tracing's formatter against itself and never invoked an
emit site, so they could not fail when one drifted — which is how three
mismatches shipped under a green suite.
All 8 mirrors are gone, replaced by tests that drive the real emits. Every
event turned out to be reachable; none had to stay a mirror.
The three mismatches they hid: the gate marker carried a U+2014 and began
with deferred_eval_backlog's entire marker; eval_frontier_hole omitted
state_applied_height; validation_rollback_failed renders ValidationError's
full Display while the mirror asserted the bare inner message and passed.
Assertions match whole tokens, not substrings — contains("height=1785500")
also matches tip_height=1785500, and a first pass passed a renamed field.
Adds tests/journal_events_contract.rs at workspace root for the two rules
that are properties of the document and need no emit sites: no marker is a
prefix of another, and markers are ASCII. Guarded against vacuous passes,
since that is the failure mode being corrected here. Verified all four can
fail by breaking each rule in turn.
Consolidates four near-identical capture helpers into sync/src/test_support.
The subscriber's max level now lives there and callers name it: fmt()
defaults to INFO, so a DEBUG contract event renders to nothing and looks
exactly like one that was never emitted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
sync/ cannot see the checkpoint at all — it travels from src/main.rs into the validator and stops. It arrives as a SyncConfig field fed from the same configured_checkpoint.unwrap_or(0) the validator is built with; two independently-derived checkpoints disagreeing by a block would rebuild the frontier hole one block wide and much harder to see. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Heights at or below checkpoint_height never dispatch an eval, so a node configured with a checkpoint above its start height could never advance the frontier to it — a permanent hole, and the reorder buffer grew a u32 per block for the life of the process. Six sites set the watermark; five now route through one private setter that applies the floor. The sixth is the += 1 loop, which only increments from a value that already respects it. The wiring is NOT configured_checkpoint.unwrap_or(0), which is what the obvious reading suggests. The validator is built in four branches and digest mode resuming from a stored tip defaults to height-100, not 0; that expression would put the floor up to a whole chain below the eval-skip boundary on an unconfigured digest node, reopening this hole at full width in the one mode nobody would check. main.rs captures the value the validator was actually constructed with, through a recorder every branch calls. Startup gap handling is untouched and stays open. Its site routes through the setter for the floor only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The at-tip cache resize has never reached state.redb. The enum wrapper in main.rs is pure delegation, but resize_cache was missed when it was added to the trait — and the trait supplies a do-nothing default returning Ok(()), so it compiled, did nothing, and logged success. UtxoValidator::resize_cache was correct the whole time and unreachable. Found by @odiseusme reading main.rs rather than trusting the metrics; his runtime data agrees, stateCacheBytes climbing to 542 MB against a claimed 64 MB resize. The contract now forbids defaulted methods on BlockValidator. Two more carry one — flush and proofs_for_transactions — and flush is the same shape with a much worse failure: state never persisted, every caller told otherwise. Removing those defaults needs explicit no-ops in DigestValidator and the sync test stubs, so it is dispatched separately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deferred evaluation buys sync throughput with a crash-consistency window: apply_state persists a block whose scripts have not been checked, so an unclean shutdown leaves applied-but-unverified state. We watched that fire on a real reboot today, gap=1. Inline mode evaluates inside apply_state and returns deferred_eval = None, so Ok means the scripts passed. The boundary is persistence, not application: the JVM validates before touching its tree, but it reads each input box separately and pays two traversals, while ours captures the box from the removal itself. Everything from the first prover operation to just before storage.update_with_height is in-memory, so evaluating in that window means nothing unverified ever reaches state.redb. Kept as a mode rather than a replacement because the cost lands hardest on slow hardware and we have no measurement from any box that is actually slow. Default stays deferred; nothing changes for existing operators. Also records a live defect the mode change forces us to fix: the digest-mismatch path returns Err after applying every operation to the prover and never undoes them, while the trait's Err postcondition claims state is unchanged. Unreachable today, routine under inline evaluation. The requirement is stated; the mechanism is validation's to choose, since pre-persist rollback is a state/ question the main session cannot answer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ScriptEvalMode { Deferred, Inline }, required positionally by every
validator constructor — a mode that can be silently forgotten is the
resize_cache failure shape, which we shipped once already this week.
Inline evaluates inside apply_state, after the state-root check and before
persisting, and returns deferred_eval = None. Ok therefore means the scripts
passed, and nothing unverified reaches state.redb.
Wired from a single binding in main.rs. SyncConfig must be fed from the same
one: two copies that disagree either freeze the frontier waiting for results
nothing dispatches, or advance it over blocks nothing verified.
Default stays deferred. Logged at startup, because which mode is active is a
durability fact an operator should not have to infer.
Two corrections from the implementing session, both accepted:
- The prover-cleanliness defect I specified did not exist. apply_state wraps
apply_state_internal and already rolls back on any Err (2992645); I read
the inner function's bare return and attributed it to the outer. Task was
verification, not construction.
- Pre-persist rollback is sound, but not for the reason assumed: the undo log
never runs there. current_version is only set after a successful commit, so
at the evaluation point rollback takes its short-circuit branch and re-reads
the persisted root. No write txn, no undo record.
Also fixes an unbounded retention they found: restore_root clears three
buffers but not base.modified_nodes, so every rejected block pinned its
touched node set for the life of the process. Worked around in
rollback_prover_to; the proper home is the avltree fork.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three corrections, two of them to claims that were simply wrong. The prover-cleanliness defect did not exist. apply_state wraps apply_state_internal and has rolled back on any Err since 2992645; the draft read the inner function's bare return and attributed it to the public entry point. Recorded rather than quietly deleted, because in a file with an _internal split the enclosing function is part of what a line means and a grep result does not carry it. Pre-persist rollback is sound for a reason neither candidate mechanism anticipated: the undo log never runs there. current_version is set only after a successful commit, so rollback takes its short-circuit branch and re-reads the persisted root. The undo-log walk runs only for the post-persist proof-digest check. Both work, for different reasons. Inline closes the SCRIPT gap only. The proof-digest check still fires after update_with_height, so a post-persist Err stays reachable and a crash there still leaves a persisted block whose watermark never advanced. That gap is now merely bookkeeping — the scripts did run — so accepting it asserts something true, which is not the case in deferred mode. "Startup gap handling" therefore stays open for deferred only. Also records the restore_root retention that inline mode promotes from never-happens to once per hostile block, and that inline currently discards the block cost evaluate_scripts returns. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
In inline mode apply_state returns deferred_eval = None, so nothing is dispatched and nothing is ever drained — script_verified_height would sit at its startup value forever. It is now set at the successful-apply site, through set_script_frontier so the checkpoint floor still applies, and persisted on the same cadence as the drained path. script_eval_inline is derived from the single mode binding in main.rs, never re-parsed. That literal ends in ..SyncConfig::default(), so omitting the line would have compiled and left sync in deferred bookkeeping while the validator evaluated inline — the exact divergence the contract warns about, silently. Noted at the site. Correction from the implementing session: eval_frontier_hole suppression fell out, but not for the reason the prompt gave. The WARN needs evals_in_flight == 0 AND a non-empty reorder buffer; inline never inserts into that buffer, so it was structurally unreachable already. The frozen watermark was the only real regression. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Mechanical follow-on from the required-mode constructors. These were left uncommitted by an over-narrow `git add validation/src` in e788b41 — the workspace was green throughout because the working tree had them, which is exactly how a staging mistake hides. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
restore_root now clears base.modified_nodes upstream, and PersistentBatchAVLProver::rollback delegates to it rather than hand-rolling a second rewind that had already drifted once (that drift is what #19 patched). Shipped as upstream PR #27, open and not merged; fork main carries the fix as a cherry-pick, and both signatures are unchanged. Our exposure was retention only, not the wrong-proof divergence the fork session measured — two provers at identical tree state emitting 740 vs 735 proof bytes. That needs a storage layer whose rollback hands back a live NodeId, as their VersionedAVLStorageMock does. Both RedbAVLStorage::rollback paths end at tree.unpack of bytes freshly copied out of a redb read txn, so the restored root is always a new allocation, and the entries pinning the address-keyed map keep their addresses alive and unrecyclable. Nothing was ever misidentified here. The second gap does not reach the node either: PersistentBatchAVLProver appears only in an example and one integration test, both on a fresh prover. The local workaround in UtxoValidator::rollback_prover_to is redundant at this rev but stays until it is removed deliberately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
validation.md: restore_root clears base.modified_nodes at the pinned rev, so the local clear in rollback_prover_to is redundant rather than load-bearing. Says so explicitly, because a reader on an older tree would otherwise conclude the opposite from the code. state.md: adds the invariant the upstream fix depends on. rollback() must return a freshly unpacked root and never a cached live Rc. modified_nodes is keyed by node address and on_node_visit inserts every VISITED node, so a recycled handle comes back into a map still holding its stale entry, pack_tree expands nodes it should have labelled, and the prover emits a different proof for identical tree state — 740 vs 735 bytes at the same digest, measured upstream against their VersionedAVLStorageMock, which does hand back the saved Rc. We are exempt by construction: both RedbAVLStorage::rollback paths end at tree.unpack of bytes copied out of a redb read txn. Written down because that exemption is a property of the current implementation, not of the design, and the obvious future optimisation — a node cache on the short-circuit path — would delete it silently. Caching the packed bytes stays safe. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…pstream b06e652 left this one deliberately: "redundant at this rev but stays until it is removed deliberately." This is that removal. restore_root at fork rev b955790 clears base.modified_nodes itself (batch_avl_prover.rs:102), right after the changed-node buffers and alongside the directions and needs_cycle_reset, so rollback_prover_to calling it again did nothing. The clear moved upstream — it did not disappear. That distinction is the whole reason the comment needed rewriting rather than deleting: the old block narrated a fork bug that no longer exists at our pin, and a comment describing a bug that has been fixed is worse than no comment at all, because the next reader concludes the line was load-bearing and puts it back. The replacement says where the clear lives now and not to reinstate it. The two modified_nodes.is_empty() assertions in the rejection tests are untouched. They assert a postcondition, not the mechanism, so with the local clear gone they cover the upstream guarantee instead of our own, which makes them the regression detector if the pin is ever moved back below b955790. The comment points at them for that reason. Behaviour is unchanged by construction — the clear was a no-op at this rev — and the suite agrees: 56 binaries, 1039 passed, 0 failed, 3 ignored, identical to the pin bump. Both rejection tests still pass on the upstream clear alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The four defaulted methods come off BlockValidator and become two traits implemented only by UtxoValidator: StatePersistence (flush, resize_cache, consumed by sync/) and MiningState (proofs_for_transactions, emission_box_id, consumed by main's update_mining_proofs). Each of the four had exactly one consumer and they did not share it, which is what makes the split fall out rather than being taste. The alternative was the contract as previously written — declare every method, no defaults, let the compiler catch the next wrapper that forgets. That still depends on someone reading nine compiler errors correctly, and costs 24 explicit no-op bodies, 20 of them in sync/'s test stubs. The split removes the per-method forwarding entirely: the wrapper exposes one accessor per trait, so a method added later needs no wrapper change and there is nothing to forget. Mode is signalled by the absence of an impl, never by a harmless return value. Two overloaded Nones die with it. proofs_for_transactions returned Option<Result<..>> whose outer Option meant "wrong mode", and emission_box_id returned None for either "digest mode" or "all ERG emitted" — two unrelated facts wearing one value that update_mining_proofs early-returned on identically. It now means all ERG emitted, and nothing else. sync.md gets the access path at all five flush sites and a new section stating the semantics main owed it rather than leaving to the implementing session: None (nothing to persist) and Err (durability unknown) must not be collapsed. That collapse IS the bug this split prevents — a validator reporting success for work it never did. Digest mode joins the existing no-validator arm. No behavioural delta. Digest mode returned Ok(()) from the defaults and now yields None from state_persistence(), handled as the same arm. Also corrects two drifts in validation.md's code block, which had gone stale against the code it specifies: apply_state was missing expected_proposed_update, and the block omitted flush/resize_cache entirely while listing the mining pair as required. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
prover_memory_estimate on StatePersistence reports all three per-cycle buffers, the resident tree, and the node count behind both — walked from the real structures, never a count times a constant. UtxoValidator implements it; DigestValidator does not implement the trait, so digest mode reports absent. Two contract errors of mine, corrected here. The buffers are NOT cleared on flush: flush is a redb fsync and never touches the prover, and the clear is the proof-cycle boundary inside apply_state. Sampled between blocks the buffer figure therefore reads its idle floor by construction and cannot show a leak, so it is tested at the proof cycle instead. And the estimate is O(resident nodes) — the same order as applying a block — so it publishes at flush cadence, not per block, which would have been a real sync regression. THE FINDING, which is the answer to the investigation this started as. AVLTree::resolve swaps a LabelOnly child for the unpacked node in place and nothing anywhere converts one back: every construction site is a fresh node or a resolver closure, there is no inverse, no pruning, no eviction. Only restore_root — a rollback — resets it. So READS grow the prover, permanently. A pure unauthenticated_lookup sweep with zero mutation materialises every path it touches and holds it for the life of the process. At ~192 B per internal node and ~724 B for a leaf carrying a 500 B box, that reaches the gigabyte range at mainnet scale — the right order for the 1356 MB unattributed on a node that applied ~27k blocks against 214 MB on one at the same tip that had not synced. It also explains why the at-tip cache resize cannot help: this is the prover's own node graph, not a redb cache, so an operator turning cache_mb down watches RSS refuse to move. Reported, not fixed. Recorded in facts/state.md § Resolver Strategy because it is invisible from every memory knob the node exposes. A test pins the measurement against perturbing what it measures: after a rollback the frontier must report fewer nodes than the seeded tree, so a walk that resolved would pull the entire UTXO set into RAM in order to size it.
Two corrections and one result. The window is not the leak. sync holds zero section payload bytes and keeps no persistent download queue: the P2P pipeline writes bytes to the modifier store and notifies sync with ids only, and the sweep reads each block back out, applies it and drops it inside one loop iteration. Downloaded-but-unapplied bytes live in redb and are already counted as storeCacheBytes, so counting them here would double-count. What remains is DeliveryTracker — an id-keyed pending map and evicted vec bounded by the 192-block window, tens of kilobytes. That leaves the AVL prover as the sole suspect, which facts/state.md then confirmed. The field was named buffered_section_bytes and documented as "section payloads received and awaiting application". No such thing exists in the crate; that was my assumption rather than the architecture. Renamed to tracker_bytes and tracker_entries so the published number is readable. Gauge type settled: Arc<AtomicU64> is the interchange, PublishedGauge is a view over one. api and sync cannot name each other's types, and their only shared crates are enr-p2p, ergo-chain-types and ergo-validation — none a sensible home for an observability primitive. Main allocates one Arc and hands it to both ends. A gauge owning its AtomicU64 outright cannot be shared with a producer in another crate, which is the entire requirement. Also records the HashMap::capacity() trap: it returns items plus growth-left, and erasure only returns a slot when the probe sequence permits EMPTY over DELETED, which depends on RandomState's per-process seed — the identical workload measured 16640 B then 8320 B. Count live entries. Vec::capacity() is unaffected and is the honest figure there.
sync landed its producer side on Arc<AtomicU64>, which facts/sync.md specifies, while the gauge owned an AtomicU64 outright — so one allocation could not serve both ends, and it has to, because producer and reader sit in crates that cannot name each other's types. The gauge now holds an Arc<AtomicU64>: from_storage() adopts one, storage() hands it back. Sentinel, ordering, publish/get and the JSON shape are all unchanged, and the nine existing debug_memory tests pass unedited. One new test covers the property the design rests on — two gauges over one storage see each other's writes.
Three Arc<AtomicU64> allocated once and shared: Validator writes the two prover figures, HeaderSync the window, the API reads all three through PublishedGauge views. The prover walk is O(resident nodes), so it runs every 512 applied blocks rather than per block. facts/validation.md said "flush cadence" and that is not implementable here: sync calls flush() through state_persistence(), so the Validator never observes one, and intercepting would mean Validator itself implementing StatePersistence to wrap the delegate — restructuring the split traits for a diagnostic gauge. Contract amended to name the interval instead. Digest mode never writes the prover gauges: state_persistence() returns None and the publish returns early, so /debug/memory omits the fields rather than reporting a zero that would assert an empty prover. A debug_assert pins the two independent sentinel definitions together. ergo_sync::WINDOW_BYTES_UNSET is stamped into the window gauge by HeaderSync itself and the reader's constant is private to the api crate; nothing enforces that they agree. If they drifted, an unmeasured window would render as a ~16 EiB reading rather than an absent key — a wrong answer shaped like a right one, which is the failure this endpoint exists to prevent. The gauges are minted through the reader's own constructor for the same reason.
…dence The contract said publish at flush cadence. That is not implementable where the publish happens: sync calls flush() through state_persistence(), so the main crate's Validator never observes one and has no hook to hang it on. Intercepting would mean Validator implementing StatePersistence purely to wrap the delegate — restructuring the split traits for a diagnostic gauge. Names PROVER_GAUGE_INTERVAL_BLOCKS and the post-apply_state hook instead, which is what shipped. The requirement was only ever "not per block", and that holds.
… suspect HeaderSync::window_memory_estimate publishes into a shared Arc<AtomicU64> after each applied block, stamped WINDOW_BYTES_UNSET by HeaderSync::new itself so the figure reads as absent rather than zero before any block lands. The measurement's result is that this crate is not where the memory goes. sync holds ZERO section payload bytes and keeps no persistent download queue: the P2P pipeline writes bytes into the modifier store and notifies sync with ids only, and the sweep reads each block back out, applies it and drops it inside one loop iteration. Downloaded-but-unapplied bytes therefore live in redb and are already counted as storeCacheBytes — counting them here would double-count. What remains is DeliveryTracker alone, bounded by the 192-block window at tens of kilobytes. It cannot be the 1356 MB, which left the AVL prover as the sole suspect and led to the finding now recorded in facts/state.md. Fields are named tracker_bytes and tracker_entries. They were first written as buffered_section_bytes and queue_entries, against a contract I wrote describing "section payloads received and awaiting application" — a thing that does not exist in this crate. Renamed so the published number reads as what it counts. Sized from live entry counts, never HashMap::capacity(): that returns items plus growth-left, and erasure only returns a slot when the probe sequence permits EMPTY over DELETED, which depends on RandomState's per-process seed. The identical workload measured 16640 B on one run and 8320 B on the next, and the crate's own test caught it. Documented as a lower bound — real allocation is at most ~2.3x with power-of-two buckets and a 7/8 load factor.
512 blocks is seconds during catch-up and roughly seventeen hours at tip, where blocks arrive every two minutes. As written the gauge published once on the first applied block and then not again until the next day — useless for watching a growth curve on a synced node, which is exactly the case an operator monitors and the case this instrumentation was built to answer. Adds PROVER_GAUGE_MAX_INTERVAL (10 min) as a ceiling, whichever trigger fires first. The two never compete: during sync the block count is reached long before ten minutes elapse, and the cost that motivated the block interval does not exist at tip, where one tree walk per two-minute block is free.
Yesterday's section claimed reads resolve LabelOnly nodes permanently and named that as the cause of a node's unattributed heap. The mechanism is real in the fork and it does not fire on the path that matters, so the conclusion drawn from it was wrong. The prover's tree is 100% resident by construction. A genesis sync builds the AVLTree in memory and inserts; no LabelOnly node can enter, because the only constructors sit behind AVLTree::resolve, which is a no-op unless the child is already LabelOnly. The set starts empty and is closed under every operation. Measured over a 689k-block sync: zero resolver misses, zero rollbacks. There was nothing to evict because nothing was ever resolved. proverResidentNodesBytes therefore measures the UTXO set itself — node_count is 2 x boxes - 1, always odd with even deltas — and it falls when the set contracts. Verified against data the node cannot influence: the last byte of stateRoot is the AVL height, and it fell 20 -> 16 between h=205,440 and h=247,000, exactly where the gauge dropped 54.5 MB to 5.3 MB. An AVL height only falls on deletion, and 16 caps the tree at 65,536 leaves. Also retracts the proposed fix. Releasing subtrees after update_internal commits is not a leak repair, it is adding eviction that never existed: the fork has no inverse of resolve, so it would have to be written, and every subsequent touch pays a redb read plus unpack. That is a throughput-for-memory trade to be measured, not a defect. Cost recorded as it actually stands: ~500 B of RAM per UTXO box carrying ~86 B of box data, 568 MB at h=688k, about 28% of live heap. The other ~72% remains unattributed.
Odiseus mined against 0.8.0 for nine minutes at 143 MH/s and submitted zero shares. Not zero accepted — zero submitted. Same height, same moment, their Scala node served b=2959...2588 (68 digits) and ours served b=3912040448 (10). mining/src/candidate.rs binds decode_compact_bits(n_bits) to a variable named `target` and ships it as WorkMessage.b. That function returns the difficulty. The target is q/difficulty, q being the secp256k1 group order. floor(q / 3912040448) reproduces their Scala value digit for digit, remainder 1309294913. Only the serve path was wrong. validate_solution goes through Header::check_pow, which computes order_bigint()/decode_compact_bits correctly — so the node stood ready the whole time to accept a block it had made ~10^58 times too hard to find. Odiseus proved it by patching only their bridge to substitute floor(q/b): block accepted, chain extended 485896 -> 485897, 50 seconds after the miner connected. Both contracts said decode_compact_bits, in three places between them, and the implementation matched the contract faithfully. Fixing the contract is therefore the first move, not a follow-up: - facts/chain.md gains pow_target(n_bits) as the single definition, and its verify_pow postcondition goes <= to <, matching the code. - facts/mining.md points b, the solution-side check, and the dependency list at it; CandidateBlock.n_bits corrected u64 -> u32 (impl was always u32, so was Header.n_bits). - Testing strategy gains the regression vector, pinned to observed values rather than to our own formula. n_bits there is derived by canonical encoding rather than read off the wire, and says so. Item 12 called msg-matches-JVM "the ultimate correctness test". A candidate can have a byte-perfect msg and still be unmineable, which is what shipped. The three existing tests over b checked that it was non-empty, stable across polls, and emitted as a bare JSON number. All three passed throughout. None asked what the number was. Implementation dispatches next: chain/ adds pow_target, mining/ consumes it. Reported by Odiseus; the diagnosis and the live proof are theirs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… wrong decode_compact_bits returns the difficulty. The target is q divided by it. Nothing in this repo said so out loud — the division lived inline in verify_pow, one expression with no name on it — so mining/ re-derived the target from the same primitive for WorkMessage.b and stopped one operation short. External miners got a target ~10^58 times too hard; Odiseus ran a GPU against 0.8.0 for nine minutes at 143 MH/s and submitted zero shares, while verify_pow, computing the target correctly, stood ready the whole time to accept a block it had made impossible to find. A regression test in mining/ would pin the one recurrence we already know about. It would not stop the third consumer from doing what the second did, because the failure was not a typo — it was a reasonable reading of a primitive whose name promises a target and delivers a difficulty. So the fix belongs here: the target is now a function with a name, a doc comment that says plainly what decode_compact_bits actually returns, and the account of what it cost. order_bigint() appears exactly once in the crate, inside it. verify_pow calls it rather than duplicating it — otherwise this is just a third place to get the division wrong. Behaviour unmoved; the comparison stays strictly <. The test pins the vector Odiseus observed on their Scala node as literals. Writing it as pow_target(n) == order_bigint() / decode_compact_bits(n) would restate the implementation and pass on any self-consistent definition, including the wrong one. It asserts the n_bits -> difficulty round-trip first, because the difficulty and target are evidence while n_bits is arithmetic I did to get there; if that derivation is wrong the round-trip fails loudly instead of the target assert quietly testing another difficulty. Contract: facts/chain.md § "Phase 2: PoW Verification" (498a538). Purely additive to the public API — mining/ still calls decode_compact_bits directly and still builds; consuming pow_target is a separate change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two errors in the same schema object, found while tracing the b defect. b was declared `type: string`. The wire has always emitted a bare JSON number — that is the entire purpose of the RawValue machinery in mining/src/types.rs, and mining's own tests assert both that `"b":123..` appears and that `"b":"123.."` does not. So a client generated from this spec did not lose precision, it failed to parse. Corrected to `number`, with the arbitrary-precision caveat stated: a real target runs ~68 digits and does not survive IEEE-754, so consumers must read the raw token rather than a native JSON number. Its description also said "Difficulty target", which is the ambiguity this whole arc is about — b is the target, `q / decode_compact_bits`, and the decoded nBits alone is the difficulty. Now says which is which and points at facts/mining.md step 4. proof was described as "Optional sigma-proof fields when the candidate requires them". It is ProofOfUpcomingTransactions: msgPreimage plus Merkle membership proofs for mandatory transactions. Nothing sigma about it. Recorded why it is omitted rather than empty, since the omission is load-bearing — a nested proof object overflows the reference Autolykos2 miner's jsmn token buffer and it stops mining outright. Neither error was reachable from the type system: nothing ties this file to the workspace, which is also how it sat at version 0.6.5 for six releases. It is checked by reading it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
WorkMessage.b carried decode_compact_bits(n_bits). That is the DIFFICULTY. The Autolykos target is q / difficulty, q being the secp256k1 group order, so every external miner was handed a bound tens of orders of magnitude tighter than the one this node actually enforces. At testnet height 485,897 we served 3912040448 where a Scala node served 29598898778389163379010897437604384363675568080188445020547283242588 — floor(q / 3912040448) digit for digit, remainder 1309294913. Odiseus ran a GPU miner at 143 MH/s for nine minutes and submitted zero shares. Not zero accepted: zero found. solution.rs validates through Header::check_pow, which divides correctly, so the node stood ready the whole time to accept a block it had made impossible to find. b now comes from enr_chain::pow_target(n_bits) — added to chain/ in c551eb5 as the single definition of this value — rather than being re-derived here. That is the fix: not a corrected expression, one call site. The naming is what did it. decode_compact_bits sounds like it yields the number you compare a hash against, and every other consumer in this repo binds it to a variable called `difficulty`. Ours was the one site that called it `target`, and the name is what made it invisible in review. How it got past three tests: mine_blocks asserted b was non-empty, candidate_generator asserted it was stable across polls, types asserted it emitted as a bare JSON number rather than a quoted string. All three passed continuously while the field was wrong. A value can be present, consistent and well-typed and still be the wrong quantity. The doc comment on serialize_b_as_number had even written the bug down as an invariant — "b is always decode_compact_bits(..).to_string()" — inside the function that emits the field. Two new assertions, both confirmed to fail against the reverted fix: - served_b_matches_scala_node_vector pins work.b to the observed literal, not to pow_target(n).to_string(), which would restate the implementation and pass on any self-consistent definition including the one that shipped. The n_bits round-trip is asserted first, because n_bits is derived by canonical compact-bits encoding while the difficulty and target are observed — a bad derivation then fails loudly instead of quietly testing another difficulty. - served_b_is_the_bound_the_solution_path_enforces scans nonces at difficulty 2 and requires validate_solution's own verdict to agree with hit < b_served on every one, with both verdicts required to occur. check_pow is upstream's and returns a bool rather than its target, so the target is observed at its boundary. This is the property that was actually violated — two paths, two numbers, nothing tying them together — and the literal above would not catch the solution side drifting later. mine_blocks' non-empty check now demands >= 20 digits; a real target only falls below that at a difficulty around 10^57, so anything shorter is the difficulty leaking through again. types.rs's shape tests move off the 12237864960 literal: valid as a JSON-shape fixture, but a difficulty-magnitude value in that field now reads as an example of the bug. Cargo.lock's one-line enr-chain entry is left unstaged — repo root, main's. ergo-mining 55 passed / 0 failed. Workspace 1069 passed / 0 failed / 3 ignored (1067 before this commit; c551eb5 added the chain-side vector). Reported and diagnosed by Odiseus, who also proved the fix live. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One line, the consequence of 4d63efa wiring mining/ to enr_chain::pow_target. Repo root, so it lands here rather than in the dispatched session — and a --locked build, which is what build-deb runs, fails until it does. Nothing else in the lockfile moved. Worth confirming rather than assuming: this workspace must never fresh-resolve, since core2 0.4.0 is yanked and the sigma-rust crates are pinned by git rev. A one-line diff is the evidence that cargo added an edge instead of re-solving the graph. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…of them I attributed this to v0.8.0 across three contracts. Wrong, and wrong in the direction that matters: it dates to 5b65e49, the commit that completed the mining crate, and `git tag --contains` puts it in every tagged release from v0.1.0 through v0.7.11. Forty-two of them. External mining has never worked in a released build of this node. What saved us from shipping it again is that it was caught on release/v0.8.0 before that tag exists — v0.7.11 is still the newest tag — and only because an operator pointed a real GPU at it rather than reading the candidate. Nothing earlier could have caught it. "Verified at runtime against mainnet tip" meant candidate assembly: well-formed, JVM-shaped, byte-identical msg. Even the two prior JVM-compat serve fixes — b as a bare number, proof omitted when empty — were about getting a real miner to PARSE the candidate. It parsed, then mined against an impossible bound, and the only symptom was silence. That generalizes into something worth keeping: a miner that receives, parses and accepts your candidate and then reports nothing is not idle. Corrected the version claims in facts/mining.md (steps 4, solution-side check, dependency note, testing item 12) and facts/openapi.yaml, and recorded the age once in step 4 rather than restating a version at each site. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two Fixed entries, placed above the mempool one because this defect is older than the project's entire release history rather than new in this cycle. The first says plainly that external mining could never find a block in any release before this one, names the releases (v0.1.0 through v0.7.11), and states what was NOT affected — block validation, consensus, solution validation — because an operator reading "mining was broken" needs to know whether to distrust their chain state. They do not. It also tells anyone who pointed a miner at this node and blamed their own hardware or pool that they were wrong to. That is the only user-facing consequence we can actually make good on, and the failure mode gave them nothing to go on: a well-formed candidate a real miner parses and accepts, and then silence. The second covers the openapi schema errors found while tracing it — b typed as a string against a wire that has always sent a bare number, which broke generated clients outright rather than degrading them. Note the section header still reads 2026-08-13 and today is the 15th. Left alone: the tag does not exist yet, so the date is whatever the release commit ends up being, and guessing it here would just be wrong differently. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contracts for the debconf first-install work. Design of record is the 2026-08-11 spec; these are what the code must do. facts/config.md is new. Layered /etc/ergo-node/conf.d/ — package defaults, debconf output, operator file — merged in lexical order. Nothing under conf.d is a dpkg conffile, which removes the confold/confnew problem rather than managing it. Merge rules: scalars later-wins PER KEY not per table, bare arrays replace and discard prior _add, _add appends. The per-key rule is called out because the naive implementation passes every test where the later file sets every key of a table and fails the moment someone sets one. Two things it pins that were not obvious: - The merged result is a DOCUMENT, and it is parsed twice — once into RootConfig, once into enr_p2p's Config. enr_p2p::config::Config::load takes a path and reads the file itself, so the p2p half would ignore every conf.d layer while the node half honoured them: one process configured two ways. facts/p2p-node.md gains from_toml_str for this, with load() redefined in terms of it so they cannot drift. Serialising the merged document to a temp file to satisfy a path-shaped API is the adapter CLAUDE.md forbids, and the contract says so where someone would reach for it. - The conffile migration is not tidiness. Without moving an existing /etc/ergo-node/ergo.toml to 99-local.toml first, shipping 00-defaults.toml layers a network default over every operator's hand-tuned file and flips working mainnet nodes to testnet on upgrade. Memory profiles are deleted from the design, not deferred. small/standard/large plus nine knobs predates v0.8.0 deriving all nine from the ceiling, so a profile is a second and worse answer to a settled question — and a fixed number against a floor that grows with the chain (chainIndexBytes 9.7 MB at 201k headers, 148 MB at 1.85M). The Memory topic is now one question: how much of this machine may the node have, blank means derive. facts/memory.md gains the startup floor. Refuse under a 3 GiB ceiling in UTXO mode, warn under 4 GiB, warn again when usable lands under the measured 3.02 GB cold-sync peak. Three deliberate choices: - It keys on the resolved ceiling, not MemAvailable. MemAvailable moves with page cache, so keying on it means a node that started at boot refuses after a busy hour for a reason the operator cannot see. - UTXO only. 4 GiB is where the AVL prover tree lives; digest and light would be refused on exactly the boxes they exist for. - ignore_memory_floor downgrades it to a warning, because a rule-of-thumb floor with no override turns our estimate into somebody else's outage. The second warning exists because ceiling and usable diverge by source: 4 GiB with no cgroup is MemTotal at 50%, so 2 GiB usable — passes the ceiling check and is the configuration most likely to struggle. And the 831k-block zero-OOM run at 3 GiB was a CGROUP ceiling, 90%, 2.7 GB usable. It is evidence that a 3 GiB stated budget works, not that 3 GB of RAM does. Recorded that way so it cannot be cited as the other thing. Marked as a precondition against the file's own "not a memory limiter" non-goal, which it would otherwise appear to contradict. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`load(path)` was the only way to get a `Config`, which made the file on disk the unit of configuration. The layered /etc/ergo-node/conf.d/ in facts/config.md merges several files into one effective config that exists nowhere on disk, so a path-shaped API cannot express it. Adds `Config::from_toml_str(&str)` per facts/p2p-node.md, with `load` redefined as `from_toml_str(read_to_string(path)?)`. `load`'s signature and behaviour are unchanged; src/main.rs keeps calling it. The split is not trivial — `load` did three validations after the parse (at least one listener, at least one seed peer, min_peers <= max_peers) and all three moved into `from_toml_str`. A `from_toml_str` that skipped one would hand the caller a config the node was never meant to run with, and no error saying so. Since `load` now delegates rather than duplicating, that failure mode is structurally unavailable rather than merely tested for. Four tests. The equivalence one writes a fixture exercising every section to a temp file and loads it both ways; `Config` derives `Debug` but not `PartialEq`, so it compares the derived `Debug` rendering — that covers every field of every nested struct, which is a full structural comparison, without adding a trait the crate does not otherwise need. Equivalence is asserted on the error side too, and each case asserts the expected message rather than just Err-vs-Err. Without that the test would pass if both entry points had failed in the *parser* instead, which is not the equivalence being claimed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pre-existing clippy::doc_lazy_continuation on the doc comment. Unrelated to the
config change, hence its own commit.
Clippy's suggested autofix — indent the line — would be wrong: it folds "Also
bounds per-peer buffering" into the last bullet ("Inv / Request / Sync / Peers:
kilobytes"), which is a different claim about a different thing. Took clippy's
other suggestion and added the blank line, so it reads as the trailing
paragraph it always was.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Implements facts/config.md and facts/memory.md § "Startup floor". Nothing under debian/ yet — this is the half the packaging will write into. A base ergo.toml and a sibling conf.d/ merge into one document. Either half alone is valid: a base file with no conf.d is a tarball install, a conf.d with no base file is where .deb installs end up after the conffile migration. All three existing load sites go through it, including the two maintenance subcommands, which were each reading and parsing the file themselves. The merged result is a DOCUMENT, and both parsers now consume that same string — RootConfig here, enr_p2p's Config via from_toml_str (added this morning in p2p/). Handing p2p a path would have left it reading one file and silently ignoring every layer, i.e. one process configured two ways. Worth recording that from_toml_str was NOT a trivial split: load() carried three post-parse validations, and the p2p session moved all three rather than duplicating the parse. Merge rules per the contract: scalars later-wins per key, bare arrays replace wholesale (deliberately discarding prior _add contributions, because an operator writing a bare array is stating the complete list), _add appends. One bug found by its own test, and it is the interesting one. A table the accumulator has not seen yet was being inserted VERBATIM, so a nested seed_peers_add in the first file to mention [outbound] went straight through to serde and was dropped without a word. Every later file took the recursing branch and behaved correctly, so the three obvious _add tests all passed — only the no-prior-field case exposed it. Unseen tables now merge into a fresh table rather than being inserted, so _add is processed at every level on first sight. The floor: refuse under a 3 GiB ceiling in UTXO mode, warn under 4 GiB, warn separately when USABLE lands under the 3.02 GB measured cold-sync peak. It runs before the data directory is created — a node that cannot finish should not leave half a database behind. Three choices worth stating, since each has a plausible-looking alternative: - Keyed on the resolved ceiling, not MemAvailable. MemAvailable moves with page cache, so a node that came up at boot would refuse after a busy hour, for a reason the operator cannot see and did not cause. - UTXO only. 4 GiB is where the AVL prover tree lives; digest runs the verifier and light holds no tree, so refusing them blocks the modes a small box should actually be running. - ignore_memory_floor exists. A rule-of-thumb floor with no override turns our estimate into somebody else's outage. The second warning is not redundant with the first. Ceiling and usable diverge by source: 4 GiB with nothing stating a budget is MemTotal at 50%, so 2 GiB usable — it clears the ceiling check and is the configuration most likely to struggle. Its message says the actionable thing, that declaring the same RAM via MemoryMax or memory_budget_mb is worth 90-100% instead of 50%. A test pins that divergence so the two checks cannot be collapsed into one later. 20 tests. Workspace 1093 passed / 0 failed / 3 ignored, from 1069. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… ports The shipped man page has been telling operators the REST API defaults to 0.0.0.0:9052 on mainnet and 0.0.0.0:9053 on testnet. It is the other way round, and has been since v0.6.10. 2e08971 corrected man/ergo-node-rust.8.md and did not re-run man/build, so the .gz — which is what actually gets installed — has not moved since 8f4b6aa, the v0.6.0 release. The source was right and the artifact was wrong for every release in between. No content change here beyond regenerating from the markdown that was already correct. This is the same inversion the debconf design catalogues in install.sh, and a fifth copy of it that the design did not list, because it is not a place anyone would think to look for a hardcoded port: it is a build artifact. Checking generated files into git buys a deb build with no pandoc dependency, and costs a copy that no grep of the sources will ever show as stale. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The .deb shipped a testnet config as the system default and gave a fresh
operator no indication anything was wrong. The fix is not a corrected value in
one file: "what a first config looks like" was written down in four places
that had already drifted, and a debconf template hardcoding ports and seeds
would have become the fifth.
So the network-dependent values get one home. deploy/defaults/{mainnet,
testnet}.toml ship to /usr/share and are copied to conf.d/00-defaults.toml on
every configure — not conffiles, deliberately, which is what lets a corrected
seed-peer list reach existing installs instead of being frozen forever.
install.sh reads the same files; its port inversion is fixed by deleting the
code that had it rather than by correcting the numbers.
The interview is two unconditional screens — network, then a topic checklist —
and then only the questions belonging to checked topics. Everything has a
default, so noninteractive installs complete and get mainnet. Mining requires
utxo, so when Mining is checked digest is simply not offered: no error path for
a combination that means something more fundamental was misunderstood.
There are no memory profiles. The node derives every cache and threshold from
its ceiling at runtime, against a floor that grows with the chain, so a number
frozen at install time is correct on the day and wrong after. The Memory topic
is one question, and a blank answer writes NOTHING — writing memory_budget_mb
"helpfully" would switch derivation off. A low-RAM notice fires under 4 GB
before the topic questions, so a small box can pick light or digest in the same
pass instead of learning from a failed start.
Three bugs found by running it rather than reading it:
- debconf returns a multiselect as "A, B, C" — comma AND space. Matching on the
comma alone honoured only the FIRST topic and silently dropped the rest,
producing a plausible config file with most of the operator's answers
missing.
- The seed-peer split used printf '%s' with no trailing newline, so `read`
returned non-zero on the final field and the loop exited before emitting it.
Every list lost its last peer.
- The systemd unit passed /etc/ergo-node/ergo.toml explicitly, and preinst
MOVES that file to conf.d/99-local.toml. The service would have failed to
start on the first upgrade after this change. ExecStart now takes no path and
WorkingDirectory is pinned so the search order is deterministic.
The migration runs in preinst, not postinst as designed. Dropping a conffile
makes dpkg decide the file's fate during unpack — before postinst — and for an
unmodified file that decision is deletion. preinst sees the operator's file in
the state they left it. Verified byte-identical after the move, and idempotent:
a second run leaves 99-local.toml alone even when a new legacy file appears.
Verified by generating a config through the real postinst with a stubbed
debconf and merging the layers: 11 shipped mainnet seeds plus 2 added = 13,
_add appended rather than replacing, min_peers/max_peers surviving a later
file that sets other keys, and no memory_budget_mb written for a blank answer.
Package builds with config 755, templates 644, and the old conffile gone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… 0.1.0 Nine crates carried `version = "0.1.0"`, each set once when it was absorbed from a submodule and never moved again — through eight releases. They are internal path dependencies, never published, and nothing pins a version on them, so the number was decoration that disagreed with the thing it looked like it described. `cargo metadata` reported a 0.1.0 workspace for a 0.8.0 node. Deleting the field would have been worse than leaving it: Cargo then defaults the package to 0.0.0, which is the same problem with a rounder number. So the version becomes a single literal in [workspace.package] and every member takes `version.workspace = true`. A release bump is now one line, and cargo metadata reports 0.8.0 for all ten packages. The excluded addons deliberately do NOT inherit — fastsync 0.1.8 and indexer 0.2.8 ship on their own cadence with their own lockfiles. Noted in the manifest so the asymmetry does not read as an oversight and get "fixed". build-deb's version extraction had to move with it. `grep '^version' | head -1` was correct only by accident of section ordering, and would have silently produced "true" from `version.workspace = true` had anyone reordered the file — yielding a package named ergo-node-rust_true_amd64.deb. It now reads the literal out of [workspace.package] specifically, and hard-fails if what it finds is not a version number. Also set debian/control's Version to 0.8.0. build-deb substitutes it at build time so the shipped value was always right, but the file said 0.1.0, which made every naive read of it wrong. The explanation for that lives in build-deb rather than in the file: debian/control is a binary package control file and dpkg-deb rejects `#` comments outright — found by trying it. Verified: cargo metadata 0.8.0 across all ten packages, workspace builds, 1093 passed / 0 failed / 3 ignored, fmt clean, and build-deb produces ergo-node-rust_0.8.0_amd64.deb whose binary reports 0.8.0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mwaddip
marked this pull request as ready for review
August 15, 2026 12:22
Odiseus restarted a node that was already at tip and it served 503 from /mining/candidate for over an hour with three peers connected, until peers delivered a new block. Where the restarted node is the only miner it does not recover at all: no candidate, so no block, so no application, so no candidate. Reproduced on e77959d, so it predates the b fix and is independent of it. His diagnosis — main's mining proof cache starts None and is only written from the block-application path — is correct, and seeding it would NOT have fixed the bug. emission_box_id has the same defect one level down: utxo.rs:99 sets it to None at construction and the only site that populates it is the insertion scan inside apply_state. update_mining_proofs returns early at the emission_box_id check, so the symptom would have been identical and the diagnosis would have looked wrong. So the defect is that UtxoValidator reconstructs none of its derived mining state on resume. Two fields, one root: maintained incrementally, never recovered. facts/validation.md gains the recovery requirement, and says three things that are easy to get wrong: - Recover from the tip block, not a UTXO-set scan. The emission box is spent and recreated by the coinbase of every block that has one, so the resume height's own insertions contain it — one block to parse rather than millions of boxes. - Reuse the existing apply-time scan rather than writing a second one. Two implementations of "which box is the emission box" is the divergence the contract exists to prevent. - Do NOT persist the ID next to the state digest. It is derivable from authoritative state, and a stored copy is a second source of truth that can disagree with the tree it describes. None after a successful recovery stays correct and still means emission has ended. A recovery that cannot read its input is a different case and must say so rather than leaving None, because None is indistinguishable from the legitimate value — which is the whole reason this went unnoticed. facts/mining.md gains the startup seeding requirement and an explicit warning that seeding alone is insufficient, so nobody implements half of this and concludes the report was wrong. Also recorded why it survived: the gap is invisible during sync. A catching-up node applies a block within seconds and the cache fills itself. Only an at-tip restart exposes it, and until this release external mining did not work at all, so nobody had reason to restart a mining node and look. Generalised in the resume section: any field maintained incrementally by apply_state is suspect on resume — ask what its value means before the first block, and whether that value is distinguishable from a legitimate one. Implementation follows: validation/ recovers the field, main seeds the cache. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Installed the package in a throwaway Ubuntu 24.04 VM and drove the interview by hand. It completed — network, topic checklist, all seven topic sets in order — and got three things wrong that reading the scripts had not caught. The memory notice talked the operator out of the mode they wanted. It fired below 4096 MB, which is the RECOMMENDATION, but its first listed remedy was "pick light or digest" — advice that only makes sense below 3072 MB, where the node actually refuses to start. On a 3915 MB machine it announced "less memory than a full node needs" and the operator duly chose light, on a box where a full node runs fine. A warning that changes a correct decision into a worse one is worse than no warning. Now two notices. Below 3 GB it steers, because the node genuinely will not start. Between 3 and 4 GB it says a full node WILL start and run and that no action is needed, then points at the lever that actually matters: with nothing stating a budget the node assumes it may use only about half of total RAM, and MemoryMax roughly doubles that on the same hardware. max_inbound was never asked, despite the operator selecting the Interfaces topic. It was `db_input medium` and debconf's default priority is high, so it was silently filtered. A question inside a topic the operator explicitly chose must not be priority-gated — the whole point of the checklist is that checking a topic means being asked about it. The Node type screen described digest while offering only utxo and light. The exclusion was right — Mining was selected and only utxo can mine — but the description did not move with the choice list, so it reads as a broken menu rather than a deliberate exclusion, and sends the operator hunting for an option that is not there. The description is now substituted alongside the choices and says why digest is absent and how to get it back. The notice text also overflowed the dialog and cut off mid-sentence at "accepting the". Both are shorter now. None of these were reachable by reading. The priority filter and the debconf seen-flag semantics only show up against a real frontend, and the memory notice's effect only shows up on someone who has not read the code and simply does what the screen tells them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…t block
`emission_box_id` was derived state written only by step 9 of `apply_state`,
so a freshly constructed `UtxoValidator` reported `None` — the documented
value for "all ERG has been emitted". `main`'s `update_mining_proofs` returns
early on it, so a node restarted at the chain tip served no mining candidate
at all until a peer delivered the next block. For a solo miner that is a
deadlock rather than a delay: no candidate, so no block, so no application,
so no candidate. Reported from the field after an at-tip restart sat at 503
for over an hour.
The reason it survived this long is that `None` is indistinguishable from the
legitimate answer, and during sync the gap closes by itself within seconds.
Only an at-tip restart exposes it, and until this release external mining did
not work at all, so nobody had reason to restart a mining node and look.
`UtxoValidator::new` now takes a required `EmissionSource` and recovers the
field before returning. Required, not optional, because forgetting it is the
defect — there is no second call to remember and no defaulted method a
wrapper can silently swallow the way the enum wrapper in `src/main.rs`
swallowed `resize_cache`.
TipBlock(&[u8]) BlockTransactions section bytes at the resume
height. The coinbase spends and recreates the
emission box every block, so one block's
insertions carry it — not a UTXO-set scan,
which would be O(millions of boxes) at startup.
GenesisBoxes(&[Insertion]) The boxes a genesis bootstrap just inserted,
where no block exists yet.
Unavailable(&str) Neither is obtainable; the reason is logged.
The variants are deliberately not an `Option`. "No block exists yet" and "I
could not read the block" are different facts with different consequences,
and collapsing them into one `None` is the exact overload being removed —
reproducing it in the fix's own API would have been the joke that writes
itself.
`find_emission_box` is extracted and is now the single implementation of
"which box is the emission box", called by both step 9 and recovery. Two
copies of that match is the divergence facts/validation.md exists to prevent,
and if they disagreed the node would mine against the wrong box. The
equivalence is asserted, not commented: a validator recovered at the tip must
name the same box one that applied that block names.
Recovery routes through `compute_state_changes`, so intra-block netting and
ordering match the apply path by construction rather than by a second
derivation that could drift.
`None` after a source that read cleanly still means emission has ended and is
logged at INFO. A source that could not be read is logged at WARN, because
that `None` is byte-identical to the legitimate one and an unlogged one is
how this hid. Construction does not fail either way: an unreadable tip block
costs mining until the next block, which is no reason to refuse to start a
node that can still validate, serve and sync.
Also adds `Insertion` as the name for the `(box_id, serialized_box)` pair
that now appears in three signatures, and `tracing-test` with `no-env-filter`
— without that feature the default filter drops the emits and an "and warns"
assertion passes on an empty buffer.
ergo-validation: 73 passed / 0 failed (was 68).
The signature change breaks `src/main.rs` at four call sites (2965 resume,
3038 genesis, 3489 snapshot bootstrap, 4773 test); wiring them is main's.
Adding the dev-dependency also dirties the root `Cargo.lock` by one line,
left uncommitted as a repo-root file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The other half of the at-tip restart deadlock. 19083a8 made `emission_box_id` valid on a fresh `UtxoValidator`; this wires main to it and seeds the proof cache that `update_mining_proofs` otherwise only ever writes from the post-apply path. Reproduced and fixed, not just reasoned about. Against an isolated copy of a parked mainnet tip at height 1,851,751, with a single unroutable seed (192.0.2.1, RFC 5737) so nothing can arrive and apply a block and fill the cache by itself — which would have masked the defect: /usr/bin (v0.8.0 as deployed) /mining/candidate 503 for 40s this build candidate at h=1851752, 2s after API up with `recovered emission box at construction height=1851751 emission_box_id=8244c77a...` in the log between them. The same response also shows b as a 40-digit target rather than a 10-digit difficulty, so both mining fixes are visible in one run. Four call sites, because EmissionSource is a required parameter rather than something to remember — the validation session's choice, and the right one: - resume: reads the BlockTransactions section at the resume height out of the store. section_ids()[0] is that section. Falls back to Unavailable with a reason if the store lacks it, since an unexplained None here is byte-identical to the legitimate "all ERG emitted". - genesis: build_genesis_boxes() is now bound to a local and iterated by reference rather than consumed, so the same three boxes can be handed to the validator. At height 0 there is no block to read. - snapshot bootstrap: Unavailable. It installs a state tree at a height whose block transactions were never downloaded. Harmless today — mining ctx is None there — and it stops being harmless the moment that TODO is done, which is why it states a reason instead of leaving a bare None. - the trait-surface test: Unavailable, empty tree, not what it tests. Seeding runs AFTER the cross-DB reconciliation handshake, deliberately. That block can reset_to a lower height, and seeding first would cache proofs for a tip the validator has since rolled back off — which the mining task would then discard silently on the tip_height comparison, i.e. the same 503 with an extra step. Workspace 1098 passed / 0 failed / 3 ignored, from 1093 (+5 from validation). fmt and clippy clean. Cargo.lock carries validation's one-line tracing-test entry; repo root, so it lands here. Reported by Odiseus, who also established it predates the b fix by reproducing it on e77959d. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… summary The CHANGELOG covered the mempool and mining fixes and nothing else from this cycle — no conf.d layering, no debconf, no memory floor, no at-tip restart fix, no version inheritance. Merging the release PR tags and builds, so an incomplete entry ships as the release notes. Adds those, and one section that changes what operators actually read: ### Release summary — one line per change, no reasoning. The release workflow now extracts this subsection alone for the tag annotation, which is what the GitHub release body is sourced from. Previously it took the whole CHANGELOG section, and this project's entries carry their reasoning inline, which is right for the durable record and wrong for a release page: 23 lines of "what changed" versus several screens of why. Falls back to the whole section when the subsection is absent, so re-tagging any earlier release produces exactly the body it always did — verified against v0.7.11. Also hardens the workflow's own version read. It used `grep -m1 '^version = '`, which was correct only by accident of section ordering once members moved to `version.workspace = true`; a reordered manifest would have tagged something other than the release. Now scoped to [workspace.package] and hard-fails on anything that is not a version number, matching build-deb. This is the step that decides what gets tagged, so it should not be the one relying on luck. The migration entry records that it was verified as an in-place upgrade of a live mainnet node at tip, not only in a VM: migrated file byte-identical, every configured value surviving the merge, service back up on the merged document. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this release does
Script evaluation is inline.
apply_stateevaluates a block's scripts after the state-root check and before persisting.Okmeans the scripts passed.The node sizes its own memory. Every memory key is optional and derived from the cgroup limit (
MemoryMax, container--memory), or a conservative share ofMemTotal. A key that is set is obeyed exactly and disables derivation for that key alone. New optionalmemory_budget_mb. The derivation is logged at startup asmemory budget derivedwith every input and output.Mining assembles full blocks. Candidates carry mempool transactions selected under both
max_block_costand the serializedBlockTransactionssection size, plus a fee transaction whose reward box is capped by measured serialized size. Candidates are rebuilt on TTL expiry, so work stays available between blocks.BlockValidatoris three traits. The consensus core, plusStatePersistence(flush,resize_cache) andMiningState(proofs_for_transactions,emission_box_id), both implemented only by the UTXO validator. Digest mode signals itself by not implementing them.proofs_from_storagecomputes AD proofs from a read-onlySnapshotReaderwithout a validator.The mempool works. Unconfirmed transactions are validated against an upcoming state context whose preheader describes the next block, and the fee is read from outputs guarded by the fee proposition. A creation height above the preheader is transient and declines rather than invalidating. Before this, a synced node rejected every transaction the network offered it and held a permanently empty pool.
Breaking changes
script_eval,eval_backlog_max_mb,eval_backlog_max_blocksare removed. A config that still sets them fails to start with an unknown-key error. Config sections now reject unknown keys.cache_mbis the total redb page cache across both databases, split by the newcache_store_pct(default 50). It previously sizedstate.redbalone.journalEventsVersionin/info).deferred_eval_backlog(replaced bycatchup_progress),deferred_eval_gate_engagedandeval_frontier_holeare removed;validation_rollback_failed.pathloseseval_failure;validation_stuck.error_kindlosesscript_evalin favour oftransaction_invalid. Consumers pinned to major 1 must update.ergo_avltree_rustfork pin moves tob955790.Known limitations
flush_max_blocksandflush_min_blocksare not derived and keep their existing defaults.CI
main, covering the workspace and both addons.release/*PR resolves the version fromCargo.toml, creates the annotated tag and builds the release.Status
Workspace: 1050 passed, 0 failed, 3 ignored. Formatting clean.
Verified on live mainnet at tip: the mempool fills and drains with each block, and mining candidates reached 47 transactions across block boundaries.
A cold sync from genesis on a 2 vCPU / 4 GB host under a 3 GB cgroup limit is in progress and is the gate on merging this.