feat(mediorum): cap ops pruning at the active-peer cursor + retention-gap signal#304
feat(mediorum): cap ops pruning at the active-peer cursor + retention-gap signal#304RolfAris wants to merge 2 commits into
Conversation
RolfAris
force-pushed
the
feat/mediorum-ops-retention-minimal
branch
from
afe1a61 to
6e59e39
Compare
RolfAris
changed the title
|
Rollout note from live fleet readiness on 2026-05-21:
Recommendation: canary #304 with the index gate left off first. Only enable |
|
Canary update from val005:
Implication: PR304 is not a physical disk-space reclamation fix by itself. It is now safe enough to continue canary observation with the expensive ops cleanup paths default-off, but actual filesystem space reclaim still needs a separate controlled |
|
Final val005 canary/fold result from the 20-validator fleet, 2026-05-22 UTC:
Operator implication: this PR is a useful safety/observability building block when the expensive paths are default-off, but it is not disk relief by itself. For production operators, the next upstream bar is either proven bounded growth or actual physical reclaim semantics. DELETE-only retention is not enough unless paired with a story that returns filesystem space; partition detach/drop, a safe table swap, or snapshot/bootstrap plus partitioned retention are the shapes that would close the operator problem. |
|
Thanks for this @RolfAris — there's real correctness work here, and I want to be clear up front that we're not closing this. The gap-signal + cursor-floor design is the right long-term direction and we want to come back to it. For the immediate situation, though, we're going to ship #325 first as a more targeted fix, with default pruning enabled. Reasoning / sequencing: The fire is disk pressure on live
So out of the box this PR is correctness-additive but doesn't reduce the footprint until an operator sets the retention flag. #325 is on-by-default at a 90-day window and prunes the actively-written tables directly, which is what we need to stop the bleeding now. It's ~130 LoC, easy to review and revert. Where your PR is genuinely better, and why we want it later: #325 handles a still-active peer that's >90 days behind (or a fresh node bootstrapping) by documentation + config, not by protocol — it can let such a peer silently skip ops. Your retention-gap signal ( Proposed sequencing:
One small thing for the record: the PR description says dormant cleanup is opt-out/on-boot, but the code gates it behind Tracking PR: #325. |
…-gap signal OpenAudio#325 prunes the crudr ops log purely by age, which can let a still-active peer behind the retention window (or one bootstrapping from genesis) silently skip the pruned range. Add the protocol safety OpenAudio#325 deferred: - pruneOps caps its cutoff at min(time_cutoff, slowest_active_peer_cursor), so it never deletes ops a still-active peer has not yet swept. Ignores self, decommissioned peers, and non-peer cursor rows (e.g. qm_fix_truncated) that would otherwise pin the floor. - ServeCrudSweep advertises the lowest available ULID when a peer sweeps from below it (X-Mediorum-Retention-Gap / X-Mediorum-Available-Min-Ulid), so the peer advances across the gap explicitly instead of silently skipping it, with a first-contact guard and implausible-ULID rejection. Supersedes the original dormant-cleanup + opt-in retention sweep, which OpenAudio#325 made redundant.
RolfAris
force-pushed
the
feat/mediorum-ops-retention-minimal
branch
from
35e3045 to
7ef282b
Compare
RolfAris
changed the title
|
Reshaped per your sequencing above. Dropped the one-time dormant cleanup and the opt-in retention sweep (both redundant once #325 prunes on-by-default), and layered the two pieces you wanted directly onto #325's
~2,200 → ~120 LOC over four files + a focused floor test. The dormant-cleanup doc/code mismatch you flagged is gone with the component. Build/vet/ |
2 participants
Summary
#325 prunes the crudr
opslog purely by age (ulid < now - retention). As noted there, a still-active peer behind the retention window — or one bootstrapping from genesis — can silently skip the pruned range. This adds the protocol-level safety #325 deferred to this PR:pruneOpscaps its cutoff atmin(time_cutoff, slowest_active_peer_cursor), so it never deletes ops a still-active peer hasn't swept. Caught-up peers sweep ~every 10 min (cursor far newer than any month/year cutoff), so this is a no-op in steady state; it only restricts pruning when an active peer is legitimately behind. The floor ignores self, decommissioned peers, and non-peer worker rows (e.g.qm_fix_truncated, which stores a CID under its own host) that would otherwise pin it.serveCrudSweepadvertises the lowest available ULID (X-Mediorum-Retention-Gap/X-Mediorum-Available-Min-Ulid). The peer advances its cursor across the gap explicitly instead of silently skipping it, with a first-contact guard and implausible-ULID rejection. Wire format unchanged; older clients ignore the headers.Supersedes this PR's original contents (one-time dormant cleanup + opt-in per-table retention sweep), which #325 made redundant — the dormant cleanup never drained the actively-written tables, and #325's pruner replaces the opt-in sweep so there aren't two systems.
+121/−1 across four files plus a focused floor test.
Tests
go test ./pkg/mediorum/crudr—TestSlowestActivePeerCursor{,_NoActivePeers,_IgnoresImplausibleActiveCursor}cover the floor: slowest active peer wins; self, decommissioned, non-peer, and implausible cursors are ignored.go build ./pkg/mediorum/...andgo vetclean.