fix: abort chain compaction cleanly on node shutdown#3895
Conversation
Compaction could keep rewriting PMMR files after stop was requested, and killing the process mid-replace led to Invalid Root on next launch. - Thread StopState into compact from NetToChainAdapter and syncer - Skip starting compact when already stopping - Before replacing live PMMR files, if stopping, discard .tmp files and leave the existing chain data untouched Addresses mimblewimble#3842.
wiesche89
left a comment
wiesche89
left a comment
There was a problem hiding this comment.
Good direction, but shutdown still needs to wait for the replace phase, and the test should exercise the actual tmp discard path.
| self.data_file.write_tmp_pruned(&pos_to_rm)?; | ||
| } | ||
|
|
||
| // Critical section: do not replace live files if we are shutting down. |
There was a problem hiding this comment.
A stop can arrive immediately after this check, while the files are replaced separately. Is shutdown guaranteed to wait until the whole replace section finishes?
| debug!("compactor: not starting, node is stopping"); | ||
| return; | ||
| } | ||
| if let Err(e) = chain.compact_with_stop(Some(stop_state)) { |
There was a problem hiding this comment.
This compactor still runs on a detached thread, so shutdown cannot wait if it has already entered the replace phase. Could the thread be tracked and joined?
|
|
||
| // Always abort: should not replace live files. | ||
| let done = backend | ||
| .check_compact_until(2, &Bitmap::new(), || true) |
There was a problem hiding this comment.
|| true trips the first check before any tmp file is written, so the discard path never runs. Could it return false first, then true, and verify that no tmp files remain?
| /// * removes historical blocks and associated data from the db (unless archive mode) | ||
| /// | ||
| pub fn compact(&self) -> Result<(), Error> { | ||
| self.compact_with_stop(None) |
There was a problem hiding this comment.
The owner API still reaches this uncancellable path. Could an API-triggered compaction hit the same shutdown problem?
| } | ||
|
|
||
| /// Drop any `.tmp` companion file without replacing the live file. | ||
| pub fn discard_tmp(&self) { |
There was a problem hiding this comment.
Could this return io::Result<()>? Otherwise the caller reports a clean abort even when tmp cleanup fails.
| /// Compact backend files, optionally aborting before live files are replaced | ||
| /// when `should_abort` returns true (e.g. node shutdown). Aborted compaction | ||
| /// discards `.tmp` files and leaves the live PMMR files untouched (#3842). | ||
| pub fn check_compact_until<F>( |
There was a problem hiding this comment.
Small naming thought, until sounds like a position limit. Would check_compact_with_abort describe this callback more clearly?
| teardown(data_dir); | ||
| } | ||
|
|
||
| /// Aborting compaction before replace must leave live files intact (#3842). |
There was a problem hiding this comment.
Could we keep this focused on the invariant and leave the issue reference in the PR?
2 participants
Summary
Fixes #3842.
Long compaction runs (especially on low-RAM machines / startup) continued after the node was asked to stop. Killing the process while live PMMR files were being replaced left the chain with Invalid Root on the next launch.
Changes
NetToChainAdapterStopState; does not spawn compactors when stopped; passes stop into compactSyncercompact_with_stopon transition to NoSyncChain::compact_with_stopPMMRBackend::check_compact_until.tmpfiles, if aborting: discard tmp, do not replace live filesAppendOnlyFile::discard_tmp.tmpcompanionsChain::compact()remains for API/tests (no stop handle).Test plan
cargo test -p grin_store --test pmmr pmmr_compact_abort_before_replacecargo test -p grin_store --test pmmr -- --test-threads=1cargo test -p grin_chain --test mine_simple_chain compact -- --test-threads=1cargo test -p grin_servers --lib -- --test-threads=1