diff --git a/ci/dash/lint-tidy.sh b/ci/dash/lint-tidy.sh index d7d63656d842..757009610a9c 100755 --- a/ci/dash/lint-tidy.sh +++ b/ci/dash/lint-tidy.sh @@ -90,6 +90,7 @@ iwyu_tool.py \ "src/init" \ "src/kernel" \ "src/node/chainstate.cpp" \ + "src/node/mempool_args.cpp" \ "src/node/minisketchwrapper.cpp" \ "src/policy/feerate.cpp" \ "src/policy/packages.cpp" \ diff --git a/doc/design/assumeutxo.md b/doc/design/assumeutxo.md index 74aff7d392f8..9846f7f26a0b 100644 --- a/doc/design/assumeutxo.md +++ b/doc/design/assumeutxo.md @@ -3,9 +3,9 @@ Assumeutxo is a feature that allows fast bootstrapping of a validating dashd instance with a very similar security model to assumevalid. -The RPC commands `dumptxoutset` and `loadtxoutset` are used to respectively generate -and load UTXO snapshots. The utility script `./contrib/devtools/utxo_snapshot.sh` may -be of use. +The RPC commands `dumptxoutset` and `loadtxoutset` (yet to be merged) are used to +respectively generate and load UTXO snapshots. The utility script +`./contrib/devtools/utxo_snapshot.sh` may be of use. ## General background @@ -17,14 +17,9 @@ be of use. - A new block index `nStatus` flag is introduced, `BLOCK_ASSUMED_VALID`, to mark block index entries that are required to be assumed-valid by a chainstate created - from a UTXO snapshot. This flag is mostly used as a way to modify certain + from a UTXO snapshot. This flag is used as a way to modify certain CheckBlockIndex() logic to account for index entries that are pending validation by a - chainstate running asynchronously in the background. We also use this flag to control - which index entries are added to setBlockIndexCandidates during LoadBlockIndex(). - -- Indexing implementations via BaseIndex can no longer assume that indexation happens - sequentially, since background validation chainstates can submit BlockConnected - events out of order with the active chain. + chainstate running asynchronously in the background. - The concept of UTXO snapshots is treated as an implementation detail that lives behind the ChainstateManager interface. The external presentation of the changes @@ -76,9 +71,15 @@ original chainstate remains in use as active. Once the snapshot chainstate is loaded and validated, it is promoted to active chainstate and a sync to tip begins. A new chainstate directory is created in the -datadir for the snapshot chainstate called `chainstate_snapshot`. When this directory -is present in the datadir, the snapshot chainstate will be detected and loaded as -active on node startup (via `DetectSnapshotChainstate()`). +datadir for the snapshot chainstate called `chainstate_snapshot`. + +When this directory is present in the datadir, the snapshot chainstate will be detected +and loaded as active on node startup (via `DetectSnapshotChainstate()`). + +A special file is created within that directory, `base_blockhash`, which contains the +serialized `uint256` of the base block of the snapshot. This is used to reinitialize +the snapshot chainstate on subsequent inits. Otherwise, the directory is a normal +leveldb database. | | | | ---------- | ----------- | @@ -88,7 +89,7 @@ active on node startup (via `DetectSnapshotChainstate()`). The snapshot begins to sync to tip from its base block, technically in parallel with the original chainstate, but it is given priority during block download and is allocated most of the cache (see `MaybeRebalanceCaches()` and usages) as our chief -consideration is getting to network tip. +goal is getting to network tip. **Failure consideration:** if shutdown happens at any point during this phase, both chainstates will be detected during the next init and the process will resume. @@ -107,33 +108,36 @@ sequentially. ### Background chainstate hits snapshot base block Once the tip of the background chainstate hits the base block of the snapshot -chainstate, we stop use of the background chainstate by setting `m_stop_use` (not yet -committed - see bitcoin#15606), in `CompleteSnapshotValidation()`, which is checked in -`ActivateBestChain()`). We hash the background chainstate's UTXO set contents and -ensure it matches the compiled value in `CMainParams::m_assumeutxo_data`. - -The background chainstate data lingers on disk until shutdown, when in -`ChainstateManager::Reset()`, the background chainstate is cleaned up with -`ValidatedSnapshotShutdownCleanup()`, which renames the `chainstate_[hash]` datadir as -`chainstate`. +chainstate, we stop use of the background chainstate by setting `m_disabled`, in +`MaybeCompleteSnapshotValidation()`, which is checked in `ActivateBestChain()`. We hash the +background chainstate's UTXO set contents and ensure it matches the compiled value in +`CMainParams::m_assumeutxo_data`. In Dash, completion additionally compares the +deterministic masternode-list hash the background chainstate derived at the base block +against the hash recorded at snapshot activation, and the EvoDB best-block markers +against both chainstates' coins tips; any divergence fails completion with +`EVO_STATE_MISMATCH` and quarantines the snapshot exactly like a UTXO hash mismatch. | | | | ---------- | ----------- | -| number of chainstates | 2 (ibd has `m_stop_use=true`) | +| number of chainstates | 2 (ibd has `m_disabled=true`) | | active chainstate | snapshot | -**Failure consideration:** if dashd unexpectedly halts after `m_stop_use` is set on -the background chainstate but before `CompleteSnapshotValidation()` can finish, the -need to complete snapshot validation will be detected on subsequent init by -`ChainstateManager::CheckForUncleanShutdown()`. +The background chainstate data lingers on disk until the program is restarted. ### Dashd restarts sometime after snapshot validation has completed -When dashd initializes again, what began as the snapshot chainstate is now -indistinguishable from a chainstate that has been built from the traditional IBD -process, and will be initialized as such. +After a shutdown and subsequent restart, `LoadChainstate()` cleans up the background +chainstate with `ValidatedSnapshotCleanup()`, which renames the `chainstate_snapshot` +datadir as `chainstate` and removes the now unnecessary background chainstate data. | | | | ---------- | ----------- | | number of chainstates | 1 | -| active chainstate | ibd | +| active chainstate | ibd (was snapshot, but is now fully validated) | + +What began as the snapshot chainstate is now indistinguishable from a chainstate that +has been built from the traditional IBD process, and will be initialized as such. + +A file will be left in `chainstate/base_blockhash`, which indicates that the +chainstate, even though now fully validated, was originally started from a snapshot +with the corresponding base blockhash. diff --git a/src/Makefile.am b/src/Makefile.am index 30fdf4702361..fb1ffde2529a 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -324,7 +324,6 @@ BITCOIN_CORE_H = \ masternode/payments.h \ masternode/sync.h \ masternode/utils.h \ - mempool_args.h \ memusage.h \ merkleblock.h \ messagesigner.h \ @@ -346,6 +345,7 @@ BITCOIN_CORE_H = \ node/context.h \ node/eviction.h \ node/interface_ui.h \ + node/mempool_args.h \ node/mempool_persist_args.h \ node/miner.h \ node/minisketchwrapper.h \ @@ -600,7 +600,6 @@ libbitcoin_node_a_SOURCES = \ masternode/payments.cpp \ masternode/sync.cpp \ masternode/utils.cpp \ - mempool_args.cpp \ net.cpp \ net_processing.cpp \ netfulfilledman.cpp \ @@ -614,6 +613,7 @@ libbitcoin_node_a_SOURCES = \ node/eviction.cpp \ node/interface_ui.cpp \ node/interfaces.cpp \ + node/mempool_args.cpp \ node/mempool_persist_args.cpp \ node/miner.cpp \ node/minisketchwrapper.cpp \ diff --git a/src/Makefile.bench.include b/src/Makefile.bench.include index be7bfe2d2dcc..c54ad6e6b2b6 100644 --- a/src/Makefile.bench.include +++ b/src/Makefile.bench.include @@ -31,6 +31,7 @@ bench_bench_dash_SOURCES = \ bench/crypto_hash.cpp \ bench/data.cpp \ bench/data.h \ + bench/descriptors.cpp \ bench/duplicate_inputs.cpp \ bench/ecdsa.cpp \ bench/ellswift.cpp \ diff --git a/src/Makefile.test_fuzz.include b/src/Makefile.test_fuzz.include index 9ad315beb0aa..e909ab9359c1 100644 --- a/src/Makefile.test_fuzz.include +++ b/src/Makefile.test_fuzz.include @@ -12,7 +12,7 @@ TEST_FUZZ_H = \ test/fuzz/FuzzedDataProvider.h \ test/fuzz/util.h \ test/util/mining.h \ - test/fuzz/mempool_utils.h \ + test/fuzz/util/mempool.h \ test/fuzz/util/net.h libtest_fuzz_a_CPPFLAGS = $(AM_CPPFLAGS) $(BITCOIN_INCLUDES) $(BOOST_CPPFLAGS) @@ -21,5 +21,6 @@ libtest_fuzz_a_SOURCES = \ test/fuzz/fuzz.cpp \ test/util/mining.cpp \ test/fuzz/util.cpp \ + test/fuzz/util/mempool.cpp \ test/fuzz/util/net.cpp \ $(TEST_FUZZ_H) diff --git a/src/bench/descriptors.cpp b/src/bench/descriptors.cpp new file mode 100644 index 000000000000..487d5118ad1c --- /dev/null +++ b/src/bench/descriptors.cpp @@ -0,0 +1,30 @@ +// Copyright (c) 2019 The Bitcoin Core developers +// Distributed under the MIT software license, see the accompanying +// file COPYING or http://www.opensource.org/licenses/mit-license.php. + +#include +#include +#include