backport: assumeutxo M6 — hardening (bitcoin#29370 + A4 batch)

doc/assumeutxo.md

@@ -0,0 +1,85 @@
+# Assumeutxo Usage
+
+Assumeutxo is a feature that allows fast bootstrapping of a validating dashd
+instance.
+
+For notes on the design of Assumeutxo, please refer to [the design doc](/doc/design/assumeutxo.md).
+
+## Loading a snapshot
+
+There is currently no canonical source for snapshots, but any downloaded snapshot
+will be checked against a hash that's been hardcoded in source code. If there is
+no source for the snapshot you need, you can generate it yourself using
+`dumptxoutset` on another node that is already synced (see
+[Generating a snapshot](#generating-a-snapshot)).
+
+Once you've obtained the snapshot, you can use the RPC command `loadtxoutset` to
+load it.
+
+```
+$ dash-cli loadtxoutset /path/to/input
+```
+
+After the snapshot has loaded, the syncing process of both the snapshot chain
+and the background IBD chain can be monitored with the `getchainstates` RPC.
+
+### Pruning
+
+A pruned node can load a snapshot. To save space, it's possible to delete the
+snapshot file as soon as `loadtxoutset` finishes.
+
+The minimum `-prune` setting is 550 MiB, but this functionality ignores that
+minimum and uses at least 1100 MiB.
+
+As the background sync continues there will be temporarily two chainstate
+directories, each multiple gigabytes in size (likely growing larger than the
+downloaded snapshot).
+
+### Indexes
+
+Indexes work but don't take advantage of this feature. They always start building
+from the genesis block and can only apply blocks in order. Once the background
+validation reaches the snapshot block, indexes will continue to build all the
+way to the tip.
+
+
+For indexes that support pruning, note that these indexes only allow blocks that
+were already indexed to be pruned. Blocks that are not indexed yet will also
+not be pruned.
+
+This means that, if the snapshot is old, then a lot of blocks after the snapshot
+block will need to be downloaded, and these blocks can't be pruned until they
+are indexed, so they could consume a lot of disk space until indexing catches up
+to the snapshot block.
+
+## Generating a snapshot
+
+The RPC command `dumptxoutset` can be used to generate a snapshot for the current
+tip (using type "latest") or a recent height (using type "rollback"). A generated
+snapshot from one node can then be loaded
+on any other node. However, keep in mind that the snapshot hash needs to be
+listed in the chainparams to make it usable. If there is no snapshot hash for
+the height you have chosen already, you will need to change the code there and
+re-compile.
+
+Using the type parameter "rollback", `dumptxoutset` can also be used to verify the
+hardcoded snapshot hash in the source code by regenerating the snapshot and
+comparing the hash.
+
+Example usage:
+
+```
+$ dash-cli -rpcclienttimeout=0 dumptxoutset /path/to/output rollback
+```
+
+For most of the duration of `dumptxoutset` running the node is in a temporary
+state that does not actually reflect reality, i.e. blocks are marked invalid
+although we know they are not invalid. Because of this it is discouraged to
+interact with the node in any other way during this time to avoid inconsistent
+results and race conditions, particularly RPCs that interact with blockstorage.
+This inconsistent state is also why network activity is temporarily disabled,
+causing us to disconnect from all peers.
+
+`dumptxoutset` takes some time to complete, independent of hardware and
+what parameter is chosen. Because of that it is recommended to increase the RPC
+client timeout value (use `-rpcclienttimeout=0` for no timeout).

doc/design/assumeutxo.md

@@ -1,11 +1,6 @@
-# assumeutxo
+# Assumeutxo Design
 
-Assumeutxo is a feature that allows fast bootstrapping of a validating dashd
-instance with a very similar security model to assumevalid.
-
-The RPC commands `dumptxoutset` and `loadtxoutset` are used to respectively generate
-and load UTXO snapshots. The utility script `./contrib/devtools/utxo_snapshot.sh` may
-be of use.
+For notes on the usage of Assumeutxo, please refer to [the usage doc](/doc/assumeutxo.md).
 
 ## General background
 
@@ -15,23 +10,12 @@ be of use.
 
 ## Design notes
 
-- A new block index `nStatus` flag is introduced, `BLOCK_ASSUMED_VALID`, to mark block
-  index entries that are required to be assumed-valid by a chainstate created
-  from a UTXO snapshot. This flag is mostly used as a way to modify certain
-  CheckBlockIndex() logic to account for index entries that are pending validation by a
-  chainstate running asynchronously in the background. We also use this flag to control
-  which index entries are added to setBlockIndexCandidates during LoadBlockIndex().
-
-- Indexing implementations via BaseIndex can no longer assume that indexation happens
-  sequentially, since background validation chainstates can submit BlockConnected
-  events out of order with the active chain.
-
 - The concept of UTXO snapshots is treated as an implementation detail that lives
   behind the ChainstateManager interface. The external presentation of the changes
   required to facilitate the use of UTXO snapshots is the understanding that there are
-  now certain regions of the chain that can be temporarily assumed to be valid (using
-  the nStatus flag mentioned above). In certain cases, e.g. wallet rescanning, this is
-  very similar to dealing with a pruned chain.
+  now certain regions of the chain that can be temporarily assumed to be valid.
+  In certain cases, e.g. wallet rescanning, this is very similar to dealing with
+  a pruned chain.
 
   Logic outside ChainstateManager should try not to know about snapshots, instead
   preferring to work in terms of more general states like assumed-valid.
@@ -54,7 +38,7 @@ data.
 ### "Normal" operation via initial block download
 
 `ChainstateManager` manages a single Chainstate object, for which
-`m_snapshot_blockhash` is null. This chainstate is (maybe obviously)
+`m_from_snapshot_blockhash` is `std::nullopt`. This chainstate is (maybe obviously)
 considered active. This is the "traditional" mode of operation for dashd.
 
 |    |    |
@@ -76,9 +60,15 @@ original chainstate remains in use as active.
 
 Once the snapshot chainstate is loaded and validated, it is promoted to active
 chainstate and a sync to tip begins. A new chainstate directory is created in the
-datadir for the snapshot chainstate called `chainstate_snapshot`. When this directory
-is present in the datadir, the snapshot chainstate will be detected and loaded as
-active on node startup (via `DetectSnapshotChainstate()`).
+datadir for the snapshot chainstate called `chainstate_snapshot`.
+
+When this directory is present in the datadir, the snapshot chainstate will be detected
+and loaded as active on node startup (via `DetectSnapshotChainstate()`).
+
+A special file is created within that directory, `base_blockhash`, which contains the
+serialized `uint256` of the base block of the snapshot. This is used to reinitialize
+the snapshot chainstate on subsequent inits. Otherwise, the directory is a normal
+leveldb database.
 
 |    |    |
 | ---------- | ----------- |
@@ -88,7 +78,7 @@ active on node startup (via `DetectSnapshotChainstate()`).
 The snapshot begins to sync to tip from its base block, technically in parallel with
 the original chainstate, but it is given priority during block download and is
 allocated most of the cache (see `MaybeRebalanceCaches()` and usages) as our chief
-consideration is getting to network tip.
+goal is getting to network tip.
 
 **Failure consideration:** if shutdown happens at any point during this phase, both
 chainstates will be detected during the next init and the process will resume.
@@ -107,33 +97,32 @@ sequentially.
 ### Background chainstate hits snapshot base block
 
 Once the tip of the background chainstate hits the base block of the snapshot
-chainstate, we stop use of the background chainstate by setting `m_stop_use` (not yet
-committed - see bitcoin#15606), in `CompleteSnapshotValidation()`, which is checked in
-`ActivateBestChain()`). We hash the background chainstate's UTXO set contents and
-ensure it matches the compiled value in `CMainParams::m_assumeutxo_data`.
-
-The background chainstate data lingers on disk until shutdown, when in
-`ChainstateManager::Reset()`, the background chainstate is cleaned up with
-`ValidatedSnapshotShutdownCleanup()`, which renames the `chainstate_[hash]` datadir as
-`chainstate`.
+chainstate, we stop use of the background chainstate by setting `m_disabled`, in
+`CompleteSnapshotValidation()`, which is checked in `ActivateBestChain()`). We hash the
+background chainstate's UTXO set contents and ensure it matches the compiled value in
+`CMainParams::m_assumeutxo_data`.
 
 |    |    |
 | ---------- | ----------- |
-| number of chainstates | 2 (ibd has `m_stop_use=true`) |
+| number of chainstates | 2 (ibd has `m_disabled=true`) |
 | active chainstate | snapshot |
 
-**Failure consideration:** if dashd unexpectedly halts after `m_stop_use` is set on
-the background chainstate but before `CompleteSnapshotValidation()` can finish, the
-need to complete snapshot validation will be detected on subsequent init by
-`ChainstateManager::CheckForUncleanShutdown()`.
+The background chainstate data lingers on disk until the program is restarted.
 
 ### Dashd restarts sometime after snapshot validation has completed
 
-When dashd initializes again, what began as the snapshot chainstate is now
-indistinguishable from a chainstate that has been built from the traditional IBD
-process, and will be initialized as such.
+After a shutdown and subsequent restart, `LoadChainstate()` cleans up the background
+chainstate with `ValidatedSnapshotCleanup()`, which renames the `chainstate_snapshot`
+datadir as `chainstate` and removes the now unnecessary background chainstate data.
 
 |    |    |
 | ---------- | ----------- |
 | number of chainstates | 1 |
-| active chainstate | ibd |
+| active chainstate | ibd (was snapshot, but is now fully validated) |
+
+What began as the snapshot chainstate is now indistinguishable from a chainstate that
+has been built from the traditional IBD process, and will be initialized as such.
+
+A file will be left in `chainstate/base_blockhash`, which indicates that the
+chainstate, even though now fully validated, was originally started from a snapshot
+with the corresponding base blockhash.

doc/release-notes-27596.md

@@ -0,0 +1,28 @@
+Pruning
+-------
+
+When using assumeutxo with `-prune`, the prune budget may be exceeded if it is set
+lower than 1100MB (i.e. `MIN_DISK_SPACE_FOR_BLOCK_FILES * 2`). Prune budget is normally
+split evenly across each chainstate, unless the resulting prune budget per chainstate
+is beneath `MIN_DISK_SPACE_FOR_BLOCK_FILES` in which case that value will be used.
+
+RPC
+---
+
+`loadtxoutset` has been added, which allows loading a UTXO snapshot of the format
+generated by `dumptxoutset`. Once this snapshot is loaded, its contents will be
+deserialized into a second chainstate data structure, which is then used to sync to
+the network's tip.
+
+Meanwhile, the original chainstate will complete the initial block download process in
+the background, eventually validating up to the block that the snapshot is based upon.
+
+The result is a usable dashd instance that is current with the network tip in a
+matter of minutes rather than hours. UTXO snapshot are typically obtained via
+third-party sources (HTTP, torrent, etc.) which is reasonable since their contents
+are always checked by hash.
+
+You can find more information on this process in the `assumeutxo` design
+document (<https://github.com/dashpay/dash/blob/master/doc/design/assumeutxo.md>).
+
+`getchainstates` has been added to aid in monitoring the assumeutxo sync process.

doc/release-notes-28685.md

@@ -0,0 +1,4 @@
+RPC
+---
+
+The `hash_serialized_2` value has been removed from `gettxoutsetinfo` since the value it calculated contained a bug and did not take all data into account. It is superseded by `hash_serialized_3` which provides the same functionality but serves the correctly calculated hash.

doc/zmq.md

@@ -141,11 +141,11 @@ Where the 8-byte uints correspond to the mempool sequence number.
     | hashtx | <32-byte transaction hash in Little Endian> | <uint32 sequence number in Little Endian>
 
 
-`rawblock`: Notifies when the chain tip is updated. Messages are ZMQ multipart messages with three parts. The first part is the topic (`rawblock`), the second part is the serialized block, and the last part is a sequence number (representing the message count to detect lost messages).
+`rawblock`: Notifies when the chain tip is updated. When assumeutxo is in use, this notification will not be issued for historical blocks connected to the background validation chainstate. Messages are ZMQ multipart messages with three parts. The first part is the topic (`rawblock`), the second part is the serialized block, and the last part is a sequence number (representing the message count to detect lost messages).
 
     | rawblock | <serialized block> | <uint32 sequence number in Little Endian>
 
-`hashblock`: Notifies when the chain tip is updated. Messages are ZMQ multipart messages with three parts. The first part is the topic (`hashblock`), the second part is the 32-byte block hash, and the last part is a sequence number (representing the message count to detect lost messages).
+`hashblock`: Notifies when the chain tip is updated. When assumeutxo is in use, this notification will not be issued for historical blocks connected to the background validation chainstate. Messages are ZMQ multipart messages with three parts. The first part is the topic (`hashblock`), the second part is the 32-byte block hash, and the last part is a sequence number (representing the message count to detect lost messages).
 
     | hashblock | <32-byte block hash in Little Endian> | <uint32 sequence number in Little Endian>