Persist unconfirmed wallet transactions and restore them at start-up

[Ergologica]

Closes #1154.

The problem

The memory pool is not persisted: ErgoNodeViewHolder builds it with ErgoMemPool.empty(settings) on both the genesis and the restore path, so a restart drops every transaction waiting in it. Peers may re-gossip them, but nothing guarantees that they will, or when.

What turns this from a nuisance into a correctness problem is the wallet side. ErgoWalletState.walletFilter decides whether an on-chain box may be spent by asking the off-chain registry:

val preStatus = if (trackedBox.chainStatus.onChain) {
  offChainRegistry.onChainBalances.exists(_.id == trackedBox.boxId)
} else { true }

and OffChainRegistry.init rebuilds onChainBalances from walletRegistry.unspentBoxes(PaymentsScanId) with offChainBoxes = ArraySeq.empty. The wallet registry still lists those boxes as unspent - the spending transaction never got onto the blockchain - so after a restart every box spent by a still-unconfirmed transaction is offered for spending again, and the wallet will build a conflicting transaction spending it. The memory pool cross-check in the same filter cannot save it, because the pool is empty too.

The change

Wallet-related unconfirmed transactions are kept in WalletStorage and replayed at start-up.

Where they are stored. WalletStorage, not WalletRegistry: it is documented as holding "version-agnostic wallet actor's mutable state (which is not a subject to rollbacks in case of forks)", which is what off-chain data is. New secondary prefix byte 3, key [0, 3] ++ txId, value height ++ txBytes.

Which ones. scanOffChainUpdate now also reports whether the transaction is of any interest to the wallet: it either pays the wallet (extractWalletOutputs found something) or spends one of its boxes (registry.getBox on the inputs). Nothing else is stored, so the store does not turn into a copy of the memory pool.

Start-up, wallet side. ErgoWalletService.restoreOffChainState replays the stored transactions through the same off-chain scan, rebuilding offChainRegistry so the spent boxes stay spent. updateOnTransaction is idempotent, so a replayed transaction being scanned again later does no harm.

Start-up, memory pool side. ErgoNodeViewHolder.preStart asks the wallet for them and feeds them back through txModify as a new RestoredTransaction message. Accepted ones land in the pool and, via SuccessfulTransaction, get gossiped again. Anything the pool refuses - it got onto the blockchain while the node was down, or a conflicting transaction did - is dropped from the store, so the wallet self-heals instead of retrying forever.

Ordering. Unconfirmed transactions form chains, and both consumers need the parent first: the off-chain registry only nets a spending out if it has already seen the box being spent, and the pool refuses a transaction whose inputs it does not know. ErgoWalletActor.orderByDependency does that.

Bounding the store. Entries go away when their transaction appears in an applied block, and are given up on after UnconfirmedTxLifetimeInBlocks = 1440 blocks (about two days). Re-storing a transaction keeps the height it was first seen at - otherwise the replay at every restart would push the expiry back forever. I kept the lifetime a constant rather than a config key to keep the diff small; happy to make it a setting if you prefer.

Tests

• WalletStorageSpec - round trip through the database, the new bucket not colliding with the public keys and scans buckets, re-storing keeping the original height, forgetting an unknown id being a no-op.
• ErgoWalletServiceSpec - a box spent off-chain is filtered out; after a simulated restart it is spendable again, and restoreOffChainState makes it unspendable (that property is this issue, start to finish); chained transactions replayed parent first; orderByDependency leaving independent transactions alone.
• ErgoWalletSpec - end to end through the actor: a generated payment is kept after scanOffchain, comes back from unconfirmedTransactionsToRestore, and is forgotten once the block carrying it is applied.

Full suite green (sbt test, 751 tests, 0 failed).

Notes for review

• ErgoNodeViewHolder.preStart uses an ask on the wallet actor. The wallet stashes messages until ReadWallet completes, so the reply arrives once it is loaded; a failure is logged and start-up carries on.
• Nothing changes for a node whose wallet has no unconfirmed transactions: readUnconfirmedTransactions() comes back empty and restoreOffChainState returns the state untouched.
• The rollback path still has the pre-existing gap of Rework offchain registry #1180 (the off-chain registry is not refreshed on rollback). This change does not make it worse, and once unconfirmed transactions are kept, refreshing on rollback could reuse the same replay.

[Ergologica]

On the two red checks - neither is caused by this change, and I checked rather than assumed.

Run it node tests fails on master itself: the CI run for commit 4a7dba0, which is the base of this branch, is red on exactly that job, and so is every other open pull request I looked at (e.g. #2462). The cause is UtxoStateNodesSyncSpec, which #2452 fixes.

Run node tests failed on CandidateGeneratorSpec, "accept solution for previous candidate after regeneration" (line 318, the reward-script assertion). That suite is flaky on master, independently of this branch. I stashed every change here, went back to a pristine 4a7dba0 and ran the suite twice:

• run 1: should ignore cached candidate when forced = true *** FAILED ***
• run 2: should ignore cached candidate when forced = true and should preserve previous candidate when forced regeneration occurs *** FAILED ***

A different test each time, none of them the one CI hit. The suite runs with blockCandidateGenerationInterval = 1.millis and blockInterval = 1.seconds and fishes for messages, so it loses races on a loaded machine; the failure mode is the test reading bestFullBlockOpt before the block it mined has been applied, which is why the assertion sees a plain P2PK script instead of the delayed miner reward script.

For the record, the full suite on this branch was green when I ran it before opening the PR (sbt test, 751 tests, 0 failed), and every wallet suite passed in the CI run above - the only failing suite was the mining one.

Happy to open a separate PR to make CandidateGeneratorSpec deterministic if that is useful; it seems worth doing regardless of this change.