[Storehouse] 007 Payloadless Checkpoint (v7) by zhangchiqing · Pull Request #8578 · onflow/flow-go

zhangchiqing · 2026-06-09T15:44:30Z

The previous PR #8577 has integrated the payloadless feature to the execution node behind a feature flag, but the payloadless feature only works for empty state. To support non-empty state, which practically to support restart, we need to support a v7 payloadless checkpoint.

Add Version7 Checkpoint file format (Payloadless).
Compactor will still use V6 checkpointer. V7 support will be added in the future.

coderabbitai · 2026-06-09T15:44:39Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 1abc5c01-e757-4d56-acc2-88233c5e340a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch leo/payloadless-checkpoint-v7

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

zhangchiqing · 2026-06-09T15:49:27Z

+	// Read the V6 checkpoint fully — the V6 reader already reads the 16 subtrie
+	// part files concurrently. The resulting tries share sub-tries via Go pointer
+	// identity, which lets FromV6Tries memoize and avoid redundant conversion.
+	v6Tries, err := LoadCheckpoint(v6Header, logger)


This loads the entire checkpoint in memory, which requires a lot of memory. An improvement would be streaming the file, iterate each node, if it's a leaf node, then convert into payload less. This could save lots of memory. Skipped for now, because that requires additional code, can be implemented later.

zhangchiqing · 2026-06-09T16:41:28Z

+		return nil, nil
+	}
+	if index > totalSubTrieNodeCount {
+		nodePos := index - totalSubTrieNodeCount


This part is same logic as v6 (getNodeByIndex) but slightly different in style, where v6 extracted it into getTopNodeByIndex, which adds little value. Instead, the logic here for getPayloadlessTopNodeByIndex is inlined.

zhangchiqing · 2026-06-09T16:43:50Z

@@ -0,0 +1,421 @@
+package wal


vimdiff ledger/complete/wal/checkpoint_v6_reader.go ledger/complete/wal/checkpoint_v7_reader.go

This file is similar to v6_reader, except some minimal coding style changes. Some read functions are not copied over but reused, such as readCheckpointHeader, allPartFileExist, readSubTriesFooter etc

zhangchiqing · 2026-06-09T17:33:32Z

@@ -0,0 +1,484 @@
+package wal


vimdiff ledger/complete/wal/checkpoint_v6_writer.go ledger/complete/wal/checkpoint_v7_writer.go

v7 is similar to v6 except some comments changes.

zhangchiqing · 2026-06-09T19:14:16Z

@@ -0,0 +1,372 @@
+package payloadless


mirrored from ledger/complete/mtrie/flattener/encoding.go and ledger/complete/mtrie/flattener/iterator.go

zhangchiqing · 2026-06-09T19:16:47Z

+	encNodeIndexSize = 8
+	encRegCountSize  = 8
+
+	encLeafHashFlagSize = 1


encRegSizeSize and encPayloadLengthSize in v6 checkpoint node (see ledger/complete/mtrie/flattener/encoding.go) is replaced by encLeafHashFlagSize in payloadless

zhangchiqing · 2026-06-09T20:28:12Z

+		return nil, fmt.Errorf("cannot read leaf hash flag: %w", err)
+	}
+
+	flag := scratch[0]


this part is different from readPayloadFromReader

zhangchiqing · 2026-06-09T20:29:32Z

+
+type EncodedTrie struct {
+	RootIndex uint64
+	RegCount  uint64


RegSize is removed

zhangchiqing · 2026-06-09T20:30:54Z

+		encLeafHashFlagSize +
+		encLeafHashSize


encPayloadLengthSize + encPayloadSize is replaced by encLeafHashFlagSize + encLeafHashSize in payloadless

zhangchiqing · 2026-06-09T20:32:01Z

+	copy(buf[pos:], path[:])
+	pos += encPathSize
+
+	// Encode leaf hash flag (1 byte) and optional leaf hash (0 or 32 bytes)


this part is different from full trie node's encodeLeafNode

zhangchiqing · 2026-06-09T21:49:16Z

 	}

-	lastCheckpointNum, err := c.checkpointer.LatestCheckpoint()
+	lastCheckpointNum, err := c.checkpointer.LatestCheckpointV6()


The compactor will eventually need to support both V6 and V7, but this PR only supports V6. To avoid ambiguity, I renamed the function to LatestCheckpointV6.

janezpodhostnik · 2026-07-22T16:18:40Z

+		// HasRootCheckpointV7 guard keeps a re-entry after an interrupted
+		// bootstrap from hitting ConvertCheckpointV6ToV7's "output exists" check.
+		//
+		// TODO: ConvertCheckpointV6ToV7 reads the entire V6 forest into memory
+		// before emitting V7, a memory/time spike at first boot for mainnet-scale
+		// root checkpoints. A future optimization is to convert subtrie-by-subtrie
+		// without loading the whole forest.
+		if exeNode.exeConf.payloadless {
+			triedir := exeNode.exeConf.triedir
+			hasV7Root, err := wal.HasRootCheckpointV7(triedir)


If the node dies mid-conversion (realistic here: this is the memory-heavy step, so an OOM kill), the part files are left on disk but the header file is not, it is written last. On the next boot HasRootCheckpointV7 returns false (it only checks the header file), so this branch converts again, but ConvertCheckpointV6ToV7 refuses because the leftover part files trip its output-already-exists check. The node then cannot boot until someone deletes the partial files by hand (StoreCheckpointV7 only cleans up after itself on error returns, not on process death).

Suggestion: when hasV7Root is false, first delete any leftover partial output, e.g. deleteCheckpointFiles(triedir, modelbootstrap.FilenameWALRootCheckpoint+wal.V7FileSuffix). The V6 source is untouched, so retrying the conversion from scratch is always safe and the manual util keeps its strict no-clobber behavior.

janezpodhostnik · 2026-07-22T16:18:40Z

-	hashes, err := wal.ReadTriesRootHash(logger, dir, fileName)
+	var hashes []ledger.RootHash
+	var err error
+	if strings.HasSuffix(fileName, wal.V7FileSuffix) {


This branch is currently unreachable through its only caller chain: GenerateProtocolSnapshotForCheckpointWithHeights → findLatestCheckpointFilePath (line 141) picks last from wal.ListCheckpoints, which since this PR includes V7 numbers, but always renders wal.NumberToFilename(last), the V6 name. On a payloadless triedir whose newest checkpoint is V7-only, that produces a path to a nonexistent file, and it can never produce the .v7 path this dispatch handles. findLatestCheckpointFilePath should use ListCheckpointsWithInfo and render NumberToFilenameV7 when the latest checkpoint is V7.

janezpodhostnik · 2026-07-22T16:18:40Z

+	// If both failed, return combined error
+	if v6Err != nil && v7Err != nil {
+		return fmt.Errorf("failed to remove checkpoint %d: v6 error: %w, v7 error: %v", checkpoint, v6Err, v7Err)
+	}
+	return nil


deleteCheckpointFiles succeeds when its glob matches nothing, so the common failure case, V6 files exist but deletion fails, no V7 files present, returns nil here. (%v also drops the v7 error from the wrap chain.) Since both compactors now use the version-specific variants and no production caller of this method remains, consider errors.Join(v6Err, v7Err) (wrapped with the checkpoint number) or deprecating this method like LoadCheckpoint.

janezpodhostnik · 2026-07-22T16:18:40Z

+	switch flag {
+	case leafHashAbsent:
+		return nil, nil
+	case leafHashPresent:
+		if len(scratch) < encHashSize {
+			scratch = make([]byte, encHashSize)
+		}
+		_, err := io.ReadFull(reader, scratch[:encHashSize])
+		if err != nil {
+			return nil, fmt.Errorf("cannot read leaf hash: %w", err)
+		}
+		leafHash, err := hash.ToHash(scratch[:encHashSize])
+		if err != nil {
+			return nil, fmt.Errorf("failed to decode leaf hash: %w", err)
+		}
+		return &leafHash, nil
+	default:
+		return nil, fmt.Errorf("invalid leaf hash flag: %d", flag)


The leaf-hash flag is the one new on-disk encoding mechanism vs V6, and no test executes the leafHashAbsent path or the invalid-flag branch: there is no flattener_test.go, and every checkpoint test fixture uses non-empty values.

janezpodhostnik · 2026-07-22T16:18:40Z


+// Version 7 includes these changes:
+//   - payloadless mode: leaf nodes store payload hashes (32 bytes) instead of full payloads
+//   - used for verification nodes that don't need actual payload values


nit: V7 checkpoints are read and written by payloadless execution nodes (this PR wires them into the EN builder); verification-node use is future/speculative.

Suggested change

// - used for verification nodes that don't need actual payload values

// - used by payloadless execution nodes, which read register values from the storehouse

janezpodhostnik · 2026-07-22T16:18:40Z

 // For now config.LedgerServiceAddr is ignored.
+//
+// Expected error returns during normal operation:
+//   - error if config.Triedir is empty


nit: the missing-V7-checkpoint refusal (both the "V6 exists, run checkpoint-convert-v7" return and the plain "no V7 found" return) is equally expected during normal operation.

janezpodhostnik · 2026-07-22T16:18:40Z

+	"github.com/onflow/flow-go/utils/unittest"
+)
+
+func TestVersionV7(t *testing.T) {


nit: three V6 unit tests have no V7 analog: TestEncodeSubTrie, TestGetNodesByIndex, and TestCannotStoreTwice.

janezpodhostnik · 2026-07-22T16:49:35Z

+					Msg("payloadless ledger: could not also list V6 checkpoints while reporting missing V7")
+			}
+			if latestV6 >= 0 {
+				logger.Warn().


Should this be Error level?

janezpodhostnik · 2026-07-22T16:52:13Z

+			v6Numbers, latestV6, v6ListErr := wal.ListV6Checkpoints(config.Triedir)
+			if v6ListErr != nil {
+				logger.Warn().Err(v6ListErr).
+					Str("triedir", config.Triedir).


All loggers contain Str("triedir", config.Triedir). you could create a new logger with that already included.

janezpodhostnik · 2026-07-22T16:59:39Z

+type PayloadlessLedgerWithCompactor struct {
+	*PayloadlessLedger
+	compactor *PayloadlessCompactor
+	logger    zerolog.Logger


*PayloadlessLedger already has a logger. Can we reuse that one?

payloadless compactor

zhangchiqing commented Jun 9, 2026

View reviewed changes

zhangchiqing force-pushed the leo/payloadless-checkpoint-v7 branch from 7b14692 to 2625248 Compare June 9, 2026 21:05

zhangchiqing commented Jun 9, 2026

View reviewed changes

zhangchiqing marked this pull request as ready for review June 9, 2026 21:52

zhangchiqing requested a review from a team as a code owner June 9, 2026 21:52

zhangchiqing force-pushed the leo/payloadless-execution-builder branch from d7dea94 to fcc22e5 Compare June 18, 2026 23:18

zhangchiqing force-pushed the leo/payloadless-checkpoint-v7 branch from c25cf48 to 267d209 Compare June 18, 2026 23:19

zhangchiqing force-pushed the leo/payloadless-execution-builder branch from fcc22e5 to f138980 Compare June 19, 2026 00:00

zhangchiqing force-pushed the leo/payloadless-checkpoint-v7 branch from 267d209 to 8d341b5 Compare June 19, 2026 00:00

zhangchiqing force-pushed the leo/payloadless-execution-builder branch from f138980 to fc173be Compare July 2, 2026 18:01

zhangchiqing force-pushed the leo/payloadless-checkpoint-v7 branch 3 times, most recently from 1ea0be7 to ea43efc Compare July 2, 2026 18:23

zhangchiqing force-pushed the leo/payloadless-execution-builder branch from fc173be to ff4a401 Compare July 13, 2026 17:24

zhangchiqing force-pushed the leo/payloadless-checkpoint-v7 branch from ea43efc to a835d1f Compare July 13, 2026 17:24

zhangchiqing mentioned this pull request Jul 13, 2026

[Storehouse] 008 Replay WAL onto Payloadless trie #8598

Open

zhangchiqing force-pushed the leo/payloadless-execution-builder branch from ff4a401 to b825a6a Compare July 14, 2026 17:16

zhangchiqing force-pushed the leo/payloadless-checkpoint-v7 branch from a835d1f to 8b9576b Compare July 14, 2026 17:56

janezpodhostnik reviewed Jul 22, 2026

View reviewed changes

add v7 checkpointer

42bb50e

zhangchiqing added 12 commits July 30, 2026 17:37

add checkpoint v7

53d14e5

add checkpoint v6->v7 converter

6f99d4c

reduce diff between checkpoint v7 and checkpoint v6

de068b6

update checkpoint cmd util

d204b4d

refactor to minimize the change

3c1c3dd

rename LatestCheckpoint to LatestCheckpointV6

554a3d0

add payloadless WAL

3ce3b46

rename v6 checkpointer related functions

5d4adde

update logger

d3371f5

Automatic V6→V7 root-checkpoint conversion at bootstrap, and refactor

57cc776

payloadless compactor

fix tests

defa80b

address review comments

d77747c

zhangchiqing force-pushed the leo/payloadless-checkpoint-v7 branch from 8b9576b to d77747c Compare July 31, 2026 02:44

zhangchiqing requested a review from janezpodhostnik July 31, 2026 02:45

janezpodhostnik approved these changes Jul 31, 2026

View reviewed changes

	// - used for verification nodes that don't need actual payload values
	// - used by payloadless execution nodes, which read register values from the storehouse

Uh oh!

Conversation

zhangchiqing commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhangchiqing Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhangchiqing Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zhangchiqing commented Jun 9, 2026 •

edited

Loading

coderabbitai Bot commented Jun 9, 2026 •

edited

Loading

zhangchiqing Jun 9, 2026 •

edited

Loading

zhangchiqing Jun 9, 2026 •

edited

Loading