[Storehouse] 010 Util convert v6 -> v7 checkpoint - #8583
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
08074ba to
4f1f09d
Compare
ce39b24 to
287bfbb
Compare
0955ccb to
8177757
Compare
287bfbb to
a9c6645
Compare
8177757 to
efd0192
Compare
a9c6645 to
ca39b1a
Compare
efd0192 to
fba2a36
Compare
ca39b1a to
4d5c8bf
Compare
fba2a36 to
dd25d6c
Compare
4d5c8bf to
91eb0a7
Compare
| ) error { | ||
| err := convertCheckpointV6ToV7Stream(inputDir, inputFileName, outputDir, outputFileName, logger, nWorker) | ||
| if err != nil { | ||
| cleanupErr := deleteCheckpointFiles(outputDir, outputFileName) |
There was a problem hiding this comment.
Re-running the conversion deletes the previously converted output. The "V7 output already exists" rejection from convertCheckpointV6ToV7Stream reaches this cleanup, and deleteCheckpointFiles removes every <outputFile>* file.
| streamFiles := filePaths(dir, streamName, subtrieLevel) | ||
| require.Equal(t, len(nonStreamFiles), len(streamFiles)) | ||
| for i, nf := range nonStreamFiles { | ||
| require.NoError(t, compareFiles(nf, streamFiles[i]), |
There was a problem hiding this comment.
compareFiles has a bug! It opens file1 twice (checkpoint_v6_test.go:451), so it can never fail. Fix the helper to os.Open(file2).
| if err != nil { | ||
| return 0, fmt.Errorf("could not read subtrie footer: %w", err) | ||
| } | ||
| if embeddedSum != expectedSum { |
There was a problem hiding this comment.
This only compares two stored checksums; the bytes actually read are never CRC-verified, so input corruption is copied into a V7 file that then carries a fresh valid checksum. The non-stream path verifies content CRC via readCheckpointSubTrie. Same applies to the top-trie path below.
| checksums := make([]uint32, subtrieCount) | ||
| for k := 0; k < subtrieCount; k++ { | ||
| r := <-results | ||
| if r.err != nil { |
There was a problem hiding this comment.
Returning on the first error leaves the other workers running. Their deferred Close renames temp files to final part files, possibly after the wrapper's deleteCheckpointFiles has run, leaving stray output that contradicts the godoc and blocks retries. Drain all subtrieCount results, then return the first error, aggregate the errors or/and build a stopping mechanism into the workers.
| go func() { | ||
| for i := range jobs { | ||
| sum, err := convertSubTrieFileV6ToV7Stream( | ||
| inputDir, inputFileName, outputDir, outputFileName, i, subtrieChecksums[i], logger) |
There was a problem hiding this comment.
subtrieChecksums is sized from the header while this loop hard-codes subtrieCount, so a checksum-valid header declaring fewer than 16 subtries panics in a worker goroutine. You could validate len(subtrieChecksums) == subtrieCount up front.
| flagNWorker, | ||
| ) | ||
| var err error | ||
| if flagStream { |
There was a problem hiding this comment.
stream could be a boolean input into ConvertCheckpointV6ToV7 so that you don't need to expose 2 methods with identical inputs.
dd25d6c to
3e9115e
Compare
f871ac3 to
91eb0a7
Compare
91eb0a7 to
1c62fec
Compare
…l V7 conversion cleanup at bootstrap, logger and godoc fixes, and tests for the leaf-hash flag and V7 subtrie paths
2 participants
To bootstrap a payloadless EN, a v7 root checkpoint file is needed. This PR adds a util to convert a v6 root checkpoint into a v7 root checkpoint.
It adds two ways to convert: