fix(devnet): make NM processedPath a sibling of srcPath, not a child - #205
Open
mateeullahmalik wants to merge 1 commit into
Open
fix(devnet): make NM processedPath a sibling of srcPath, not a child#205mateeullahmalik wants to merge 1 commit into
mateeullahmalik wants to merge 1 commit into
Conversation
e357c6e fixed the scanner block to emit correctly-typed inline tables, but derived the destination as "${dir%/}/processed" -- i.e. INSIDE the directory being scanned. scanner/scanner.go walks srcPath with filepath.Walk (recursive) and dedups on (dir, name). A nested processed dir is therefore re-scanned on the next tick, and because the move changed `dir` the DB lookup misses, so the file is registered a SECOND time. Every upload costs two cascade registrations (~15120ulume each) and double supernode load. Observed on the shared devnet: 13 `create-metadata start` calls whose source path was already under .../processed/. Deriving the sibling "${dir%/}-processed" restores the uploader's own convention -- its shipped config.toml pairs ~/.lumera-uploader/drop with the sibling ~/.lumera-uploader/processed, never a child. Verified on the shared devnet (5 validators, chain v1.20.2-rc1): after the change, 210 uploads with 0 re-registrations sourced from the processed dir, and processed_files advanced 1242 -> 1276. Note for existing devnets: files already sitting in a nested <srcPath>/processed must be moved out of the scan path before restarting, otherwise they are re-registered once on the next pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
e357c6e7fixed the[scanner].directoriesblock to emit correctly-typed inline tables (the crudini bug), but derived the destination as:scanner/scanner.gowalkssrcPathwithfilepath.Walk— recursive — and dedups on(dir, name):So a nested
<srcPath>/processedis re-scanned on the next tick. The move changeddir, soHasFile(dir, name)misses and the file looks brand new — it is registered a second time.Cost: two cascade registrations per file (~15120ulume each, enforced at
x/action/v1/keeper/action.go:69) plus double supernode upload load, silently.Evidence
Observed on the shared devnet —
create-metadata startwhose source path is already under the processed dir:13 such re-registrations before the fix.
Fix
Derive a sibling:
This restores the uploader's own convention — its shipped
config.tomlpairs~/.lumera-uploader/dropwith the sibling~/.lumera-uploader/processed, never a child:Verification
Deployed to the shared devnet (5 validators, chain
v1.20.2-rc1, SNv2.6.4-testnet), NM restarted through the normalstart.shpath with no overrides:processed_filesGenerated config after the change:
Sustained over ~10 minutes of live load with zero re-registrations.
Risks
Low. Devnet tooling only — no chain code, no state machine, no consensus surface.
bash -nclean. No other file in the repo references the nested path.Rollback
Revert the commit; the previous behavior returns immediately on the next
lumera-uploader-setup.shrun.Operator note
On a devnet that already ran the nested version, move stranded files out of the scan path before restarting, or they get re-registered once on the next pass:
mv -f /shared/nm-files/processed/* /shared/nm-files-processed/ rmdir /shared/nm-files/processed