-
Notifications
You must be signed in to change notification settings - Fork 885
Implement new Giga GarbageCollector interface #3868
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 13 commits
e71f50b
6935ce7
42a667d
c9f3de9
2a10f31
f43b8b3
47a6dfd
45b80e0
29e7d7d
7ccdf29
9b96d16
c07fa75
a682ec3
9c0e746
5104f64
c15c7f1
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -7,44 +7,51 @@ import ( | |
| littdb "github.com/sei-protocol/sei-chain/sei-db/db_engine/litt" | ||
| ) | ||
|
|
||
| // LittBlockConfig configures a LittDB-backed types.BlockDB. | ||
| type LittBlockConfig struct { | ||
| // BlockDBConfig configures a LittDB-backed types.BlockDB. | ||
| type BlockDBConfig struct { | ||
| // Litt is the underlying LittDB configuration, including the data directory | ||
| // paths. The block store builds its two tables (blocks, qcs) on top of this | ||
| // DB. Required; use DefaultConfig to obtain one with sane defaults, then | ||
| // override fields as needed (e.g. Litt.Fsync, Litt.GCPeriod). | ||
| // paths. The block store builds its single table (see tableName, which holds | ||
| // blocks and QCs both) on top of this DB. Required; use DefaultConfig to obtain | ||
| // one with sane defaults, then override fields as needed (e.g. Litt.Fsync, | ||
| // Litt.GCPeriod). | ||
| Litt *littdb.Config | ||
|
|
||
| // Retention is the failsafe minimum age before any pruned record may be | ||
| // RetentionTime is the failsafe minimum age before any pruned record may be | ||
| // reclaimed. Reclamation requires BOTH this age to elapse AND the prune | ||
| // watermark to advance past the record, so even an over-eager watermark | ||
| // cannot delete data younger than Retention. Must be positive. | ||
| Retention time.Duration | ||
| // cannot delete data younger than RetentionTime. Must be positive. | ||
| // | ||
| // It is an age floor, not a retention policy: how much history this store keeps | ||
| // is the RollbackWindow and LookbackWindow on | ||
| // gc.StorageGarbageCollectorConfig, which cover every managed store at once. | ||
| // Raising this only delays reclaiming what the watermark has already released, | ||
| // which costs disk and buys nothing those windows do not already express. | ||
| RetentionTime time.Duration | ||
| } | ||
|
|
||
| // DefaultConfig returns a LittBlockConfig preloaded with all defaults, rooted at | ||
| // DefaultConfig returns a BlockDBConfig preloaded with all defaults, rooted at | ||
| // dir. Override fields as needed, then pass it to NewBlockDB (which validates). | ||
| func DefaultConfig(dir string) (*LittBlockConfig, error) { | ||
| func DefaultConfig(dir string) (*BlockDBConfig, error) { | ||
| littConfig, err := littdb.DefaultConfig(dir) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("failed to build litt config: %w", err) | ||
| } | ||
| return &LittBlockConfig{ | ||
| Litt: littConfig, | ||
| Retention: 24 * time.Hour, | ||
| return &BlockDBConfig{ | ||
| Litt: littConfig, | ||
| RetentionTime: time.Hour, | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [suggestion] The default drops 24h → 1h, shrinking by 24× the failsafe the field doc describes as protecting against an over-eager watermark ("even an over-eager watermark cannot delete data younger than RetentionTime"). The PR description explains at length why the receipt TTL became a flat duration and why Note this also flows into There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [suggestion] The BlockDB default TTL failsafe drops from BlockDB is not wired to a The PR body mentions "defaulting to 1 hour" only in the context of the flat-duration rework; worth calling out explicitly as a default change (and confirming 1h is intended for the pre-collector world, not just the collector-managed one). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [suggestion] This 24h → 1h default is a live behavior change on the only path BlockDB currently runs on, and the field doc's framing ("this only reclaims already-released records sooner") understates it slightly. The watermark this TTL gates is driven today by ABCI |
||
| }, nil | ||
| } | ||
|
|
||
| // Validate performs a sanity check on the configuration. | ||
| func (c *LittBlockConfig) Validate() error { | ||
| func (c *BlockDBConfig) Validate() error { | ||
| if c == nil { | ||
| return fmt.Errorf("config is required") | ||
| } | ||
| if c.Litt == nil { | ||
| return fmt.Errorf("config.Litt is required") | ||
| } | ||
| if c.Retention <= 0 { | ||
| return fmt.Errorf("config.Retention must be positive (got %s)", c.Retention) | ||
| if c.RetentionTime <= 0 { | ||
| return fmt.Errorf("config.RetentionTime must be positive (got %s)", c.RetentionTime) | ||
| } | ||
| return nil | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2,6 +2,8 @@ package littblock | |
|
|
||
| import ( | ||
| "fmt" | ||
| "os" | ||
| "path/filepath" | ||
| "sync" | ||
| "sync/atomic" | ||
|
|
||
|
|
@@ -12,18 +14,27 @@ import ( | |
| "github.com/sei-protocol/sei-chain/sei-tendermint/libs/utils" | ||
| ) | ||
|
|
||
| // ledgerTableName is the single table holding both blocks and QCs. They share | ||
| // one table so a crash leaves a contiguous write-order prefix spanning both | ||
| // record kinds (see NewBlockDB), which is what guarantees a persisted block is | ||
| // always covered by a persisted QC. | ||
| const ledgerTableName = "ledger" | ||
| // tableName is the single table holding blocks and QCs both, despite the name. They share one | ||
| // table so a crash leaves a contiguous write-order prefix spanning both record kinds (see | ||
| // NewBlockDB), which is what guarantees a persisted block is always covered by a persisted QC. | ||
| // | ||
| // This value is persisted layout, not just an identifier: littdb puts a table's data at | ||
| // <root>/<tableName>/segments, so changing it makes NewBlockDB open a fresh empty table while the | ||
| // old data sits untouched under the previous name — neither served nor reclaimed. refuseLegacyTable | ||
| // turns that into a startup error rather than a store that looks healthy and empty. | ||
| const tableName = "blocks" | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [suggestion] The comment you added here states the hazard precisely — a rename makes Since the check is cheap and the comment already argues for it: There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [nit] The rename itself seems like net-negative churn. The comment two lines up has to open with "the single table holding blocks and QCs both, despite the name" — i.e. the new name is less accurate than |
||
|
|
||
| // legacyTableName is what tableName was called before the rename. Nothing opens it; it exists only | ||
| // so refuseLegacyTable can recognize a directory written before the rename. | ||
| const legacyTableName = "ledger" | ||
|
|
||
| var _ types.BlockDB = (*blockDB)(nil) | ||
|
|
||
| // blockDB is a durable types.BlockDB backed by LittDB | ||
| type blockDB struct { | ||
| db littdb.DB | ||
| table littdb.Table | ||
| db littdb.DB | ||
| table littdb.Table | ||
| config *BlockDBConfig | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [suggestion] This field is assigned once ( There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [suggestion] Struct fields aren't caught by the |
||
|
|
||
| // watermark is a retention floor, always a QC boundary (a GlobalRange().First): | ||
| // PruneBefore rounds a requested prune point down to the start of the cohort | ||
|
|
@@ -74,13 +85,18 @@ type blockDB struct { | |
| } | ||
|
|
||
| // NewBlockDB opens (or creates) a LittDB-backed types.BlockDB from config. The | ||
| // underlying LittDB is built from config.Litt, and the two tables apply | ||
| // config.Retention as a TTL failsafe (pruning never reclaims data younger than | ||
| // that even once the watermark has advanced past it). | ||
| func NewBlockDB(config *LittBlockConfig) (types.BlockDB, error) { | ||
| // underlying LittDB is built from config.Litt, and the table applies | ||
| // config.RetentionTime as a TTL failsafe (pruning never reclaims data younger | ||
| // than that even once the watermark has advanced past it). | ||
| func NewBlockDB(config *BlockDBConfig) (types.BlockDB, error) { | ||
| if err := config.Validate(); err != nil { | ||
| return nil, fmt.Errorf("invalid block db config: %w", err) | ||
| } | ||
| // Before littbuilder.NewDB, so a refused open leaves the directory exactly as it found it | ||
| // rather than adding an empty table beside the one it is complaining about. | ||
| if err := refuseLegacyTable(config.Litt.Paths); err != nil { | ||
| return nil, err | ||
| } | ||
| db, err := littbuilder.NewDB(config.Litt) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("failed to open litt db: %w", err) | ||
|
|
@@ -96,8 +112,8 @@ func NewBlockDB(config *LittBlockConfig) (types.BlockDB, error) { | |
| // guarantees a persisted block is always covered by a persisted QC. It also | ||
| // backs the write-order cursors and contiguous-QC recovery. ShardingFactor | ||
| // > 1, or splitting blocks and QCs across two tables, would void this. | ||
| tableConfig := littdb.DefaultTableConfig(ledgerTableName) | ||
| tableConfig.TTL = config.Retention | ||
| tableConfig := littdb.DefaultTableConfig(tableName) | ||
| tableConfig.TTL = config.RetentionTime | ||
| tableConfig.GCFilter = s.gcFilter | ||
| tableConfig.ShardingFactor = 1 // DO NOT CHANGE!! | ||
| table, err := db.BuildTable(tableConfig) | ||
|
|
@@ -107,6 +123,7 @@ func NewBlockDB(config *LittBlockConfig) (types.BlockDB, error) { | |
| } | ||
|
|
||
| s.table = table | ||
| s.config = config | ||
|
|
||
| if err := s.recoverCursors(); err != nil { | ||
| _ = db.Close() | ||
|
|
@@ -119,6 +136,37 @@ func NewBlockDB(config *LittBlockConfig) (types.BlockDB, error) { | |
| return s, nil | ||
| } | ||
|
|
||
| // refuseLegacyTable fails the open when any root path holds a table directory under | ||
| // legacyTableName. Such a directory is blocks and QCs that this process cannot reach: littdb | ||
| // resolves a table to <root>/<tableName>/segments, so the data is neither served nor reclaimed, and | ||
| // the store would otherwise present itself as healthy and empty. An empty store is indistinguishable | ||
| // from a correct one until something asks for history that is no longer there, which makes it the | ||
| // worst shape this failure could take. | ||
| // | ||
| // No deployment can reach this — nothing has ever run littblock with the old name persisted — so it | ||
| // exists for dev, CI, and devnet homes written before the rename. The operator action is to delete | ||
| // the directory (or move it aside); this check can be deleted once no such directory remains. | ||
| // | ||
| // A root that cannot be stat'd is refused for the same reason it is refused when the directory is | ||
| // present: an unreadable root cannot rule out data hiding under the old name. | ||
| func refuseLegacyTable(paths []string) error { | ||
| for _, root := range paths { | ||
| legacy := filepath.Join(root, legacyTableName) | ||
| switch _, err := os.Stat(legacy); { | ||
| case err == nil: | ||
| return fmt.Errorf( | ||
| "block db: found a pre-rename %q table at %s; the table is now named %q, so those "+ | ||
| "blocks and QCs would be neither served nor reclaimed. Delete or move the "+ | ||
| "directory aside to start from an empty store", | ||
| legacyTableName, legacy, tableName) | ||
| case !os.IsNotExist(err): | ||
| return fmt.Errorf("block db: check for a pre-rename %q table at %s: %w", | ||
| legacyTableName, legacy, err) | ||
| } | ||
| } | ||
| return nil | ||
| } | ||
|
|
||
| // recoverCursors reloads the write-order cursors (lastBlockNumber, lastQCNext, | ||
| // and their presence flags) from on-disk state. Without this, a reopened DB | ||
| // would treat itself as empty and let WriteBlock/WriteQC silently accept | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,91 @@ | ||
| package littblock | ||
|
|
||
| import ( | ||
| "github.com/sei-protocol/sei-chain/sei-db/management/gc" | ||
| "github.com/sei-protocol/sei-chain/sei-tendermint/autobahn/types" | ||
| ) | ||
|
|
||
| // In terms of the collector's RollbackWindow and LookbackWindow, garbage collection guarantees: | ||
| // | ||
| // 1. Garbage collection will not delete any data that is necessary to roll back to any block | ||
| // between LatestBlock and (LatestBlock - RollbackWindow), inclusive. | ||
| // 2. Garbage collection will not delete block DB data that is at or after | ||
| // (LatestBlock - RollbackWindow - LookbackWindow). This ensures that however far the system | ||
| // rolls back inside the rollback window, it is still possible to read at least LookbackWindow | ||
| // blocks of history below wherever it landed. | ||
| // 3. Garbage collection will eventually delete block data older than | ||
| // (LatestBlock - RollbackWindow - LookbackWindow). | ||
| // | ||
| // Eventually, in guarantee 3, because PruneHistory only records a watermark: reclamation also | ||
| // waits for BlockDBConfig.RetentionTime and for LittDB's own GC to come round. | ||
| var _ gc.PrunableStore = (*blockDB)(nil) | ||
|
|
||
| func (s *blockDB) Name() string { | ||
| return "BlockDB" | ||
| } | ||
|
|
||
| // ExternalPruning is unconditionally true: this store has no pruner of its own for the collector to | ||
| // collide with. LittDB's GC reclaims what PruneBefore has already released, on a | ||
| // config.RetentionTime timer, so it enforces no retention policy — it only carries out one this | ||
| // store has recorded. | ||
| func (s *blockDB) ExternalPruning() bool { | ||
| return true | ||
| } | ||
|
|
||
| // PruneHistory advances the retention watermark to blockNumber. It only records the watermark; | ||
| // reclamation happens on LittDB's own GC schedule and no earlier than config.RetentionTime (see | ||
| // PruneBefore). | ||
| // | ||
| // blockNumber is a minimum shared across every managed store, so it may sit above this store's | ||
| // own head — a store that ingests ahead of blockDB pulls the head up, and a QC boundary can | ||
| // leave the newest retained cohort below it. PruneBefore caps the request at the newest | ||
| // retained (block, QC) pair, which is what keeps the store from emptying itself here. | ||
| func (s *blockDB) PruneHistory(blockNumber uint64) error { | ||
| return s.PruneBefore(types.GlobalBlockNumber(blockNumber)) | ||
| } | ||
|
|
||
| // PruneSnapshots does nothing: blockDB keeps no snapshots. It restores by reading the blocks it | ||
| // holds, so its whole retention story is the watermark PruneHistory moves. | ||
| func (s *blockDB) PruneSnapshots(uint64) error { | ||
| return nil | ||
| } | ||
|
|
||
| // GetRollbackFloor returns head - rollbackWindow, the contract's answer for a contiguous store: | ||
| // every block from its floor to its head is retained, so that height is restorable directly and | ||
| // nothing below it has to be held back. It keeps no snapshots, so there is nothing to resolve the | ||
| // window against beyond its own head. | ||
| // | ||
| // It reports against its own head even when that runs ahead of the fleet's, because the collector | ||
| // takes a minimum across stores. A lagging store therefore sets the depth, and answering high here | ||
| // cannot prune anything out from under it. | ||
| // | ||
| // 0 when the window is deeper than the whole history — including the empty store, whose head is 0. | ||
| // Nothing here is eligible for pruning yet: the rollback owed reaches past genesis, so no part of the | ||
| // history can be given up until the head clears the window. Nothing is logged from here; the | ||
| // collector logs every store's answer each cycle. | ||
| func (s *blockDB) GetRollbackFloor(rollbackWindow uint64) uint64 { | ||
| head, err := s.GetLatestBlock() | ||
| if err != nil || head <= rollbackWindow { | ||
| return 0 // cannot say what it holds, so nothing may be dropped anywhere | ||
| } | ||
| return head - rollbackWindow | ||
| } | ||
|
|
||
| // GetLatestBlock returns the newest block number written, or 0 when none has been. | ||
| // | ||
| // Global block numbers start at genesis block 0, so a store holding only that block is | ||
| // indistinguishable from an empty one. That is the safe direction: GetRollbackFloor then answers 0, | ||
| // which holds the fleet's history where it is, and the prune it receives is capped by PruneBefore | ||
| // to a no-op. | ||
| // | ||
| // Reports the written cursor, not the flushed one. A block that a crash would lose still counts | ||
| // as ingested — recovery re-derives this cursor from what survived, so the head can only move | ||
| // back, never past a prune that was already issued. | ||
| func (s *blockDB) GetLatestBlock() (uint64, error) { | ||
| s.mu.Lock() | ||
| defer s.mu.Unlock() | ||
| if !s.hasBlocks { | ||
| return 0, nil | ||
| } | ||
| return uint64(s.lastBlockNumber), nil | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[suggestion] This comment states as fact that
ExternalPruningis taggedmapstructure:"-", butreceipt_config.go:74tags itmapstructure:"external-pruning". Pertestutil/configtest/AGENTS.mdthese manifest exclusions are the recorded contract a replacement implementation reads, so a comment that misdescribes the tag is the specific drift the suite is meant to prevent. Fix the tag (preferred) or the comment.