Add opt-in zstd page compression to reduce database size - #1159
Conversation
Introduce a `Compression` option on DB/Options that enables transparent zstd compression of leaf and branch page data. Compression happens at node spill time—before page allocation—so fewer pages are allocated for compressible data. Decompression is transparent on read via a per-transaction cache using sync.Map for concurrent reader safety. Key changes: - New orthogonal CompressedPageFlag (0x20) on page headers; type checks changed from == to bitwise & so the flag coexists with page types - CompressInodes serializes and compresses node data, only used when it reduces the page count - Split threshold increased 4x when compression is enabled so nodes accumulate enough data for meaningful compression - DecompressPage preserves on-disk overflow for correct freelist accounting This was largely written by Claude Opus. Signed-off-by: Thomas Jungblut <tjungblu@redhat.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: tjungblu The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
498f893 to
5ff64cf
Compare
|
/retest |
Benchmark iteration (rough medians, -benchmem -count=5) Benchmark Before (~ns/op) After (~ns/op) CompressInodes_Small ~9.8µs ~8.2µs CompressInodes_Large ~13.4µs ~11.4µs CompressInodes_OverPool ~41.5µs ~32.6µs DecompressPage ~14.7µs ~8.7µs Signed-off-by: Thomas Jungblut <tjungblu@redhat.com>
5ff64cf to
bcdd1ce
Compare
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
This enables the bbolt compression raised upstream in etcd-io/bbolt#1159 Do not merge, this is only for testing purposes. /hold Signed-off-by: Thomas Jungblut <tjungblu@redhat.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: tjungblu The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
3 participants
Introduce a
Compressionoption on DB/Options that enables transparent zstd compression of leaf and branch page data. Compression happens at node spill time—before page allocation—so fewer pages are allocated for compressible data. Decompression is transparent on read via a per-transaction cache using sync.Map for concurrent reader safety.Key changes:
This was largely written by Claude Opus.
Benchmark results and comparison to snappy and without compression can be found here: #1149 (comment)