Summary
Please consider supporting pure-Rust compression backends (or making the compression codecs pluggable), as an alternative to the current C-backed lz4-sys / zstd-sys.
Current behavior
The compression feature bundles four codecs and is all-or-nothing:
compression = ["lz4", "flate2", "zstd", "snap"]
lz4 → lz4-sys (C, liblz4)zstd → zstd-sys (C, libzstd)flate2 → miniz_oxide (Rust) ✅snap → Rust ✅
So consumers that need lz4 or zstd (the common broker defaults) unavoidably link two C libraries, even though pulsar-rs is otherwise a pure-Rust client and offers pure-Rust runtimes (tokio-rustls-runtime-*) for TLS.
Motivation
We hit a production memory issue that traces directly to the C codecs. Under a large backlog, ~90 consumers decompress concurrently; the C malloc calls (invisible to a prefixed jemalloc build) went to glibc, whose per-thread arenas bloated RSS to ~15Gi and were never returned to the OS. Pure-Rust codecs would route those allocations through the application's configured Rust allocator, making them observable and reclaimable.
Beyond our specific case, pure-Rust codecs also help:
- musl / fully-static builds (no C toolchain / cross-compile pain for liblz4/libzstd)
- supply-chain / audit surface (fewer
-sys crates and system libs)
- build simplicity (no cmake/cc for these codecs)
Proposed options (any one would help)
- Per-codec pure-Rust feature flags, e.g.
lz4-rust (via lz4_flex) and zstd-rust (via ruzstd), mutually exclusive with the existing C-backed features.
- A trait-based codec registry so applications can supply their own compress/decompress implementations.
- At minimum, split
compression into per-codec features so users can pull in only what they need.
Notes / caveats
ruzstd is decompression-focused; a producer path may still need a Rust zstd encoder (or keep the C encoder). For consumer-only workloads, decompress-only is sufficient.- Happy to help test or contribute a PR if there's agreement on the preferred approach (option 1 seems least invasive).
Thanks for maintaining pulsar-rs!
Summary
Please consider supporting pure-Rust compression backends (or making the compression codecs pluggable), as an alternative to the current C-backed
lz4-sys/zstd-sys.Current behavior
The
compressionfeature bundles four codecs and is all-or-nothing:lz4→lz4-sys(C, liblz4)zstd→zstd-sys(C, libzstd)flate2→miniz_oxide(Rust) ✅snap→ Rust ✅So consumers that need lz4 or zstd (the common broker defaults) unavoidably link two C libraries, even though
pulsar-rsis otherwise a pure-Rust client and offers pure-Rust runtimes (tokio-rustls-runtime-*) for TLS.Motivation
We hit a production memory issue that traces directly to the C codecs. Under a large backlog, ~90 consumers decompress concurrently; the C
malloccalls (invisible to a prefixed jemalloc build) went to glibc, whose per-thread arenas bloated RSS to ~15Gi and were never returned to the OS. Pure-Rust codecs would route those allocations through the application's configured Rust allocator, making them observable and reclaimable.Beyond our specific case, pure-Rust codecs also help:
-syscrates and system libs)Proposed options (any one would help)
lz4-rust(vialz4_flex) andzstd-rust(viaruzstd), mutually exclusive with the existing C-backed features.compressioninto per-codec features so users can pull in only what they need.Notes / caveats
ruzstdis decompression-focused; a producer path may still need a Rust zstd encoder (or keep the C encoder). For consumer-only workloads, decompress-only is sufficient.Thanks for maintaining pulsar-rs!