TCP B3-B8: compression frames (CityHash v1.0.2, LZ4/ZSTD), on by default

[alex-clickhouse]

Compression on the native protocol, in both directions, on by default with LZ4. Compression=lz4|zstd|none on a connection string, or ClickHouseTcpClientOptions.Compressor for a codec instance.

The frame, and the two spans that are easy to confuse

[16 B  checksum ] CityHash128 over the 9-byte header + body, low64 LE then high64 LE
[ 1 B  method   ] 0x02 NONE · 0x82 LZ4 (block format, no magic) · 0x90 ZSTD (raw frame, with magic)
[ 4 B  csize LE ] counts the 9-byte header, EXCLUDES the checksum
[ 4 B  usize LE ] plaintext length
[ N B  body     ] N = csize - 9

The checksum covers the header and body, so the declared sizes are protected too; compressed_size counts the header but not the checksum. Both spans are asserted separately.

CityHash needed its own implementation. ClickHouse uses CityHash v1.0.2, the historical variant; release 1.1 changed the 128-bit functions, so any 1.1+ package disagrees on every input and every frame it writes is rejected. No NuGet package ships 1.0.2, so CityHash102 is a port of the reference city.cc that ClickHouse vendors as contrib/cityhash102. Two details a port gets wrong silently, both commented in place: CityMurmur compares a signed ssize_t, so a length below 16 must compare as negative; and the tail of a long input walks backwards past the current position, which is in range only because the main loop already consumed 128 bytes.

The expected values in the tests come from compiling that reference and hashing the same inputs, so they pin the port against the implementation the server uses rather than against our own output. As an end-to-end check, that same reference reproduces byte for byte the checksum on a real 4 KB LZ4 frame the server produced over HTTP compress=1 (low=0x931eb99176f35655 high=0xed539c8c0e18acc2). The generator is kept at ignored-docs/tcp/cityhash102-vectors.cc.

Design decisions and trade-offs

Where plaintext lands: two readers, not one. The frame decoder reads compressed bytes from the existing raw buffer — one owner of socket bytes — and decodes into a plaintext array. The question is how values are then decoded from it:

	chosen (A)	alternative (B)
plaintext	copied into a second ReadBuffer	read directly from the frame array
ClickHouseBinaryReader	untouched	must take an IByteSource interface
extra memory per compressed connection	one 16 KiB buffer	none
cost to users with compression off	none	an interface dispatch per primitive read

B is tempting — one less copy, one less buffer, and a simpler leftover invariant — but its cost is global: it de-inlines a sealed class on ReadVarUIntAsync, the hottest loop in the driver, for everyone. And A's copy is smaller than it looks, because ReadBuffer.ReadIntoAsync already reads bulk payloads (whole columns) straight from the frame into the destination; the second buffer only handles the metadata minority of bytes. Filed as B3a with the condition that a T1 read-throughput number must justify it first.

Read-ahead stops at the current frame. ReadBuffer is deliberately greedy — asked for one byte it will take 16 KiB — and that is not changed. Instead the frame source refuses to serve across a boundary, so a greedy request gets a short read. The rule that falls out is: a frame is pulled only inside a read that is still unsatisfied, i.e. only while a decoder is waiting. This is load-bearing rather than tidy: reading ahead past a block's last frame would consume the next packet's uncompressed envelope as frame bytes, and a test asserts the envelope survives untouched.

EndBlock() asserts a block consumed its frames exactly. The format's invariant runs one way only — every block end is a frame boundary, but a frame boundary is not a block end — and the forward direction is what makes an assertion possible. Leftover plaintext means the column decoders and the peer disagree about the body's length; failing there beats desynchronising three blocks later.

Only Data, Totals and Extremes bodies are framed. This was the sharpest trap. Log and ProfileEvents also carry blocks and go through the same block reader, but the server sends them uncompressed at our 54460 target (they become compressible only at 54481+). Framing them would read a block name as a frame checksum, breaking any query against a server with send_logs_level set. FramedPackets states the list once and a test pins it, including that every other packet stays out.

Write side streams frames. Plaintext is cut into frames of at most the server's own ~1 MiB and each finished frame is flushed, so outstanding compressed bytes stay at one frame however large the block. The existing between-column flush threshold still bounds plaintext one level up, so the two mechanisms compose.

Default on with LZ4, as the cheapest codec in CPU and the lightest on the server — what clickhouse-client uses on this protocol. It is flipped in its own one-line commit, after the wiring commit, so that if the flip breaks something the cause is not in doubt.

A per-query override is deliberately absent. The wire flag is per query, but a nullable IClickHouseCompressor cannot express "off for this query" when the client default is on — null already means inherit. That wants a small three-state type and its own decision, filed as B3b. Today the codec is fixed for a connection's life; the flag is still written per query.

An HTTP-only codec is refused at construction. GZipCompressor and BrotliCompressor throw from MethodByte. Finding that mid-query would leave the Query packet already promising the server frames.

Differences from the Go client

I read ch-go and clickhouse-go rather than working from memory, and the comparison changed two things in this PR.

Same split point. ch-go reads the table name from the raw stream, then calls EnableCompression() before decoding the block. So "compression switches on mid-packet" is forced by the format, not by our layering.

They swap the source in place; we cannot. proto.Reader holds raw, data and decompressed, and toggling is r.data = r.decompressed. That is safe only because their plaintext path does exact io.ReadFull reads and therefore never buffers, while their greedy buffer (bufio, 128 KiB) sits below the split. Our ReadBuffer is greedy by design, so an in-place swap would serve greedily-read frame bytes as plaintext. Hence two buffers.

The flip side favours ours. In ch-go, a column decoder that under-reads leaves plaintext in r.data; DisableCompression() swaps away, and the next EnableCompression() finds r.pos < len(r.data) and serves stale bytes from the previous block's frame before pulling a new one. Silent desync, and the checksum cannot catch it because every frame was individually valid. Our EndBlock() throws at the block that caused it.

ServerCode.Compressible() admits the same three packets we do — Data, Totals, Extremes. That cross-check is what caught the Log/ProfileEvents trap here before it caught us.

They buffer, we stream. encodeBlock writes the body into a buffer, compresses that region and splices the frame back over it, one frame per block, with a source comment admitting it: // TODO(tdakkota): find out if we can actually stream compressed blocks. That holds the whole plaintext and a compressed copy, which is why clickhouse-go needs MaxCompressionBuffer. Ours is bounded at one frame.

Smaller points. Their header offsets confirm ours byte for byte (hMethod = 16, hRawSize = 17, hDataSize = 21). Their size caps are 128 MB where mine were 1 GiB, copied from our reader's string cap — theirs is the saner bound on a peer-controlled allocation and I have left a note to tighten. CompressionNone in clickhouse-go means do not frame, matching our null Compressor, so method 0x02 is in practice read-only. And encodedLZ4HC == encodedLZ4, so compression effort never changes the wire method — good for our Lz4Level.

Verification

• 3015 pass, 0 fail on net9.0 against a real server, and with the default on every existing integration test now round-trips through the frame layer rather than only the cases written for it.
• 41 new tests. Unit: 34 CityHash vectors from the reference implementation covering every length branch; frame wire bytes and every error path; the streaming reader/writer over a memory stream (bodies spanning many frames, UInt64s straddling boundaries with a frame target that is not a multiple of 8, the next packet's envelope left unread, the leftover assertion, corruption).
• Integration, both codecs: query and insert round trips, bodies larger than one frame in both directions, Log packets arriving mid-stream while compressed, framed Totals, sequential compressed queries on one pooled connection, a compressed and an uncompressed client against the same server, and the server answering ZSTD to a client that asked for LZ4 (the reader dispatches per frame, not on what we configured).
• Whole solution builds on all TFMs, 0 errors, no new analyzer warnings.

Notes

• No CHANGELOG/RELEASENOTES entry, consistent with every PR in this stack; they land with the epic's final step.
• No coverage numbers. Collecting coverage for ClickHouse.Driver.Tcp.Tests produces a report in which all 460 classes read line-rate="0", including files this PR does not touch, while 3015 tests pass — so the report is empty rather than the code uncovered. Pre-existing and filed as S18; likely the Fody/InlineMethod weaver dropping the sequence points coverlet counts.
• Follow-ups filed in the epic TODO: B3a (drop the second buffer, benchmark-gated), B3b (per-query override), S18 (coverage harness).

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds default LZ4 compression to the native TCP protocol, including framed streaming, CityHash v1.0.2 checksums, ZSTD support, configuration, and tests.

Changes:

• Implements compression frame encoding, decoding, and checksums.
• Integrates framed block I/O into TCP queries and inserts.
• Adds compression options and comprehensive unit/integration coverage.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
ClickHouse.Driver.Tests/CompressionFrameTests.cs	Tests frame layout and validation.
ClickHouse.Driver.Tests/CityHash102Tests.cs	Tests CityHash reference vectors.
ClickHouse.Driver.Tcp/Protocol/Query.cs	Writes the query compression flag.
ClickHouse.Driver.Tcp/Protocol/ClickHouseTcpConnection.cs	Integrates compressed block I/O.
ClickHouse.Driver.Tcp/Format/BlockWriter.cs	Separates block envelope and body writing.
ClickHouse.Driver.Tcp/Format/BlockReader.cs	Separates block envelope and body reading.
ClickHouse.Driver.Tcp/Compression/FramedPackets.cs	Defines framed packet types.
ClickHouse.Driver.Tcp/Compression/CompressedFrameWriter.cs	Streams outgoing compression frames.
ClickHouse.Driver.Tcp/Compression/CompressedFrameReader.cs	Streams and validates incoming frames.
ClickHouse.Driver.Tcp/Client/IConnectionFactory.cs	Passes compressors to connections.
ClickHouse.Driver.Tcp/Client/ClickHouseTcpConnectionStringBuilder.cs	Adds the compression connection-string option.
ClickHouse.Driver.Tcp/Client/ClickHouseTcpClientOptions.cs	Adds compressor configuration and defaults.
ClickHouse.Driver.Tcp/ClickHouse.Driver.Tcp.csproj	References shared compression code.
ClickHouse.Driver.Tcp.Tests/Integration/CompressionIntegrationTests.cs	Tests compression against ClickHouse.
ClickHouse.Driver.Tcp.Tests/Compression/FramedPacketsTests.cs	Tests packet framing selection.
ClickHouse.Driver.Tcp.Tests/Compression/CompressedFrameStreamingTests.cs	Tests streaming and frame boundaries.
ClickHouse.Driver.Tcp.Tests/Client/CompressionOptionTests.cs	Tests compression configuration.
ClickHouse.Driver.Tcp.Tests/Client/ClickHouseTcpClientOptionsTests.cs	Covers compressor option copying.
ClickHouse.Driver.Tcp.Tests/ClickHouse.Driver.Tcp.Tests.csproj	References shared compression types.
ClickHouse.Driver.Common/Compression/CompressionFrame.cs	Implements frame encoding and decoding.
ClickHouse.Driver.Common/Compression/CityHash102.cs	Implements CityHash v1.0.2.
ClickHouse.Driver.Common/AssemblyInfo.cs	Grants TCP assemblies internal access.

Suppressed comments (1)

ClickHouse.Driver.Tcp/Client/ClickHouseTcpClientOptions.cs:285

• Making LZ4 framing the default adds hashing, compression, copying, and frame I/O to every TCP query, but the PR provides no throughput/allocation comparison. The repository's performance guidance requires BenchmarkDotNet measurements for common-path changes; include representative query-read results for none versus default LZ4 before enabling it globally.

    public IClickHouseCompressor Compressor { get; init; } = ResolveCompressor(DefaultCompression);

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[alex-clickhouse]

Thanks — one of these was a real bug. Addressed in 9f29035.

Fixed: a failed framed end-of-input marker left the connection reusable. Confirmed and fixed. The query path encodes the Query packet inside a try whose catch resets the buffer and returns the connection to Ready, on the premise that nothing can have been sent yet — and framing the marker broke that premise, because each frame flushes as it is emitted. A socket failure or cancellation there left the Query packet on the wire while the connection went back to the pool looking healthy, so the next caller to lease it would read the previous query's response. Only Query.Write stays in that block now; the marker moved into the outer try, whose failure terminates.

CompressedSendFailureTests pins it, and I restored the old structure to check the test actually catches it: Expected: Terminated, But was: Ready and IsReusable Expected: False, But was: True. Worth noting the insert path needed no change — its finally already terminates unless the operation completed — so the insert case in that fixture passes either way and is a guard rather than a regression test.

Fixed: the frame size caps. Right, and the comment claiming a bad length could not drive a huge rent was self-contradictory. Both sizes come from the peer and each sizes a rent before the checksum can be verified. Now 128 MiB rather than 1 GiB — ample over the server's ~1 MiB flush, and the same figure ch-go uses (maxDataSize/maxBlockSize).

Fixed: both documentation defects. The builder still documented the default as none after the flip to lz4, and Validate's summary and exception tags had been stranded above ResolveCompressor, so Validate was undocumented and the tags described the wrong member.

Declined: the changelog.d fragment. Every PR in this stack deliberately adds none. That is a standing decision (TODO story R3, confirmed 2026-08-17 for the TLS PR), because the native client is [Experimental], has no PublicAPI surface defined and is not packaged yet — so there is no released behaviour to describe, and a per-PR fragment would describe a client no user can reach. R3 exists to sweep the whole stack into one entry when the client ships, and I have added this default to it explicitly so the sweep cannot miss it.

On the benchmark for default-on LZ4 (suppressed comment): fair, and unresolved. I tried to answer it and the honest result is that I cannot with the instrument available:

• Wire size is a real, deterministic win: 3,200,125 → 1,600,940 bytes (2.00x) for SELECT number FROM numbers(400000), and 4,489,037 → 2,429,284 (1.85x) for a mixed UInt64/String/Float payload, measured through the same frames over HTTP compress=1.
• Wall clock over loopback is inconclusive. A 5-run probe gave medians of none 96 ms / lz4 79 ms / zstd 112 ms on the narrow shape but none 185 ms / lz4 273 ms / zstd 124 ms on the wide one. Self-contradictory, so it is noise plus CPU cost, not a ranking — over loopback there is no bandwidth to save, so it charges the cost and none of the benefit. I deleted the probe rather than quote it as support.

So the default currently rests on matching clickhouse-client and on real deployments being bandwidth- and latency-bound rather than CPU-bound. Reasonable, but unproven. T1/T2 (read/insert throughput) are unstarted stories, and I have filed T1a: measure over a non-loopback path or with a bandwidth limit, report rows/s and CPU for none/lz4/zstd on both shapes, and revisit the default before the client ships if LZ4 costs materially there — cheap to change now, expensive later.

Full suite after the fixes: 3018 pass, 0 fail on net9.0 against a real server.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!