fix(columnar): estimate BlockwiseLinear's residuals per block, not per column - #3050
fix(columnar): estimate BlockwiseLinear's residuals per block, not per column#3050marcbachmann wants to merge 3 commits into
Conversation
…r column `serialize` divides the values by the column's gcd before it trains each block's line, so `BlockwiseLinearEstimator` has to account for a division it cannot perform: the column's gcd is only known once every value has been collected. It accounted for it by measuring the residual widths on the raw values and then subtracting `log2(gcd) * num_rows / 8` bytes from the total. That credit is column-wide and unbounded, while the saving it stands for is per block and bounded by that block's own width. A column whose blocks are flat -- a large increment gcd but residuals already at zero -- has nothing to save, yet the credit is subtracted anyway and `saturating_sub` clamps the estimate to 0. Selection is a `min_by_key` over the estimates, so a zero estimate wins every column it is offered. Measured on `benches/hdfs.json` (100k rows), the estimate reaches 0 on columns `BlockwiseLinear` goes on to write 1290 to 7982 bytes to. On a fixed-cadence timestamp column it wins against `Linear` by 1750 bytes to 28. Normalize each block by its own increment gcd when the residual is measured -- the column's gcd always divides it -- and divide by the ratio in `estimate`. A block whose own gcd is 1 settles right away -- the column's gcd divides every block's, so a block gcd of 1 forces the column's to 1 and leaves nothing to divide by -- and only the blocks that did have a gcd are held until `estimate` knows the ratio. Columns whose blocks carry no common factor therefore keep the old single running counter and never allocate; on `benches/hdfs.json` that is every timestamp column in seconds, and 188 of 196 blocks of the hourly-bucketed one. Holding those blocks is load-bearing: dropping the ratio and trusting the per-block normalization alone still passes 3000 random columns, but on a column whose blocks have a gcd of 1000 while the column's is 1 it estimates 165065 bytes against the 290458 `serialize` writes, and picks `BlockwiseLinear` where `Bitpacked` is smaller. Over 3000 random columns the estimate now lands within 0.89x..1.05x of the bytes `serialize` writes, where it previously ranged from 0x to 54.9x, and selection picks the smallest available codec on every one of them. Measuring the block costs a pass the old formula did not need. The block's minimum and gcd come out of one pass that stops computing the gcd once it reaches 1, and the division is skipped entirely in that case, which is the common one. In isolation the estimator is then 1.24x the old one on gcd-1 columns and 2.03x when there is a gcd to divide by; against a whole `serialize_u64_based_column_values` over `[Bitpacked, BlockwiseLinear]` that is +1.7% and +6.6% (1M rows, 15.5ms and 16.6ms respectively). `serialize` is unchanged, so this only moves which codec is picked, not how any codec encodes. Claude-Session: https://claude.ai/code/session_01S2MMqqhJiEJqaAu6BFCiwg
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e710a93e8d
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
`compute_slope` gives up above a `1 << 31` endpoint delta and returns a flat line. The estimator measures each block after dividing by the block's own gcd, so its endpoint delta is `scale` times smaller than the one `serialize` will see. A block can therefore sit under the cutoff here, get an exact line and a zero residual, while `serialize` -- working in units of the column's gcd -- lands above it, trains a flat line and bitpacks the full span. Two 512-row ramps of step `1 << 32` with the second offset by one are enough: every block has a gcd of `1 << 32`, the offset drops the column's gcd to 1, and the estimate came out at 28 bytes against the 5269 `serialize` writes. Scaled to 200 blocks that is 1415 bytes estimated for 525416 written -- metadata only, for a half-megabyte encoding, which is the same defect this branch set out to fix. Detect the crossing from the block's own endpoint delta and charge those blocks the full 64 bits rather than a residual that does not apply. The residual of a flat line cannot be derived from the one measured here: past a `1 << 32` span the intercept heuristic in `Line::train_from` stops picking the block minimum and the residuals wrap, so the block's amplitude is not an answer either -- measured 34 bits predicted against 64 actual before this was reverted to the conservative bound. Costs an over-estimate of at most ~1.6x on the blocks that cross, and only on those. Over 3000 random columns extended with this shape the worst under-count goes from 0.003x to 0.73x of the bytes written; the earlier corpora are unchanged at 0.89x..1.05x, selection still picks the smallest available codec on every column, and the hdfs index is byte-identical. Claude-Session: https://claude.ai/code/session_01S2MMqqhJiEJqaAu6BFCiwg
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a952b6cb13
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
`BlockwiseLinearEstimator` measures each block after dividing by the block's own gcd and rescales the measurement by `block_gcd / column_gcd`, because `serialize` works in units of the column's gcd. Two review findings on that rescaling. The residual widths do not scale. `Line::train` rounds its slope into a 32 bit fraction and `Line::eval` floors, so the line `serialize` trains is not the measured line scaled up. Two 512-row blocks alternating between two values one apart and ending on the higher one, multiplied by 3 with the second offset by one -- block gcd 3, column gcd 1 -- measure a residual of 1, which this estimated at 2 bits from `1 * 3`, while `serialize` retrains on 0/3 values and needs 3 bits for a residual of 5. Repeated over 40 blocks the estimate came out at 5370 bytes against the 8291 written, and `BlockwiseLinear` was picked over `Bitpacked`'s 7687. Charging 64 bits to the blocks that cross the slope cutoff over-estimates them. Two 512-row ramps of step `1 << 23`, the second based at `(1 << 40) + 1` -- block gcd `1 << 23`, column gcd 1 -- get a flat line from `serialize`, and a flat line still only spans the block: 32 bits per row, 4122 bytes. The estimate said 8220, so `Bitpacked` won with the 41 bits the column-wide range needs, 28% larger. Measure each block against the exact line through its endpoints instead of the fixed point one. Those residuals are the same at either scale, so they rescale exactly; they are held multiplied by `num_rows - 1` to stay integral. Both fixed point lines land within one unit of the exact line, so the estimate charges a unit -- except where the delta spreads evenly over the rows and the slope comes out exact, which is what keeps a rescaled ramp at the 0 bits `serialize` writes for it. A decreasing line always pays it: `compute_slope` complements to `u64::MAX`, one short of zero. A block that crosses the cutoff is charged its own amplitude, which is the residual of the flat line `serialize` falls back to. The `2^32` guard stays. `Line::train_from` only finds the smallest residual when it sits within `2^32` of the one at the first row; past that the intercept lands too high and the rest wraps, which really is 64 bits. Over 156000 rescaled blocks -- 4 seeds x 20000 columns of planted per-block gcds, ramps, staircases, alternations and scatter, every branch exercised -- no block is under-charged, 1.8% are charged exactly one bit too many, and a block `serialize` writes at 0 bits is never charged more. At the column level the estimate lands within 0.74x..1.38x of the bytes written, against 0.65x..12.0x before this commit and 0x..41.4x before the branch, and selection picks within 1.02x of the smallest codec, against 1.92x and 36.6x. The widest ratios sit on columns of a few hundred bytes. The hdfs and gh fast fields are byte-identical. What is left of the under-count is not the residual widths: the estimate charges the metadata of the line it trained, whose slope and intercept are in the block's own units and so serialize to shorter `VInt`s than the ones `serialize` writes. That is worth up to ~10 bytes per block, and predates the per-block normalization that widened it. Measuring the exact line costs one pass per rescaled block. The estimator goes from 6.74ms to 8.63ms on a 1M row column whose blocks all carry a gcd, +5% over a whole `serialize_u64_based_column_values`; columns without per-block gcds are untouched, 6.25ms to 6.28ms. Claude-Session: https://claude.ai/code/session_013A2TdSNBPVYQbhmuR77D8o
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ec83eba732
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| let (span, first_span) = if endpoint_delta >= 1 << 31 { | ||
| ( | ||
| self.amplitude.saturating_mul(scale), | ||
| self.first_val.saturating_mul(scale), | ||
| ) |
There was a problem hiding this comment.
For descending blocks near u64::MAX, the zero-slope line can encode the wrapping complement rather than the ordinary amplitude used here. Fresh evidence after the earlier cutoff fix is two 512-row blocks [A, 0, ...] and [A + 1, 1, ...], where A = u64::MAX - (1 << 40): their local gcd is A and the column gcd is 1, so this branch estimates 64 bits per row, but serialization chooses intercepts A/A + 1 and writes residual 2^40 + 1, requiring only 41 bits. The overestimate makes codec selection prefer 64-bit Bitpacked data even though BlockwiseLinear is roughly 36% smaller, so the cutoff calculation should account for the line's modular residual interval.
Useful? React with 👍 / 👎.
| } else { | ||
| self.values_num_bits += compute_num_bits(max_value) as u64 * u64::from(num_rows); | ||
| } | ||
| self.meta_num_bytes += 1 + line.num_bytes(); |
There was a problem hiding this comment.
When a block has a large offset from the column minimum, this counts the VInt sizes of the locally normalized line rather than the line that serialize actually writes. For example, after an initial block containing 0 and 1 makes the column gcd 1, a constant block at 1 << 56 is normalized to all zeros here, so its intercept is counted as one byte, while serialization trains an intercept of 1 << 56, whose VInt requires nine bytes. The resulting eight-byte underestimate repeats for every such block and can bias codec selection in columns mixing constant and less-compressible blocks; compute the metadata size from the rescaled line as well as its residual width.
Useful? React with 👍 / 👎.
| let num_rows = self.block.len() as u32; | ||
| if block_gcd.get() > 1 { | ||
| let (residual_span, first_residual_span) = exact_line_spans(&self.block); | ||
| self.gcd_blocks.push(GcdBlock { |
There was a problem hiding this comment.
When most blocks have a local gcd greater than one, this retains a GcdBlock for every 512 input rows, changing the estimation pass from a roughly 4 KiB block buffer to memory linear in the column length. Each entry contains two u128s plus five u64s and a row count (roughly 80 bytes after alignment), so a 100-million-row timestamp or counter column can add about 15 MiB of estimator state, and billion-row columns about 150 MiB, even though values are otherwise streamed. This can materially increase peak indexing memory or cause OOMs on large segments; the summaries should be folded or represented without one allocation-sized record per block.
Useful? React with 👍 / 👎.
| let Some(scaled) = span.checked_mul(u128::from(scale)) else { | ||
| return u64::MAX; | ||
| }; | ||
| let span = scaled.div_ceil(u128::from(self.idx_last_val())) + u128::from(rounding); |
There was a problem hiding this comment.
Adding rounding after already ceiling the scaled exact span can overestimate the retrained residual by a full bit. Fresh evidence in the current revision is a 512-row block [0, ..., 0, 3] followed by [1, ..., 1, 4]: each local block normalizes to [0, ..., 0, 1] with gcd 3, while the column gcd is 1. This calculation returns a span of 4 and estimates 3 bits per row, but Line::train on the serialized 0/3 values produces residuals requiring only 2 bits; consequently the estimator chooses 3-bit Bitpacked data, while BlockwiseLinear's actual data plus footer is about 24% smaller. Model the fixed-point rounding exactly rather than unconditionally adding it to the ceiled span.
Useful? React with 👍 / 👎.
|
@PSeitz can you review? |
2 participants
The bug
BlockwiseLinearEstimator::estimatecan return 0 bytes for a column thatBlockwiseLinearCodec::serializethen writes megabytes to. Codec selection is amin_by_keyover the estimates, so a zero estimate wins every column it is offered.serializedivides the values by the column's gcd before it trains each block's line,which narrows every residual. The estimator cannot do the same — the column's gcd is only
known once every value has been collected — so it measured the residual widths on the raw
values and then corrected for the division afterwards:
That credit is column-wide and unbounded, while the saving it stands for is per block and
bounded by that block's own residual width. A column whose blocks are flat — a large
increment gcd, but residuals already at zero — has nothing to save, and the credit is
subtracted anyway until
saturating_subclamps the estimate to 0.The fix
Measure each block's residual after normalizing the block by its own increment gcd.
The column's gcd always divides a block's gcd, so
estimatecan rescale by the ratio oncestats.gcdis known, per block, with nothing left to saturate.serializeis untouched: this changes which codec gets picked, not how any codec encodes.Why some blocks are held back
The old estimator held a single running byte count. This one still does, for every block
whose own increment gcd is 1: the column's gcd divides every block's, so a block gcd of 1
forces the column's to 1 and there is nothing left to divide the residual by. Only the
blocks that did have a gcd are held as
(residual, gcd, rows)untilestimateknows theratio. On
benches/hdfs.jsonthat is nothing at all for the timestamp columns in seconds,and 8 of 196 blocks for the hourly-bucketed one -- those columns never allocate.
Holding them is load-bearing. Dropping the ratio and trusting the per-block normalization
alone collapses it back to one counter and survives 3 000 random columns unscathed, but it
is wrong on exactly the shape this PR is about: on a column whose blocks each have a gcd of
1 000 while the column's gcd is 1, it estimates 165 065 bytes where
serializewrites290 458, and picks
BlockwiseLinearat 290 458 bytes whenBitpackedneeds 275 010.A fixed-size histogram of residual magnitudes is the other way to stay O(1). Measured on the
same 3 000 columns (mis-picks / bytes over best,
[Bitpacked, BlockwiseLinear]): 65 buckets(520 B) 105 / +0.118%, 1 025 buckets (8 KB) 55 / +0.018%, 4 097 buckets (32 KB) 36 / +0.009%.
All beat
main, none are exact.Evidence
All measurements below are on
main(039a729) vs this branch.1. Estimate vs. bytes actually written, on
benches/hdfs.json(100 000 rows)Columns derived from the dataset's own
timestampandseverityfields:mainserializewritesOn the last row the zero estimate also changes the outcome: selection writes 1 750 bytes as
BlockwiseLinearwhereLinearneeds 28 — a 62x miss on an evenly sampled timestampcolumn.
The credit and the measured total both scale linearly with the row count, so what saturates
the estimate is the shape, not the size — the zero does not go away on larger columns. Same
minute-bucketed shape, one bucket per 1 000 rows:
serializewrites2. An index built from that dataset
100 000 documents, single segment,
timestamp/hour_bucket/sampled_atas i64 fastfields and
severityas a str fast field.sampled_atcarries a fixed 250 ms cadence, theshape a metrics pipeline produces.
serialize_column_mappable_to_u64only offersBitpackedandBlockwiseLinear, so this is that two-codec path.Index sorted by
timestamp:main.fasttotalIn document order the two are byte-identical (477 801), so nothing regressed.
3. Estimate accuracy over 3 000 randomly shaped columns
Ratio of
BlockwiseLinearEstimator::estimateto the bytesserializewrites:main4. Selection quality over the same 3 000 columns
Each column is serialized with every codec individually to find the smallest, then with the
whole set to see what selection picks. "off" counts columns where the pick is more than 2%
larger than the smallest available.
Over
ALL_U64_CODEC_TYPES, and over the[Bitpacked, BlockwiseLinear]pair thatserialize_column_mappable_to_u64actually offers:mainRepeating it with gcds drawn up to 2^39 instead of realistic ones gives the same picture:
main275 / +0.485% and 103 / +0.432%, this branch 0 / +0.000% on both.5. Cost
Measuring a block costs a pass the old formula did not need. The block's minimum and gcd
come out of a single pass that stops computing the gcd once it reaches 1, and the division
is skipped entirely in that case -- the common one.
In-process A/B of the estimator alone (best of 7,
min), against the old single-counterversion doing the same
Line::train:In absolute terms that is +0.27 ns per row with no gcd and +1.1 ns per row with one:
+0.18 ms and +1.09 ms over a million rows. Against a whole
serialize_u64_based_column_valuesover[Bitpacked, BlockwiseLinear]at 1M rows (15.5 msand 16.6 ms), +1.2% and +6.6%.
The per-block storage itself does not register: swapping the
Vecfor 16 inline entrieswith a spill measures 0.99x-1.03x at every size above, including the 512-row column.
Tests
Two new tests in
columnar/src/column_values/u64_based/tests.rs, both red onmain:estimation_blockwise_linear_accounts_for_the_gcd_per_block— fails withestimated 0 bytes, wrote 739test_selection_does_not_pick_a_much_larger_codec— fails withfixed_cadence over [Bitpacked, Linear, BlockwiseLinear]: selection wrote 1750 bytes (BlockwiseLinear) when 28 were availableThe second one asserts the outcome rather than the estimate: whatever selection picks has to
land within 5% of the smallest codec actually available, over six column shapes and both
codec sets.
cargo test --workspaceis green (1 674 tests, exit 0),cargo clippy -p tantivy-columnar --all-targetsis clean.https://claude.ai/code/session_01S2MMqqhJiEJqaAu6BFCiwg