feat(index): add zone map support for all data types by westonpace · Pull Request #8017 · lance-format/lance

westonpace · 2026-07-27T21:07:17Z

We are using zonemap as our "statistics" and it is also important for recording nullability bitmaps. As a result, we need it to support all types. For nested types we don't track the min/max but we still track the nullability bitmap and the null count (we could track more things in the future)

codecov · 2026-07-27T21:54:40Z

Codecov Report

❌ Patch coverage is 98.36066% with 6 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-index/src/scalar/zonemap.rs	98.36%	2 Missing and 4 partials ⚠️

📢 Thoughts on this report? Let us know!

wjones127

Looks good! Just a minor nitpick on the code, though looks like there's a python test failure that seems legit.

wjones127 · 2026-07-31T02:38:48Z

+                min: ScalarValue::try_from(&self.data_type)
+                    .map_err(|e| Error::invalid_input(format!("{e}")))?,
+                max: ScalarValue::try_from(&self.data_type)
+                    .map_err(|e| Error::invalid_input(format!("{e}")))?,


nitpick: this is a non-obvious way to construct a null. Perhaps would be more clear as a ScalarValue::try_new_null()

Yes, great idea, I've switched to this.

Adds per-position min/max statistics for FixedSizeList columns so the zone map can prune zones during equality queries on fixed-size list (e.g. embedding vector) columns. Key changes: - New `FixedSizeListZoneAccumulator` tracks per-position min/max as `Vec<Option<ScalarValue>>` across all non-null list entries in a zone. - `ZoneMapProcessor` dispatches through a new `ZoneMapAccumulator` enum that selects between scalar and FSL accumulators based on the column type. - `evaluate_fsl_equals` performs per-position range checks: a zone is pruned when any element of the query list falls outside `[min[j], max[j]]`. - `Equals` and `IsIn` queries delegate to `evaluate_fsl_equals` for FSL targets; `Range` and `LikePrefix` conservatively return true. - `value_range` / `value_range_over` return `None` for FSL zones. - The `is_nested()` guard in `new_training_request` and `create_seed_writer` is lifted for `FixedSizeList<non-nested>` types. - `default_use_seeds` returns true for all `FixedSizeList` columns. - Six new tests cover pruning, null lists, IsIn, NaN elements, all-null zones, and value_range behaviour. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Replace the per-position FixedSizeList accumulator with a simpler design: all nested types (List, LargeList, FixedSizeList, Struct, Map, etc.) are now supported, tracking only null_count and null bitmap — min/max are stored as typed null ScalarValues. For non-nested types, behavior is unchanged. For nested types, query evaluation prunes only when null_count equals the zone length (all rows null) or when checking IsNull/null equality; all other queries conserve. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ments - Restore deleted comment in Equals handler and simplify comments in value_range/value_range_over/default_use_seeds per PR review - Add test_zonemap_fsl_column: verifies zone map creation on a FixedSizeList column and that IS NULL is ScalarIndex-accelerated - Add test_vector_and_zonemap_on_fsl_column: verifies vector index (IVF_PQ) and zone map can coexist on the same FSL column, with vector search and IS NULL both working correctly Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…idation The Python-level type guard for create_scalar_index incorrectly rejected FixedSizeList and other nested types for ZONEMAP indexes. Split the BTREE/ BITMAP check from the ZONEMAP check and add pa.types.is_nested() as an accepted type for ZONEMAP. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

ZONEMAP works on any column type at the Rust level (nested types get null-only tracking, everything else gets min/max). Remove the Python whitelist entirely instead of extending it. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

scalar_index_info filtered out all FixedSizeList columns using is_vector_field(), which classifies any FSL type as a vector column. Now that zone maps can be created on FSL columns, this caused the zone map to be invisible to the query planner and IS NULL / other queries on FSL columns never used the scalar index. Remove the is_vector_field() pre-filter from the outer loop; the inner index_details.is_vector() check already correctly skips true vector indexes based on index type, not column type. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

wirybeaver · 2026-08-04T14:42:26Z

I have a PR to implement the zonemap for all data types as well. Will take a look whether I can merge something
#8190

wirybeaver · 2026-08-04T14:59:42Z

I compared this merged implementation with #8190. This PR covers the core nested-type enablement and also includes important Python/planner fixes that #8190 should preserve. There are still a few useful follow-ups in #8190:

Classify by supported logical ordering, not only is_nested(). In the merged code, every non-nested type follows the ordered query path, but lance-arrow-stats does not currently produce extrema for several non-nested orderable types (for example Decimal128/256, fixed-size binary, and dictionaries). Those zones get null extrema; an equality/range query can then conclude that the zone has no finite minimum and prune it even when it contains a match. fix(index): isolate null-only zone maps in version 1 #8190 adds extrema implementations plus a table-driven no-false-negative matrix.
Preserve exact null rows through seed updates. try_update_with_seeds rebuilds the zone list but leaves builder.null_rows unset, while the seed payload contains counts but not row offsets. The resulting index loses both the existing and appended exact null bitmap. fix(index): isolate null-only zone maps in version 1 #8190 stores per-zone null offsets, validates them, merges the resulting RowAddrTreeMap, and falls back to scanning for legacy seeds without offsets.
Advertise null-only capabilities explicitly. fix(index): isolate null-only zone maps in version 1 #8190 persists ordered/null-only mode and gives null-only indexes a parser that accepts only IS NULL / IS NOT NULL, so other predicates go directly to the normal scan instead of unnecessarily consulting the index.
Version the new physical layout. It keeps existing ordered zone maps at v0, writes null-only maps at v1 with physical Arrow Null extrema columns and the logical type in a separate buffer, and includes a released-v9 v0 fixture. This lets older readers ignore the new layout instead of attempting to interpret it as the old ordered format.

I’ll rebase #8190 onto this merge and narrow it to these incremental pieces, retaining this PR’s Python validation, FSL scalar-index discovery, and scalar/vector coexistence fixes.

github-actions Bot added A-index Vector index, linalg, tokenizer enhancement New feature or request labels Jul 27, 2026

westonpace commented Jul 29, 2026

View reviewed changes

github-actions Bot added the A-python Python bindings label Jul 29, 2026

westonpace changed the title ~~feat(index): add zone map support for FixedSizeList of primitive types~~ feat(index): add zone map support for all data types Jul 29, 2026

westonpace marked this pull request as ready for review July 29, 2026 18:33

westonpace requested a review from wjones127 July 29, 2026 18:33

wjones127 approved these changes Jul 31, 2026

View reviewed changes

westonpace force-pushed the feat/zone-map-fsl branch 2 times, most recently from 9216a4d to 938635c Compare August 4, 2026 11:33

westonpace and others added 9 commits August 4, 2026 05:06

Factor the index type into calculating a unique index name

42df3e1

Remove functions incorrectly added during rebase

0b855c8

Handle IndexType::Scalar correctly

aefbbba

westonpace force-pushed the feat/zone-map-fsl branch from 938635c to aefbbba Compare August 4, 2026 12:06

cargo fmt and clippy

f06da69

westonpace merged commit ea3d0d3 into lance-format:main Aug 4, 2026
40 checks passed

wirybeaver mentioned this pull request Aug 4, 2026

fix(index): isolate null-only zone maps in version 1 #8190

Open

Navigation Menu

Uh oh!

feat(index): add zone map support for all data types - #8017

feat(index): add zone map support for all data types#8017
westonpace merged 10 commits into
lance-format:mainfrom
westonpace:feat/zone-map-fsl

westonpace commented Jul 27, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Jul 27, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wjones127 left a comment

Uh oh!

wjones127 Jul 31, 2026

Uh oh!

westonpace Aug 3, 2026

Uh oh!

Uh oh!

wirybeaver commented Aug 4, 2026

Uh oh!

wirybeaver commented Aug 4, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

westonpace commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wjones127 left a comment

Choose a reason for hiding this comment

Uh oh!

wjones127 Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

westonpace Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wirybeaver commented Aug 4, 2026

Uh oh!

wirybeaver commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

westonpace commented Jul 27, 2026 •

edited

Loading

codecov Bot commented Jul 27, 2026 •

edited

Loading

wirybeaver commented Aug 4, 2026 •

edited

Loading