fix(index): preserve zone map correctness for all data types by wirybeaver · Pull Request #8190 · lance-format/lance

wirybeaver · 2026-08-03T14:08:59Z

Summary

This is now a focused follow-up to #8017. It preserves that PR's merged nested-type enablement, Python validation removal, FSL scalar-index discovery, index naming fixes, and scalar/vector coexistence coverage while adding the remaining correctness and compatibility work:

add sound extrema and NaN behavior for Lance-supported ordered scalar types that previously produced missing bounds, including Decimal128/256, fixed-size binary, and dictionary values
add an explicit ordered/null-only mode and advertise only null predicates for null-only zone maps
write null-only zone maps as version 1 with physical Arrow Null extrema columns while leaving existing ordered maps on version 0
preserve the persisted mode and exact top-level null bitmap across append/update, seed updates, segment merges, and optimization
persist and validate per-zone seed null offsets; fall back to a scanned update for legacy seeds that cannot reconstruct exact null rows
add a checked-in Lance 9.0.0 version-0 fixture and document both physical layouts and reader behavior

Design Decisions

Represent the runtime layout with an explicit ZoneMapMode::{Ordered, NullOnly}. Convert to the protobuf supports_min_max boolean only at the persistence boundary; an absent field means ordered mode for legacy details.
Classify new indices by logical query-ordering semantics through lance_arrow_stats::supports_ordered_extrema, rather than only checking Arrow nesting or physical representation.
Keep ordered zone maps on index version 0. Write null-only zone maps as version 1 so older readers ignore the new layout and safely fall back to scanning.
Store null-only min and max columns physically as Arrow Null arrays. Persist the indexed logical type separately in a data_type global buffer encoded as an Arrow IPC schema.
Preserve the persisted mode during loading, updates, seed harvesting, and segment merges instead of reclassifying an existing index with the current type classifier.
Use the serialized RowAddrTreeMap as the complete top-level null bitmap. Child nulls and empty collections do not make the containing row null.
Store exact per-zone null offsets in new seed payloads. Reject malformed or incomplete payloads, and use the scanned update path when a legacy payload lacks offsets.
Apply dictionary ordering and NaN behavior using the dictionary's logical values, not its integer keys. Dictionaries of non-orderable values use null-only mode.
Use a Lance 9.0.0 fixture to verify that released version-0 ordered maps retain equality, null, range, IN, and value-range behavior.

Relationship to #8017

The remaining differences and their rationale are summarized in a follow-up comment on #8017. This branch is rebased on its merge commit and intentionally does not duplicate or replace its Python/planner changes.

Related to #7987.

Test Plan

cargo fmt --all
cargo check --workspace --tests --benches
cargo clippy --all --tests --benches -- -D warnings
cargo test -p lance-arrow-stats (83 passed; 3 doctests passed)
cargo test -p lance-index --no-fail-fast (1020 passed; 2 ignored; 7 doctests passed)
cargo test -p lance test_dataset_null_only_zonemap_for_list_column --no-fail-fast
cargo test -p lance test_released_zonemap_fixture_preserves_ordered_behavior --no-fail-fast

github-actions · 2026-08-03T14:09:12Z

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

Start a vote following the Lance community voting process.
Format specification modifications need 3 binding +1 votes (excluding the
proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
Once the vote passes, link the completed vote in this PR. It should not be
merged until the vote is linked.

westonpace

This is a great follow-up, a nice extensive set of tests (I appreciate that it even has a migration test) and I think a worthy use of a new version.

It is a spec change, so we will need a vote. I'll get that started and try to come back later and look at this with more detail.

westonpace · 2026-08-05T15:00:27Z

+#[derive(Clone, Copy)]
+#[repr(u32)]
+enum ZoneMapIndexVersion {
+    Ordered = 0,
+    NullOnly = 1,
+}


Are you saying that version 1 is only used for the null-only case? In other words, a writer will choose version 0 or version 1 based on the data type?

I think we want version numbers to be more of an increasing, inclusive concept. In other words...

Version 0 does not know how to create null-only zone maps.
Version 1 can create everything version 0 can and can also do null-only zone maps.

Is this a correct understanding?

Yes, that is the intended compatibility model. ZoneMapIndexPlugin::version() reports the maximum version this implementation supports (1), while each created index records the minimum reader version required by its physical layout. Ordered indices therefore continue to write version 0, and null-only indices write version 1. A version-1 implementation can read and create both layouts; a version-0 reader retains ordered indices and ignores null-only indices so it falls back to scanning.

westonpace · 2026-08-05T15:12:07Z

+fn serialize_data_type(data_type: &DataType) -> Result<bytes::Bytes> {
+    let schema = Arc::new(arrow_schema::Schema::new(vec![Field::new(
+        "value",
+        data_type.clone(),
+        true,
+    )]));
+    let mut buffer = Cursor::new(Vec::new());
+    let mut writer = FileWriter::try_new(&mut buffer, &schema)?;
+    writer.finish()?;
+    Ok(bytes::Bytes::from(buffer.into_inner()))
+}


Why do we need to store the data type in the index?

Null-only zone maps store min and max physically as Arrow Null, so their file schema no longer carries the indexed logical type. The scalar-index loader receives the index store and details, but not the dataset field, and the logical type is needed after loading to validate updates and rebuild the correct processor/seeds. The data_type global buffer preserves it as a one-field Arrow IPC schema. Ordered version-0 maps still infer the type from min/max and do not write this buffer.

westonpace · 2026-08-05T16:37:06Z

            Field::new("null_count", DataType::UInt32, false),
            Field::new("nan_count", DataType::UInt32, false),
            Field::new("zone_length", DataType::UInt64, false),
+            Field::new("null_offsets", DataType::Binary, false),


What is null_offsets here?

null_offsets contains the exact top-level-null row positions relative to the start of each seed zone, encoded as little-endian u64 values in the binary field. null_count alone is insufficient to reconstruct the complete RowAddrTreeMap when an append/update is built from seeds. During seed loading, each zone-relative offset is combined with the zone start and fragment ID to recover the absolute row address. Legacy seeds without this field deliberately fall back to the scanned update path.

Keep dense-null seed payloads compact while validating decoded offsets and documenting zone span and null-query guarantees.

lance-gatekeeper

🟡 Gate recommendation: maintainer decision required.

The rebase preserves ordered version-0 compatibility while isolating null-only maps in version 1; no independent code blocker remains.

Maintainers still need to decide whether to adopt Zone Map Version 1: version 1 enables the explicit null-only layout and exact null pruning for non-orderable types while older readers safely fall back to scans. Retaining version-0-only behavior avoids the new stable layout but forgoes that capability. The required format vote still has no recorded votes or result.

github-actions Bot added enhancement New feature or request A-index Vector index, linalg, tokenizer A-format On-disk format: protos and format spec docs labels Aug 3, 2026