feat: report the row ids deleted between two versions by wkalt · Pull Request #8589 · lance-format/lance

wkalt · 2026-08-17T17:23:26Z

A delta can stream the rows a version range inserted or updated, but not the ones it deleted, so a caller cannot see removals without reading the whole dataset.

The result is the ids live at the begin version and absent at the end version. Each begin fragment's deletion-vector growth is mapped through its row id sequence, and ids still live at the end -- moved by compaction or update, or revived by restore -- are excluded with a sort-merge anti join that spills past the session memory pool, so memory stays bounded. Version 0 is the empty snapshot; reversed ranges and endpoints without stable row ids are rejected. Batches follow the scanner's default size.

Comparing endpoints surfaced a restore bug, fixed separately in #8671. Exposed in Python and Java alongside the existing readers.

codecov · 2026-08-17T20:32:30Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

A delta can stream the rows a version range inserted or updated, but not the ones it deleted: they cannot be scanned at the end version, leaving a caller unable to see removals without reading the whole dataset. The result is the ids live at the begin version and absent at the end version. Each begin fragment's deletion-vector growth is mapped through its row id sequence, and ids still live at the end are excluded -- a compaction or update moves rows without deleting them, and a restore revives them in place. A moved row can only land in a fragment the range created and a revived one where a deletion vector shrank, so the exclusion is a sort-merge anti join against exactly those newly live ids, run the way merge_insert runs its joins: bounded by the session memory pool and spilling past it. Version 0 is the empty snapshot, and a reversed range is rejected rather than read backwards. Batches follow the scanner's default size, bounding a consumer's per-batch work rather than the read. Requires stable row ids at both endpoints: a history can hold versions from before the ids were enabled, and either endpoint without them is rejected rather than misread. Exposed in Python and Java alongside the existing readers.

lance-gatekeeper

✅ Gate recommendation: approve.

The rebase keeps the feature patch unchanged and places the restore high-water fix in the base. Endpoint deletion semantics, spill-bounded anti-join behavior, and single-pass row-ID traversal remain intact on the current head.

Xuanwo

Thank you, nice work!

wkalt mentioned this pull request Aug 17, 2026

feat: materialized views lancedb/lancedb#3959

Closed

wkalt marked this pull request as draft August 17, 2026 20:32

wkalt marked this pull request as ready for review August 17, 2026 20:32

github-actions Bot added A-python Python bindings A-java Java bindings + JNI labels Aug 18, 2026

wkalt force-pushed the ticket/ent-1961/delta-deleted-row-ids branch from ee949fb to f7e54da Compare August 18, 2026 14:29

github-actions Bot added the enhancement New feature or request label Aug 18, 2026

wkalt force-pushed the ticket/ent-1961/delta-deleted-row-ids branch from f7e54da to caf9029 Compare August 18, 2026 14:58