fix: defer blank rows when a fully deleted batch has no row to copy

[LuciferYang]

What

Adding or merging a column fails with NotSupported: Missing too many rows in merge, run compaction to materialize deletions first when a fragment's deleted rows cover its first read batch.

This supersedes #7318, which goes after the same failure. Parameterizing #7233's test by where the fully deleted batch sits shows the middle and trailing cases already pass on main, for both nullable and non-nullable new columns. So add_blanks rejecting an empty batch is not a middle-of-fragment problem, and the oversized-batch slicing in that PR does not change any observable behavior. Only a run starting at physical row 0 fails.

Root cause

Every data file in a fragment has to hold the same physical row count, so the updater restores the deleted rows before writing. add_blanks materializes a placeholder by copying the batch's first row, which means the batch needs at least one live row.

A deleted run that trails live rows gets greedily appended to the preceding batch, so it always has one. A run starting at physical row 0 arrives as an empty batch carrying every one of its offsets, and the copy has nothing to copy from.

Fix

Defer those blanks rather than invent values for an arbitrary schema: DeletionRestorer remembers how many rows it owes and prepends them to the next batch that does have a live row. Deleted rows sort before the live rows that follow them, so the physical row order is unchanged. Copying a real row also keeps the placeholder valid for a non-nullable column, which a null placeholder would not.

Legacy files have to reproduce their original row group size, which deferring would break, so they keep reporting the existing error and add_blanks keeps rejecting an empty batch.

Two hardening changes in add_blanks came with it. Offsets that are not strictly increasing used to underflow u32 silently, and an offset past the batch's live rows used to reach arrow::compute::take with bounds checking off. Both now return Internal naming the offending offset.

Tests

• updater.rs: six DeletionRestorer cases covering the deferral, a second empty batch carrying the debt through, the offset shift, is_exhausted while blanks are owed, and no double counting of deferred rows. Plus add_blanks rejection cases for non-increasing, equal, and out-of-range offsets, and a blank landing exactly at the end of a batch.
• schema_evolution.rs: test_add_columns_with_fully_deleted_batch parameterized over leading/middle/trailing by nullable/non-nullable, and a legacy case asserting the error still comes from add_blanks.
• fragment.rs: a deletion vector naming a row past the end of the fragment must fail the stream at its end instead of writing a short data file.
• test_dataset.py: merge_columns plus LanceOperation::Merge over a dataset with a deletion file, parameterized the same three ways, with the merged column declared non-nullable.

[LuciferYang]

@chenyu-x

[LuciferYang]

cc @Xuanwo @wjones127 @hamersaw FYI

[lance-gatekeeper]

✅ Gate recommendation: approve.

The change addresses the leading fully deleted-batch failure at the deletion-restoration boundary: current-format rewrites carry the owed physical rows into the next live batch, preserving row order and non-nullable values, while legacy fixed row-group behavior remains unchanged. The state-machine and end-to-end Rust regressions cover the relevant positions and failure boundaries.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!