Skip to content

feat: validate mixed V2 data file versions - #8581

Draft
Xuanwo wants to merge 1 commit into
xuanwo/mixed-file-versions-1-capabilityfrom
xuanwo/mixed-file-versions-2-validation
Draft

feat: validate mixed V2 data file versions#8581
Xuanwo wants to merge 1 commit into
xuanwo/mixed-file-versions-1-capabilityfrom
xuanwo/mixed-file-versions-2-validation

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • atomically enable the paired mixed-version capability with complete base and overlay storage validation
  • derive the capability from the final manifest and dispatch reads and column metadata by each exact normalized file version
  • define data_format.version as the default output fallback rather than a snapshot-wide maximum or identity
  • reject unknown identities, V1 and V2 mixtures, and non-fallback files without capability

Stack

  1. feat: add mixed data file version capability #8580 - capability reservation and commit gates
  2. feat: validate mixed V2 data file versions #8581 - capability activation, storage validation, and exact read dispatch
  3. feat: support per-operation V2 write targets #8582 - per-operation write targets
  4. feat: propagate exact V2 versions across dataset operations #8583 - dataset operation propagation
  5. feat: target exact V2 versions in compaction #8584 - compaction targets and binary copy policy
  6. feat: expose exact V2 versions in bindings #8585 - bindings, documentation, and rollout contract

Testing

  • cargo test -p lance-table feature_flags
  • cargo test -p lance-table infer_exact_file_version_and_reject_mixed_fragments
  • targeted mixed scan and storage-contract tests
  • cargo fmt --all -- --check
  • cargo clippy --all --tests --benches -- -D warnings

@github-actions github-actions Bot added the enhancement New feature or request label Aug 17, 2026
@Xuanwo
Xuanwo force-pushed the xuanwo/mixed-file-versions-2-validation branch from d472a84 to e219677 Compare August 18, 2026 18:37
@Xuanwo
Xuanwo force-pushed the xuanwo/mixed-file-versions-2-validation branch from e219677 to dfcb0c0 Compare August 18, 2026 18:43
@Xuanwo
Xuanwo force-pushed the xuanwo/mixed-file-versions-2-validation branch from dfcb0c0 to 37117b1 Compare August 20, 2026 10:27
@Xuanwo
Xuanwo force-pushed the xuanwo/mixed-file-versions-2-validation branch 2 times, most recently from ab44d9b to c49f09c Compare August 20, 2026 14:15
@github-actions github-actions Bot added the A-format On-disk format: protos and format spec docs label Aug 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

  • Start a vote following the Lance community voting process.
    Format specification modifications need 3 binding +1 votes (excluding the
    proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
  • Once the vote passes, link the completed vote in this PR. It should not be
    merged until the vote is linked.

@Xuanwo
Xuanwo force-pushed the xuanwo/mixed-file-versions-2-validation branch from c49f09c to ffba427 Compare August 20, 2026 14:26
@Xuanwo
Xuanwo force-pushed the xuanwo/mixed-file-versions-2-validation branch from ffba427 to 46a1119 Compare August 20, 2026 15:15
@Xuanwo
Xuanwo force-pushed the xuanwo/mixed-file-versions-2-validation branch from 46a1119 to 63b60b1 Compare August 20, 2026 15:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-format On-disk format: protos and format spec docs enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant