feat: add physical column layout optimization - #8614
Conversation
|
Important This PR touches the Lance format specification. Substantive changes to the format specification — the If this is a meaningful format change:
|
github-actions
Bot
added
A-format
wjones127
left a comment
wjones127
left a comment
There was a problem hiding this comment.
Providing some high level suggestions for now, since this isn't marked ready for review yet.
There was a problem hiding this comment.
suggestion: An alternative simple design would be to just use the existing DataReplacement operation. If you want to represent that fact that values didn't change, you can add a data_change: bool argument that indicates whether data changed or was just moved. The data_change parameter is how I'm implementing this concept in Composite Transactions, so I think it's more aligned with our future roadmap. You can see this in the AddDataFile action here: https://github.com/lance-format/lance/pull/8644/changes#diff-0ad4747ac7f2cd6192e47a4c12b1887a193a66de1d07eb1623518c3fa462c2ffR15-R26
| /// groups: vec![ColumnGroup { | ||
| /// fields: vec!["feature_a".into(), "feature_b".into()], | ||
| /// }], |
There was a problem hiding this comment.
suggestion: Instead of having to pass these each time, can we also make this a table configuration? I was making some docs that list all the table config keys, and I noticed we have already added the compaction options there. So I think this might naturally fit amongst those.
Those configs aren't universally enforced, so we should separately make that happen.
| } | ||
|
|
||
| /// Reorganize selected top-level fields into explicit per-fragment data files. | ||
| pub async fn optimize_columns( |
There was a problem hiding this comment.
suggestion: It would be nice to not have to create a whole new API for this. Ideally this would run in the same compaction jobs. I say this partly to keep API simple. But also if someone is regularly running background maintenance jobs, it's nice to have fewer of those maintenance jobs rather than many separate ones, especially if they might conflict.
2 participants
Wide datasets that materialize fields independently can accumulate one provider file per field per fragment. This inflates manifests and descriptor traversal, while row compaction is not the right maintenance primitive because it changes row layout and row addresses.
This PR introduces a physical-only
OptimizeColumnsoperation. Callers provide explicit top-level field groups, and Lance rewrites those groups into standard per-fragment data files while preserving logical values, fragment IDs, physical row order, deletion files, stable row IDs, and row-version metadata. A fixed snapshot cutoff materializes visible overlays while newer overlays remain authoritative, and field-level conflict intent allows safe rebasing of disjoint work.Index coverage is conservatively removed for rewritten fields, including legacy indices whose unknown coverage is recovered during commit. Blob columns reuse the existing rewrite machinery for legacy Blob, Blob v2, nested Blob v2, and external references.
Benchmark
Measured at PR head
b308a540on an EC2c7i.4xlargeinus-east-2. The dataset has 100,000 rows, 1,000 independently materialized scalar fields, and 64 fragments. Optimization groups 10 fields per output file. Values below are median application-cold full-stream latencies from fresh dataset handles; dataset-open time is measured separately and excluded.The physical layout shrank from 64,064 to 6,528 live files (-89.8%), while the manifest shrank from 4.78 MB to 1.47 MB (-69.3%). All before/after logical checksums matched.
This is a maintenance operation with a one-time rewrite cost: 58.96 s on local FS and 121.05 s on S3 for this dataset, writing about 418 MB. Grouping also determines read amplification: the deliberately scattered S3 narrow projection improved by only 1.11x while bytes read increased from 1.31 MB to 9.52 MB. Callers should therefore group fields that are commonly read together.
This is an explicit layout-policy API rather than an automatic scheduler. Index coverage may require rebuilding after maintenance, and superseded provider objects remain until normal version retention and cleanup make them collectible. Benchmark harnesses are maintained outside this repository.