fix(fsst): reject corrupt symbol tables and offsets on decode - #8588
Conversation
Treat on-disk FSST symbol lengths and value offsets as untrusted input so a crafted table cannot overflow the 8x decode buffer.
github-actions
Bot
added
A-encoding
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
lance-gatekeeper
Bot
added
the
K-changes
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
westonpace
left a comment
westonpace
left a comment
There was a problem hiding this comment.
Changing back to request changes since gatekeeper's suggestions seem valid.
Reject corrupt offsets on the raw-copy path, and return InvalidData when an FSST_ESC is missing its payload inside the current value.
lance-gatekeeper
Bot
removed
the
K-changes
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The revision closes both previously identified malformed-input paths: validation now covers switch-off offsets, and escape payloads are bounded to their current value. The decoder now enforces the intended corruption boundary while preserving valid-file behavior and the existing fast-path safety invariants.
lance-gatekeeper
Bot
added
the
K-approved
westonpace
left a comment
westonpace
left a comment
There was a problem hiding this comment.
Nice job! Good to see a safe & fast impl 😄
2 participants
Why
#7589made the FSST output-buffer contract 8×, but still trusted on-disk symbol lengths and value offsets. A crafted Lance file can inflatelens[]sodecompress_bulkwrites past that buffer. Readers that open untrusted datasets (dataset viewers, upload scanners) crash, and the overflow is a heap write with attacker-controlled values and stride.This change makes
fsst::decompressthe security boundary. Declared symbol lengths must be1..=8. Offsets must convert withto_usize, be non-decreasing, and stay inside the compressed buffer. Corrupt input returnsInvalidData, mapped tocorrupt_fileby the encoding adapters. Valid files and the 8×write_unalignedfast path are unchanged.Benchmark
FSST string decode of a 1 MiB Hamlet corpus, 2000
decompresscalls, release, same host. The measured head is this PR; the baseline isorigin/main(8a8fb20c32).The difference is within run-to-run noise. Decode cost is unchanged.