Skip to content

Fix apultra safe_dist under-reporting for multi-block inputs#898

Open
meeq wants to merge 1 commit into
DragonMinded:previewfrom
meeq:apultra-encoder-bug
Open

Fix apultra safe_dist under-reporting for multi-block inputs#898
meeq wants to merge 1 commit into
DragonMinded:previewfrom
meeq:apultra-encoder-bug

Conversation

@meeq

@meeq meeq commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Problem

For inputs larger than BLOCK_SIZE (1 MiB), apultra_compress can write a stats.safe_dist that is too small for safe in-place decompression. Consumers that size their in-place buffer from safe_dist (e.g., bufsize = orig_size + (safe_dist + cmp_size - orig_size)) will see the decoder's output pointer overtake the unread compressed input, corrupting subsequent reads.

The bitstream itself is valid — streaming/non-in-place decoders decode correctly. Only in-place decoders are affected, and only on multi-block inputs.

Symptom

libdragon's mkasset -c 2 (vendored apultra) on a 2,282,582-byte input produced cmp_size = 1,621,797, safe_dist = 660,785 (i.e. emitted inplace_margin = 0). libdragon's in-place asm decoder returned at byte 1,619,454 (premature accidental EOS in corrupted-input territory). Traced byte-exact decode showed the actual worst-case (output_pos − input_pos) reached ~672,860 (safe_dist was under-reported by ~12 KB).

Root cause

In apultra_write_block:

const int nCurSafeDist = (i - nStartOffset) - nOutOffset;
if (nCurSafeDist >= 0 && pCompressor->stats.safe_dist < nCurSafeDist)
   pCompressor->stats.safe_dist = nCurSafeDist;
  • i - nStartOffset is bytes consumed from the current block's input.
  • nOutOffset is bytes written to the current block's output (reset to 0 at the top of apultra_write_block; pOutData is already advanced by the caller).

So nCurSafeDist measures the per-block local lead of output over input. For a single-block compression the local lead equals the global lead and the bug is invisible. For multi-block compression the local lead resets to 0 at each block boundary but the global lead, which is what an in-place decoder actually experiences, does not. The accumulated lead from earlier blocks is lost.

Concretely, if block 0 finishes with delta_0_end = block_0_input − block_0_output bytes of lead, then during block 1 the actual global_delta is delta_0_end + local_block_1_delta. stats.safe_dist misses the delta_0_end term and any later additions.

Fix

Thread nPriorDelta — the cumulative (decoder-output − decoder-input) at block start — through apultra_compressor_shrink_blockapultra_optimize_and_write_blockapultra_write_block, and add it to nCurSafeDist. The top-level loop in apultra_compress already has the cumulative nOriginalSize and nCompressedSize, so the value is just:

const int nPriorDelta = (nOriginalSize - (int)nDictionarySize) - nCompressedSize;

Dictionary bytes are not emitted by the decoder, so they're excluded from the input side.

Also submitted upstream as emmanuel-marty/apultra#26

@meeq meeq force-pushed the apultra-encoder-bug branch from 62a90e9 to 1808087 Compare June 8, 2026 18:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant