Skip to content

fix: don't corrupt multi-byte runes split across a stream buffer refill - #600

Open
r0h1tb wants to merge 1 commit into
goccy:masterfrom
r0h1tb:fix/utf8-split-across-stream-buffer
Open

fix: don't corrupt multi-byte runes split across a stream buffer refill#600
r0h1tb wants to merge 1 commit into
goccy:masterfrom
r0h1tb:fix/utf8-split-across-stream-buffer

Conversation

@r0h1tb

@r0h1tb r0h1tb commented Aug 1, 2026

Copy link
Copy Markdown

Fixes #574

Root cause

internal/decoder/string.go, the multi-byte branch of the streaming string decoder:

if !utf8.FullRune(s.buf[cursor : len(s.buf)-1]) {

len(s.buf) is the allocated capacity, not the number of bytes actually read — that's s.length. Everything between the two is nul padding.

So when a rune straddles a read boundary, FullRune is handed a truncated sequence followed by 0x00, judges it complete-but-invalid, and the refill immediately below is skipped. DecodeRune then returns RuneError and each byte of the sequence is replaced with U+FFFD — leaving the decoded string 6 bytes longer than the input for a 3-byte rune.

Bounding the check by s.length lets the split sequence fall through to s.read() and be completed by the next chunk.

Reproduction

Decoding ~8KB of Japanese text through an http.Request body reproduces it consistently. Sweeping the payload length, corruption begins at 321 repetitions of "日本語更新テスト " (JSON length 8060) and the first U+FFFD lands at output byte 8006 every time, with len(got) == len(want) + 6:

reps=321 (json len=8060): mismatch, U+FFFD at 8006 (len got=8031 want=8025)
reps=322 (json len=8085): mismatch, U+FFFD at 8006 (len got=8056 want=8050)

The +6 is the fingerprint: one 3-byte sequence becoming three 3-byte replacement characters.

Tests

TestDecodeStreamMultiByteAcrossBufferBoundary feeds the decoder through a reader capped at 1024 bytes per Read, which puts the split at a buffer boundary without needing a socket — the test stays hermetic. It covers 2-, 3- and 4-byte runes across lengths 300–340 so the split lands at every offset within a rune.

All three widths fail on the current default branch:

--- FAIL: TestDecodeStreamMultiByteAcrossBufferBoundary/3-byte
    reps=300: body corrupted
--- FAIL: TestDecodeStreamMultiByteAcrossBufferBoundary/2-byte
--- FAIL: TestDecodeStreamMultiByteAcrossBufferBoundary/4-byte

With the change, go test ./... is green across all packages, matching the pre-change baseline. Separately I swept the original HTTP-body reproduction over 1–1200 repetitions: 7 failures before (test stops early), none after. gofmt clean; the go vet findings in internal/encoder/compiler.go, internal/encoder/string.go and internal/decoder/type.go are pre-existing and untouched here.

Possibly related

#572 (SIGSEGV in appendNormalizedHTMLString after a stream decode) and #571 (index out of range on Windows) both involve decoding from a response body, and a string whose bytes were rewritten in place is a plausible route to both. I could not reproduce either directly — #572 has no minimal case and #571 is Windows-only — so this is a lead rather than a claim, and I haven't marked them fixed.

The multi-byte branch of the streaming string decoder bounded its
utf8.FullRune check by len(s.buf) — the allocated capacity — instead of
s.length, the number of bytes actually read.

Everything between s.length and len(s.buf) is nul padding. When a rune
straddles a read boundary, FullRune therefore saw a truncated sequence
followed by nul and reported it complete, so the refill below was skipped
and DecodeRune returned RuneError. Each byte of the sequence was then
replaced with U+FFFD, leaving the decoded string 6 bytes longer than the
input for a 3-byte rune.

Decoding an ~8KB JSON body of Japanese text over an http.Request body
reproduced this consistently, with the first U+FFFD landing at the same
offset every time.

Bounding the check by s.length lets the split sequence fall through to
s.read() and be completed by the next chunk.

Adds a hermetic regression test that feeds the decoder through a reader
capped at 1024 bytes per Read, covering 2-, 3- and 4-byte runes across a
range of lengths so the split lands at every offset within a rune.

Fixes goccy#574
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Corrupted JSON string when decoding (demo code)

1 participant