Skip to content

fix: prevent stream decode buffer overflow panic (#424) - #603

Open
chiliec wants to merge 1 commit into
goccy:masterfrom
chiliec:fix/stream-buffer-overflow-424
Open

fix: prevent stream decode buffer overflow panic (#424)#603
chiliec wants to merge 1 commit into
goccy:masterfrom
chiliec:fix/stream-buffer-overflow-424

Conversation

@chiliec

@chiliec chiliec commented Aug 18, 2026

Copy link
Copy Markdown

What

Closes #424.

Decoding certain JSON strings from a stream panicked:

panic: runtime error: index out of range [1025] with length 1024
    .../internal/decoder.(*Stream).readBuf(...) stream.go:201
    .../internal/decoder.(*Stream).read(...)    stream.go:214
    .../internal/decoder.stringBytes(...)       string.go:247

Reproduced with a JSON string containing a long run of incomplete multi-byte UTF-8 lead bytes (0xE2).

Cause

When stringBytes encounters invalid or incomplete multi-byte characters it replaces each with RuneError in place:

s.buf = append(append(append([]byte{}, s.buf[:cursor]...), runeErrBytes...), s.buf[cursor+1:]...)
s.length += runeErrBytesLen

This grows s.buf (via append) and s.length, but never updates s.bufSize. Later, when the stream needs more data, readBuf grows the buffer from the stale s.bufSize:

s.bufSize *= 2
s.buf = make([]byte, s.bufSize)
copy(s.buf, remainBuf)

At the panic, the real state was len(s.buf)=1026, s.length=1282, s.bufSize=512. Doubling 512 → 1024 produced a buffer smaller than both the existing buffer and s.length. The copy truncated the data, but s.length still referred to the old size, so the subsequent scan s.buf[s.cursor+i] (for i < s.length-s.cursor) ran off the end.

Fix

In readBuf, resync s.bufSize to the actual buffer length before doubling, and keep doubling until the new buffer can hold everything already read:

if int64(len(s.buf)) > s.bufSize {
    s.bufSize = int64(len(s.buf))
}
s.bufSize *= 2
for s.bufSize < s.length {
    s.bufSize *= 2
}

The crafted input now returns a normal decode error instead of panicking.

Tests

Added TestIssue424 in stream_test.go, reproducing the exact input.

Verified RED→GREEN: without the fix, TestIssue424 panics with index out of range [1025] with length 1024; with the fix it passes.

Validation (real results, Go)

  • go test ./...all packages pass (including test/cover).
  • Added a local round-trip check (not part of this PR): valid strings of 2-, 3- and 4-byte runes and long ASCII that cross the 512-byte stream buffer decode identically to encoding/json, confirming the buffer growth does not corrupt data.
  • gofmt -l clean. (go vet reports a pre-existing UnmarshalJSON signature note in internal/decoder/type.go, present on a clean master checkout and unrelated to this change.)

When decoding a JSON string from a stream, invalid or incomplete
multi-byte UTF-8 characters are expanded to RuneError in place, growing
s.buf via append and s.length, but leaving s.bufSize unchanged. When the
buffer later needs to grow in readBuf, it doubled the stale s.bufSize,
which could produce a new buffer smaller than the current buffer and
smaller than s.length. The copy then truncated the data while s.length
still referred to the old size, so scanning the buffer panicked with
"index out of range".

Resync s.bufSize to the real buffer length before doubling, and keep
doubling until the new buffer can hold everything already read.

Closes goccy#424.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Panic from stream overflow

1 participant