Repository navigation
fix(media): saturate MP3/JPEG size overflows and free FFmpeg packet on throw - #12199
Conversation
ParseMp3Ofl computed frames * samples_per_channel - (delay + padding) in uint32. A hostile Xing/LAME header could wrap the multiplication or make the subtraction underflow, reporting a huge bogus total_samples. Compute in uint64, return 0 when the trimmed amount covers all samples and clamp to UINT32_MAX otherwise.
Add a header whose frame count is smaller than the delay plus padding (expects 0) and one with a 0xFFFFFFFF frame count (expects UINT32_MAX).
… overflows fillImageInfo computed mcuColumns * mcuRows * blocksPerMcu * 128 + 64 in uint32. A progressive header with huge dimensions wrapped the result, so sceJpegDecParseHeader reported a small coefficient_mem_size and the coefficient memory check in sceJpegDecDecode was bypassed. Compute the size in uint64 and fail with the coefficient memory error (0x80650005, already returned when the supplied coefficient memory is too small) when it does not fit in uint32.
A progressive header patched to 65535x65535 must make sceJpegDecParseHeader return 0x80650005 instead of a wrapped size.
…mat is unsupported decodePacket allocated the AVPacket and AVFrame as raw pointers and only released them at the end of the function, so the std::runtime_error thrown for an unconverted sample format leaked both. Hold them in unique_ptr with av_packet_free/av_frame_free deleters so every exit path releases them. Behaviour is otherwise unchanged.
|
Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.
Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.
|
|
A Windows run, since the description lists none. Head
AI-assisted: yes (Claude Code). |
|
Windows result for head Host: Windows 11 26300, Intel i7-12700K, WinLibs GCC 15.2.0 (x86_64-ucrt-posix-seh, r6), Release, The tree also carried the open PRs #12290, #12288, #12285, #12269, #12265, #12232, #12206, #12201, #12197, #12165, #12146, all merged cleanly, so the run is a joint one; the tests of this PR are AI-assisted: yes (Claude Code), the runs and the numbers above are from this machine. |
What
Three independent integer-overflow / leak fixes in media libraries:
ParseMp3Oflcomputedframes * samples_per_channel - (delay + padding)in uint32, so a hostile Xing/LAME header could wrap the product or underflow the subtraction. It is now computed in uint64, returns 0 when delay plus padding covers all samples, and clamps toUINT32_MAX. Existing parse results are unchanged.fillImageInfocomputed the progressive coefficient size (mcuColumns * mcuRows * blocksPerMcu * 128 + 64) in uint32, so huge dimensions wrapped it,sceJpegDecParseHeaderreported a small size and thecoefficient_mem_sizecheck insceJpegDecDecodewas bypassed. It is now computed in uint64; if it does not fit in uint32, both functions return0x80650005, the codesceJpegDecDecodealready returns for insufficient coefficient memory (now a named constant).decodePacketheld theAVPacketandAVFrameas raw pointers freed only at the end, so thestd::runtime_errorfor an unsupported FFmpeg sample format leaked both. They are nowunique_ptrs withav_packet_free/av_frame_freedeleters. No test: reaching that branch needs a decoder that emits a format other than FLTP/S16P, which the bundled MP3/AAC decoders do not do.Each fix is its own commit, followed by its test commit (Ajm and JpegDec).
Not included, on purpose:
ConsumeSamples(out-of-rangellroundwith a huge pitch): open PR feat(ngs2): implement custom PitchShift and external render jobs #11315 rewrites and moves those lines, so a change here would conflict.sceAudioOutOpenlength validation:AudioOutBufferSize.cppasserts that opening with len0x80000000/0x40000000succeeds and that the first output throws, so rejecting at open would change behaviour an existing test asserts.Tested
Linux x86_64. Built
guest_ajm_tests,guest_jpeg_dec_testsandguest_audiodec_tests, and ran ctestguest_ajm,guest_jpeg_dec,guest_audiodec: 3/3 passed. New cases: a LAME header with a frame count smaller than delay plus padding (expects 0) and one with frame count0xFFFFFFFF(expectsUINT32_MAX); a progressive JPEG patched to 65535x65535 (expects0x80650005fromsceJpegDecParseHeader).tools/check_conventions.py --base upstream/mainandgit diff --checkpass. The Audiodec change is only reasoned (existing guest_audiodec still passes). The helper's SYNTAX pre-check failed on missing include paths on the host (known limitation); the build itself exited 0.Checklist