Skip to content

Debugger page crashes with UnicodeDecodeError when a source file is not UTF-8 decodable #34

Description

@amol-

Goal

Degrade gracefully when a frame's source file cannot be decoded, instead of
letting the whole debugger page raise.

Supporting arbitrary source encodings is explicitly NOT the goal. A frame whose
source cannot be read should simply render without a source listing, exactly as
already happens when the file is missing.

Done when: rendering a traceback that contains a frame from an undecodable
source file produces a debug page (that frame showing no source lines) rather
than raising, and a test covers it.

Reproduction (reproduces on Python 3.9 through 3.14) - a latin-1 encoded module
executed so that the frame has no usable loader:

src = open('mod_latin1.py', 'rb').read()   # contains a 0xe9 byte
ns = {}
exec(compile(src, 'mod_latin1.py', 'exec'), ns)
from backlash.tbtools import get_current_traceback
try:
    ns['boom']()
except Exception:
    tb = get_current_traceback()
tb.render_full()

Current behaviour:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 52:
invalid continuation byte

render_full, render_source and plaintext all raise, so the middleware
fails while trying to report the original error and the user sees nothing
useful.

Cause: Frame.sourcelines calls plain open(self.filename), which decodes
using the locale/default encoding, and only guards against IOError.

Suggested minimal fix: widen the existing guard so a decode failure is treated
like an unreadable file, for example
except (OSError, UnicodeDecodeError): return []. The IOError spelling in
that guard is also the Python 2 alias for OSError and can be modernized in
the same edit.

The same crash occurs for ordinary UTF-8 sources when the interpreter's locale
encoding is ASCII (verified with LC_ALL=C PYTHONCOERCECLOCALE=0 PYTHONUTF8=0
on 3.9 and 3.14), which makes this reachable without any unusual source
encoding.

Note that _coding_re in backlash/tbtools.py is now unused; it is the
leftover of a coding-cookie detection block that only ever ran on Python 2.
It can be removed alongside this fix.

Why

A debugger that raises while rendering an error is worse than one that shows an
incomplete frame: the original exception is lost and the middleware itself
becomes the failure. Falling back to "no source available" keeps the rest of
the page, including the traceback, the other frames, and the console.

The locale-encoding variant means this is not limited to legacy encodings, so a
container or CI environment with a minimal locale is enough to trigger it. No
special setup on the application side is required.

Verified as pre-existing rather than a regression: the pre-modernization code
carried an encoding-detection block, but on Python 3 it was already unreachable

  • text-mode open() returns str, so f.read() raised before the detection
    logic could run. The identical reproduction fails on the preceding commit.

References

  • backlash/tbtools.py (Frame.sourcelines, Frame.current_line, the unused
    _coding_re)
  • backlash/tbtools.py (Traceback.render_full, Traceback.plaintext) are the
    callers that currently propagate the error
  • tests/test_tbtools.py may be a natural home for a regression test

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions