Skip to content

Print builtin classes, loop bound methods and imported modules as Jinja2 does (0305, 0309) - #721

Merged
rmorozov merged 6 commits into
masterfrom
claude/project-thread-o8q2jl
Nov 1, 2026
Merged

rmorozov merged 6 commits into
masterfrom
claude/project-thread-o8q2jl

Conversation

@rmorozov

@rmorozov rmorozov commented Nov 1, 2026 •

Copy link
Copy Markdown
Member

Requested by Ruslan · project thread

Parity batch 2, group B.

Problem

  • 0305: Jinja2's globals range, dict, cycler, joiner and namespace are classes. Python prints them as <class 'range'> and <class 'jinja2.utils.Cycler'>, and xmlattr keeps them. Jinja2C++ printed nothing for any callable except a macro. The same gap covered loop.cycle and loop.changed, which Python prints as <bound method LoopContext.cycle of <LoopContext 1/2>>.
  • 0309: pprint of a global printed <callable>. {% import 'lib' as m %} bound a plain dict, so {{ m }} and pprint printed {'x': 1}. In Jinja2, str(m) is the module's output and repr(m) is <TemplateModule 'lib'>.

Resulting behaviour

  • Callable::SetRepr gives each of the five globals its static class text (MakeClass in global_functions.cpp). It is not an attribute, so range.name stays undefined. ValueRendererBase prints the text, so it shows wherever a str or repr is taken: {{ }}, ~, %, format, pprint, inside containers, and in xmlattr. lipsum is a function in Python, whose repr names an address, so it still prints nothing.
  • loop.cycle and loop.changed print as bound methods, through Callable::SetBoundMethod(self, method, receiver) with an IBoundReceiver.
    • The receiver is a LoopCycleCell that replaces LoopState::cyclePosition. It holds the position and a pointer to the loop while the loop's frame holds it. The cell is detached in Recycle and ~LoopFrame, and keeps the last position and length after that.
    • The callable does not keep the loop alive. If it did, a kept callable would make the loop collect its rest at the end and run the filter on items Jinja2 never filters (statements.loop_cycle_stored_filtered_break).
    • A repr taken while the loop runs counts the rest the same way printing loop does.
  • import ... as m binds a ModuleMapAdapter (internal_value.cpp), an object of class TemplateModule.
    • Its repr argument is the template's name. The new IMapAccessor::GetStrValue() holds the module's top-level output, which ImportedModule::Render now keeps.
    • {{ m }} prints that output, unescaped under autoescape, as Jinja2's __html__ gives it (markup.h).
    • pprint, [m] and string give <TemplateModule 'lib'>.
    • Because it is an object, m|length, for k in m, m|list, m.items(), m|dictsort and m|tojson now fail, as they do in Jinja2.
    • Each module makes one object (ImportedModule::MakeObject), so two imports of a shared module compare equal and sameas.
  • Divergence lines removed: calls.global_class_str, calls.global_class_xmlattr, statements.loop_cycle_string, filters.pprint_global_and_module.
  • New corpus cases: 8 for 0305 (calls, statements) and 12 for 0309 (loader.module_object_*).
  • Unit tests: ForLoopTest.BoundMethodAndClassRepr (narrow and wide) and ImportTest.ModuleObjectPrintsAsTemplateModule (narrow, wide, autoescape).

Filed follow-ups (each pinned by a divergence line)

  • 0366: a loop.cycle kept past a break out of a lazily filtered loop prints its length as ?. Jinja2 iterates the rest of the loop to count it.
  • 0367: m is escaped is False where Jinja2 says True, and join and xmlattr take str() of a module where Jinja2 takes __html__. The fix belongs in testers.cpp and filters.cpp, which group A owns.

Overlaps

ImportStatement::ImportNames in statements.cpp now takes const ImportedModule&. Group C's 0299 uses the same signature but different lines in that function, so the train needs one manual merge there: C's namespace branch should end with = module.MakeObject(std::move(importedNs)). string_converter_filter.cpp (group A) gets one 4-line branch, in the safe/striptags/forceescape __html__ block. The allow-lists calls.txt and statements.txt are shared with groups A and C; only this PR's own lines change.

Validation

  • GCC Debug with strict warnings: full suite 8338 passed, 6 skipped.
  • Parity: expectations regenerated with Python 3.12.3 and Jinja2 3.1.6.
  • scripts/preflight.sh --perf: clang-format, clang-tidy 22.1.8, null comparisons, use_count, CodeQL pattern and task files all pass. The instruction gate passes; the largest change is Render/macros +0.32%, and Render/for_loop_vars is +0.03%.
  • Verifier on clang ASan+UBSan: 8336 passed with no sanitizer report. It also ran about 65 adversarial templates (stored cycle and changed callables across reused, pooled-out and recursive frames, kept loops, break and continue, wide strings).
  • Chat-template suite (test: Hugging Face chat template suite against transformers (0360 phase 1) #720) on a local merge with master b4249a6 (not pushed): passes, no stale lines; full suite 9785 passed.
  • Not checked here: MSVC and Apple Clang. The verifier reviewed the code for them only.

Process

  • Roles: main session implemented; verifier ran 1 round, which found 1 blocking issue (two imports of a shared module compared unequal), now fixed.
  • Red CI pushes: none. On the first head, the libstdc++ debug-containers job lost its runner mid-build (infra; passed on re-run), and the two macos-14 jobs were cancelled by GitHub's image brownout (infra; ci: move macOS jobs off the retiring macos-14 runner image #724 moved them off macos-14). Head 4d9e639: all green.
  • Codex: 1 finding, 1 fixed, 0 declined (P2: forceescape of a module returned its output unescaped; fixed in string_converter_filter.cpp's __html__ branch, which is group A's file, a 4-line region next to the host-object branch). Round 2 on 4d9e639 was clean. The requested security review never started.

Generated by Claude Code

range, dict, cycler, joiner and namespace print as their Python class
(<class 'range'>), in {{ }}, xmlattr, pprint and every str()/repr() site.
loop.cycle and loop.changed print as bound methods with their loop
(<bound method LoopContext.cycle of <LoopContext 1/2>>) through a cell
they share with the loop, which does not keep the loop alive. Files 0366
for the length after a break out of a lazily filtered loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AWNL7mSUNFU4HfnrEazAXX
{% import 'lib' as m %} binds an object of class TemplateModule: str()
is the module's top-level output (unescaped under autoescape, its
__html__), repr() is <TemplateModule 'lib'>, and its exports stay
attributes. pprint of a builtin class prints the class (0305). Files
0367 for m is escaped (testers.cpp, another group's file).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AWNL7mSUNFU4HfnrEazAXX
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AWNL7mSUNFU4HfnrEazAXX
…l (0309)

Found by the verifier: two imports of one shared module compared unequal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AWNL7mSUNFU4HfnrEazAXX
@rmorozov rmorozov self-assigned this Nov 1, 2026

rmorozov commented Nov 1, 2026

Copy link
Copy Markdown
Member Author

I checked this branch against current master (b4249a6) on a local merge, which was not pushed.

  • ctest -R chat_templates passed, and no line in test/chat_templates/divergences.txt went stale.
  • The full jinja2cpp_tests run passed 9785 tests.

Generated by Claude Code

rmorozov commented Nov 1, 2026

Copy link
Copy Markdown
Member Author

CI status on 118d961:

  • Passed: everything else, including all three MSVC jobs, macOS 15, both sanitizer jobs, parity, fuzz and the instruction counts.
  • libstdc++ debug containers C++20 (gcc-14): the first run failed during Build with "The hosted runner lost communication with the server". No test had run, so I re-ran it once, and it passed.
  • macos-14 AppleClang C++17 / C++23 Release: cancelled before any step ran. The annotation says this is GitHub's scheduled brownout of the macOS 14 image, which is retired on 2026-11-02 ([macOS] macOS 14 Sonoma retirement: GitHub Actions and ADO arm64 on November 2; ADO Intel on November 23 actions/runner-images#13518). That is not caused by this PR. The workflow needs those jobs moved off macos-14, and I've raised it with the batch lead.

@codex review
@codex security review (this changes autoescape handling in markup.h for imported-module objects)


Generated by Claude Code

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-01T17:59:13.551888Z 4d9e639 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 118d961870

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/markup.h Resolved
…309)

Codex P2: forceescape returned the module's output unescaped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AWNL7mSUNFU4HfnrEazAXX

rmorozov commented Nov 1, 2026

Copy link
Copy Markdown
Member Author

4d9e639 fixes the forceescape finding with new logic in the escaping path, so I'm asking for another pass. The security review I asked for earlier never started.

@codex review
@codex security review


Generated by Claude Code

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: 4d9e6395e0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@rmorozov
rmorozov marked this pull request as ready for review November 1, 2026 18:25
rmorozov added a commit that referenced this pull request Nov 1, 2026
Carries #721 (parity batch 2 group B, 0305/0309). Perf approved on this head (Render/dict_ops +0.60% named item). Verifier 1 round, Codex 1 P2 fixed; the requested security review never started.
@rmorozov
rmorozov merged commit 111ee7b into master Nov 1, 2026
46 checks passed
rmorozov pushed a commit that referenced this pull request Nov 1, 2026
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DTQTriEsFrrBgcYNKDUUF4
rmorozov pushed a commit that referenced this pull request Nov 1, 2026
Resolve the divergence allow-lists: #721 removed the 0305 lines and #728 removes the 0306/0310 lines.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DTQTriEsFrrBgcYNKDUUF4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants