Skip to content

Keep only the names a kept closure reads (0375) - #758

Open
rmorozov wants to merge 4 commits into
masterfrom
claude/project-thread-b4totu
Open

rmorozov wants to merge 4 commits into
masterfrom
claude/project-thread-b4totu

Conversation

@rmorozov

@rmorozov rmorozov commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Parity batch 3, group A, second PR: task 0375. Follows #751 (0373).

Problem

A macro kept past the frame it was defined in copied every name of that frame when the frame ended. Python closes over only the names the body reads. Copying everything made a kept closure in a frame of many macros slow to call, kept values alive that nothing reads, and closed reference cycles through containers the closure never touches (0373 item 4).

What changes

  • The resolver records the hash of every name read while a macro or call block is parsed (NameResolver::BeginReads/EndReads, AddUse, AddRead). Indirect reads are recorded too: the namespace of a set ns.attr, a filter or test that falls back to a value of its name, the gettext function of a trans, and the names given as constants to map/select/reject/selectattr/rejectattr.
  • AddReadOfAll stands for every name. It covers an include or import with context, a block, applymacro, and those filters when the name is not a constant.
  • Each macro keeps its sorted, unique read hashes in the arena (MacroStatement::m_bodyReads, checked by VisitRefs).
  • ClosureCapture holds the union of the reads of the closures sharing it and passes it on to the frame below. CollectNamesDownTo copies only those names.
  • self sites and an included template's top level still keep every name, as Jinja2's include context holds them all.

Behaviour note: a host callable taking *context inside a kept closure now sees only the names the closure reads among its ended frames' locals. Python's pass_context sees none of a macro's locals. This is recorded in the task file.

Validation

  • Full suite passes (Debug, GCC). generate.py --check passes. Python 3.12 / Jinja2 3.1.6.
  • New rows:
    • corpus: statements.macro_closure_sets_attr_of_frame_namespace plus five more, loader.macro_closure_includes_with_frame_names, loader.macro_closure_imports_with_context;
    • unit test: MacroTest.KeptClosureAppliesMacroByName.
  • Debug probe:
    • a kept closure in a frame of 40 macros costs 35k instructions a call instead of 283k;
    • the 0373 item 4 loop, when the stored closure reads a name other than the container, stays at 13 MB for 10k and 40k calls (21 and 58 MB before);
    • a sibling closure of the same frame that reads the container still keeps it (0386).
  • preflight --perf against 84be673 passes:
    • Render rows are unchanged (within ±0.04%);
    • Load rows are +0.5% to +2.1% (Load/filters +2.08%, Load/macros +1.57% with one more allocation, for the read list).

Roles, rounds, CI

Roles: verifier (1 round). It found no rendering or lifetime finding in about 600 probe templates compared with Python and with 84be673, plus a clang-18 ASan+UBSan build and run. It blocked on Load instruction counts (+3 to +8%, because every template recorded every read). That is fixed in 3a5bf41 (reads recorded only inside macros, sorted in place) and 263db4b (hashes taken from the interned symbol; no merge when the reads are already held). The gate was then re-run green. CI red pushes: none (two pushes, both green). Codex: 1 finding, 1 fixed, 0 declined (P2: a heap read list per capture, now inline in 42cbb8a); the security review was requested but never started. Scratch merge with master 8e93fe8: ctest -R 'parity|chat_templates' passes.


Generated by Claude Code

A macro or call block records the hashes of the names its parameters and
body read at parse time; the capture shared by a frame's closures holds
their union, passes it to the frame below, and the copy made when the
frame ends keeps only those names. Includes and imports with context,
blocks, applymacro and filters naming another filter or test by a
non-constant read every name; self sites and an included template's top
level keep the whole frame.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AWNL7pMTLZKmURPZ8DQAjj
The resolver's read list grew on every name use of every template, which
cost the load benchmarks up to 8% more instructions. Reads are now kept
only while a macro or call block is being parsed, and each one's are
sorted and deduplicated in place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AWNL7pMTLZKmURPZ8DQAjj

rmorozov commented Nov 3, 2026

Copy link
Copy Markdown
Member Author

@codex review
@codex security review


Generated by Claude Code

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-03T17:33:42.708805Z 263db4b Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 263db4b9d9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/statements.cpp Resolved
Codex review on #758: a capture made for a closure that never escapes
allocated its read list on the heap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AWNL7pMTLZKmURPZ8DQAjj
@rmorozov
rmorozov marked this pull request as ready for review November 3, 2026 18:10
rmorozov pushed a commit that referenced this pull request Nov 3, 2026
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DTQTriEsFrrBgcYNKDUUF4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants