Skip to content

Small filter and lexer parity: ordering over None, select/reject binding, fold by choice, \N{name} (0045, 0295, 0291, 0041) - #646

Merged
rmorozov merged 6 commits into
masterfrom
claude/project-thread-42iq0k
Oct 24, 2026
Merged

rmorozov merged 6 commits into
masterfrom
claude/project-thread-42iq0k

Conversation

@rmorozov

@rmorozov rmorozov commented Oct 24, 2026 •

Copy link
Copy Markdown
Member

Requested by Ruslan · project thread

Parity thread P5 of the 2026-10-24 batch: four small gaps, each with corpus cases, all checked against Jinja2 3.1.6 on Python 3.12.

Problem and resulting behaviour

  • 0045: ordering over None and undefined. CompareForOrder let None and undefined operands through. Now min, max, dictsort (key and value) and groupby raise TypeError for None and UndefinedError for an undefined key under every Undefined class, as Python's < does. A single item still needs no comparison ([none]|min is None).
  • 0295: select/reject argument binding. Tester now binds as select_or_reject: the attribute and the test name only by position, every other argument (keywords included) to the test, and none at all when there is no test. A missing attribute fails lazily at the first item. User-visible: selectattr(attribute='x') was a Jinja2C++-only spelling and is now an error, as in Python. The two unit rows that used it pass the attribute by position. Also, an undefined attribute name now finds nothing, as in Python, while None still means the item itself.
  • 0291: fold by choice. Jinja2 folds a if test else b and and/or by the side a foldable test picks. The new Expression::PrepareFold does one Load walk and records, per operand, whether it folds always or by choice. IsFoldableAt evaluates those tests at render time; foldable tests are constants, so they have the same value Jinja2 folds. FinalizedExpressionRenderer (finalize ordering, 0288) and StaticOutputRenderer (volatile code, 0216) use it. A choice nested in a ~ chain or another composite is filed as 0307, pinned by autoescape.volatile_folded_choice_nested.
  • 0041: \N{name} escapes. These are now decoded from a generated Unicode 15.0 name and alias table: src/unicode_names.h, from scripts/gen_unicode_names.py and scripts/unicode_name_aliases.txt, checked in parity-expectations.yml. Lookup matches CPython's: stored names and aliases in any ASCII case, computed CJK unified and Hangul names only as spelled. The table is about 450 KB of read-only data, with a sorted index so each lookup takes about 1 µs. New CMake option JINJA2CPP_WITH_UNICODE_NAMES (default ON, Ruslan's choice in the thread); with OFF, a \N{} escape is a parse error and the tests skip those expectations. detail::LookupUnicodeName is JINJA2CPP_EXPORT so that tests can decode escapes in shared builds; an added symbol only.
  • Filed 0308, found by the verifier: 'a' is divisibleby 2 answers False where Python raises. Pinned by tests.divisibleby_string.

Validation

  • CI on f039a0b (base 72ad3e6): all 42 checks green. That includes MSVC (3 configurations), AppleClang (4), both sanitizer jobs, the thread job, fuzz, Conan, the parity expectations check and the instruction counts against base. f039a0b only corrects the CMake option's help text (350 → 450 KB) on top of 5182a21, which was also 42/42 green.
  • Local Debug GCC: ctest 5/5, parity 2092 cases with the new lines removed from divergences/{autoescape,literals}.txt. generate.py --check and gen_unicode_names.py --check pass under Python 3.12.
  • All 430k spellings the generator knows (every name in upper, lower and title case, every alias, CJK variants) were compared with Python's unicode-escape codec: 0 mismatches. The verifier separately compared 163k + 59k names, including mutated ones: 0 mismatches.
  • Perf verdict: scripts/preflight.sh --perf passes the instruction gate, and the perf track approved it (job 120893827560). The largest change is Load/strings at +0.24%; every Render benchmark is unchanged.
  • The one extra allocation on Load/chat_mistral and Load/strings (+8/+4 B) is the Tester constructor's leadParams.posParams vector. It holds the attribute and test-name arguments split off for ParseParams, and is built once per select/selectattr call site at Load (selectattr("role", "equalto", "user") and select('ne', 'the') in those templates). It is not on the render path.
  • A JINJA2CPP_WITH_UNICODE_NAMES=OFF Release build has ctest 5/5.
  • A local merge of master a1e4165 (with Call protocol: *args/**kwargs in calls, Python argument order, callables as values (0220, 0294, 0214) #645) was clean and passed ctest 5/5. It was not pushed; the steward stacks it in the train.

Process

  • Roles: main session (explore and implement), verifier (Sonnet) 1 round. It found two blocking items, a missing #include <exception> and test-line formatting; preflight had already caught both before any push. Its non-blocking findings were also fixed: basic_string<uint8_t> would break on libc++ 19, the OFF build failed tests, and selectattr(undefined) was wrong. It also checked ASan+UBSan with clang (5/5, clean), a shared build (links, 5/5), linear PrepareFold on deep chains, and that IsFoldableAt's std::exception catch loses no limit error, since the normal evaluation rethrows.
  • Red CI pushes: 0 of 2.
  • Codex: no result. The review and security review were requested at 15:01; the summary has said "running" ever since, with no findings and no 👍.
  • Train note: Statement parity: self after a store, output after extends, iterating loop, extending includes (0211, 0203, 0219, 0298) #643 also touches src/filters.cpp and src/renderer.h.

🤖 Generated with Claude Code

https://claude.ai/code/session_015FP4qroXpXbNJdUJrvfJVQ

- 0045: min, max, dictsort and groupby reject None (TypeError) and an undefined
  key (UndefinedError under every Undefined class), as Python's `<` does.
- 0295: select/reject/selectattr/rejectattr bind as select_or_reject: the
  attribute and test name only by position, the rest to the test, nothing
  without a test; a missing attribute fails at the first item.
- 0291: a value folds by the side a foldable test or and/or operand picks.
  PrepareFold records it in one Load walk; the finalize and volatile renderers
  decide it at render with IsFoldableAt. Nested choices are filed as 0304.
- 0041: \N{name} escapes through a generated Unicode 15.0 name and alias table
  (scripts/gen_unicode_names.py, checked in CI), behind
  JINJA2CPP_WITH_UNICODE_NAMES (default ON).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015FP4qroXpXbNJdUJrvfJVQ
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015FP4qroXpXbNJdUJrvfJVQ
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015FP4qroXpXbNJdUJrvfJVQ
…ectattr(undefined)

- unicode_names.cpp compares name keys as std::vector<uint8_t>: libc++ 19 drops
  char_traits<unsigned char>, which basic_string<uint8_t> needed.
- With JINJA2CPP_WITH_UNICODE_NAMES=OFF the tests skip the \N{} expectations.
- selectattr/rejectattr look up an undefined attribute name (finding nothing),
  as Python does; None still means the item itself.
- File 0305: divisibleby of a str (found by the verifier), pinned by
  tests.divisibleby_string.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015FP4qroXpXbNJdUJrvfJVQ
…n PRs)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015FP4qroXpXbNJdUJrvfJVQ
@rmorozov rmorozov self-assigned this Oct 24, 2026

Copy link
Copy Markdown
Member Author

@codex review

@codex security review


Generated by Claude Code

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review 🔄 Running since 2026-10-24T15:01:37.021177Z 5182a21 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@rmorozov
rmorozov marked this pull request as ready for review October 24, 2026 18:47
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015FP4qroXpXbNJdUJrvfJVQ
@rmorozov
rmorozov merged commit b38365e into master Oct 24, 2026
42 checks passed
rmorozov added a commit that referenced this pull request Oct 24, 2026
Merge train 33: #646 and #647 (small filter and lexer parity, output and format parity)
rmorozov pushed a commit that referenced this pull request Oct 24, 2026
At the merge steward's request (#646, #647, #649 merged).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UQ5fopVvhaFveMqaxcKYeK
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants