[rmorozov]

Requested by Ruslan · project thread

Problem

Task 0360, phase 2 (parity batch 2, group F). Phase 1 (#720) added test/chat_templates/: 72 real Hugging Face chat templates, each rendered with 12 conversations by transformers 5.22.1. 140 of the 722 renders differed from transformers, in four ways:

• transformers' tojson;
• the {% generation %} tag;
• the immutable sandbox;
• two lines closed by group E.

The API decision (Ruslan, 2026-11-01) is option 1 from the plan:

• a public entry point in its own header, behind a CMake switch (default ON), with a feature macro;
• the internals always compiled into the core and dormant unless the preset turns them on.

Behaviour

Public part: include/jinja2cpp/chat_template.h declares ConfigureForChatTemplates(TemplateEnv&, ChatTemplateOptions = {}). ChatTemplateOptions::now is the clock for strftime_now. The function:

• sets trimBlocks, lstripBlocks and loopControls, turns autoescape off, and caps limits.maxRange at 100000 (the sandbox's MAX_RANGE);
• adds raise_exception(message), which fails the render with ErrorCode::TemplateRuntimeError and str(message);
• adds strftime_now(format), which formats as CPython on Linux does in the "C" locale:
  • it supports the portable put_time directives, %f, empty %z/%Z and glibc's - flag;
  • any other directive is copied as written, never handed to the C library, because MSVC's C library aborts on directives it does not know;
  • the result counts against maxSequenceSize as it grows;
• switches the environment's internal dialect.

Internal part: detail::Dialect (src/dialect.h). It is kept in LoadSettings the way the big-int backend is, so a template keeps the dialect it was loaded with. The parser reads it from t_parseLimits; render code reads it through RenderContext::GetDialect(). In the Transformers dialect:

• tojson is transformers' variant: json.dumps(x, ensure_ascii=False, indent=None, separators=None, sort_keys=False). Keys stay in insertion order, nothing is HTML-escaped, and the result is a plain string, not Markup.
  • It is a new Serialize::TransformersJsonMode, picked by name at parse time in FindBuiltinFilter. It is also picked at render time for map('tojson').
  • separators unpacks as Python does, from a list, a two-character string or a dict's keys. It stops at the third item.
  • PythonJsonWriter now takes JsonOptions, and Jinja2's tojson is the same writer with its old options.
  • Each separator and indent append is checked against maxSequenceSize.
  • The per-byte check is now a 256-entry table per mode.
• {% generation %}…{% endgeneration %} renders its body in a scope of its own, like {% with %}. As in transformers' call block, break and continue cannot cross it. The assistant-token mask is not offered; that is task 0377.
• List and dict methods that change their receiver (append, pop, update, …) become the sandbox's unsafe attribute: an undefined whose use fails with "access to attribute 'append' of 'list' object is unsafe.".
  • This covers x.m, x['m'], |attr, map(attribute=) and direct calls.
  • It stays off the hot path: the check runs only after a mutating method has been found.

Plain environments are unchanged.

CMake. JINJA2CPP_WITH_CHAT_TEMPLATES (default ON) builds src/chat_template.cpp and installs the header.

• The generated jinja2cpp/features.h, included by config.h, defines:
  • JINJA2CPP_HAS_CHAT_TEMPLATES;
  • JINJA2CPP_CHAT_TEMPLATES_TRANSFORMERS, the transformers version, read from the pin in test/chat_templates/requirements.txt.
• With the switch OFF, nothing else in the core changes.
• One linux-build row (clang-18, C++20, Debug) now builds with chat: 'OFF'.

Instruction gate. When the base's bench driver does not know the transformers setting, the base counts its own bench/cases, in both benchmark.yml and scripts/preflight.sh. Otherwise an older base would render the new hf_* cases in a plain environment and fail.

Docs.

• docs/api-2.0.md: section 3.3 and decision 7.8.
• test/chat_templates/README.md and bench/README.md are updated.
• Task 0360 has a progress note.
• New task 0377: the assistant mask, and whether the sandbox's rejection should get a SecurityError code.

Bench. New cases hf_qwen3, hf_gpt_oss, hf_deepseek_v31 and hf_mistral_small are next to chat_llama:

• They are real templates with Apache-2.0 or MIT model licenses, rendered with the suite's tools conversation.
• Their settings.json has "transformers": true. python_bench.py builds the same environment without depending on transformers, and its output equals the transformers expectations.
• engines_bench skips these cases.

Validation

• test/chat_templates: all 722 renders match transformers byte for byte, narrow and wide. divergences.txt is now empty.
• test/chat_template_preset_test.cpp covers every behaviour above, narrow and wide, including the sequence-cap tests for strftime_now and for separators and indent. Each expectation was checked against transformers 5.22.1.
• Full ctest, Debug (GCC, -Werror): passed. A Debug build with -DJINJA2CPP_WITH_CHAT_TEMPLATES=OFF: built, and its tests passed.
• scripts/preflight.sh --perf at 6682416: all checks PASS (clang-format, clang-tidy 22.1.8, null comparisons, use_count, CodeQL pattern, task files, instruction gate).
• Instruction gate (bench/count.py, threshold 3%):
  • Render/chat_llama −0.27%, Render/chat_qwen −0.69%, Render/chat_mistral −0.14%;
  • every other row within ±0.5%;
  • the hf_* rows are new.
• CI at 6682416: all checks green, including MSVC, AppleClang, sanitizers, the chat=OFF row and the instruction gate.
• Verifier (1 round), report summary:
  • ASan+UBSan RelWithDebInfo run of ChatTemplate*:Preset*:*ojson*:*Parity*:*Statement*: 7584 passed, no reports.
  • Adversarial probes against transformers, narrow and wide:
    • generation tags: 50/50 match;
    • unsafe-method paths: about 120 templates, all match;
    • tojson arguments: 86 templates.
  • It found non-blocking gaps, all fixed, with tests:
    • string separators;
    • non-string arguments to the two globals;
    • glibc's %-d flag;
    • a trailing %;
    • the MSVC risk: an unknown strftime directive now never reaches the C library.
  • Noted, not from this diff: under GCC 13 with -O2, test/expressions_test.cpp:756 trips -Wsign-conversion. CI does not build with GCC 13.

Roles: main session (Opus) explored and implemented; verifier ran as a subagent. Verifier rounds: 1. Red CI pushes: 2.

• e253ae2: MSVC and bindings jobs failed with C1083, features.h not found. #include "features.h" in config.h found glibc's <features.h> on Linux, so local builds hid it. Fixed in 28e1ac1 with "jinja2cpp/features.h".
• 28e1ac1: "instruction counts vs base" failed. The base binary ran the PR's hf_* cases and could not render hf_gpt_oss. Fixed in 8ff6ce0: such a base counts its own cases.

Codex: 5 findings, 5 fixed, 0 declined.

• Round 1 at 8ff6ce0, fixed in d010800:
  • P1: strftime_now sequence cap;
  • P1: string separators unpack stops at the third item;
  • P2: indent is evaluated for strings.
• Round 2 at d010800, fixed in 6682416:
  • P1: separator and indent appends are checked against the cap;
  • P2: dict separators.
• Fixing round 1 also exposed a host string reaching strftime_now as a std::string_view, which was rejected; that is fixed too.
• The @codex security review was requested twice, but its status stayed empty, so no security review result exists. Both allowed requests are used.

Limits

• This is not the rest of apply_chat_template: the host builds messages, tools and bos_token, and handles the reasoning aliases.
• It is not a security sandbox beyond what it lists; untrusted templates still need UntrustedLimits.
• An unsafe method fails with UndefinedError, not a distinct code (task 0377).
• The Windows and macOS strftime results for the portable directives come from those platforms' C libraries in the "C" locale.

---

Generated by Claude Code

🤖 Generated with Claude Code

https://claude.ai/code/session_01AWNL7nyP1j6LYCMdbnWsHk

[rmorozov]

@codex review

@codex security review

The diff touches the parser (the generation tag), filter lookup and the evaluator's method dispatch.

---

Generated by Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-02T13:02:29.542230Z	d010800	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8ff6ce0f8a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[rmorozov]

@codex review

@codex security review

d010800 fixes the three round-1 findings: the strftime_now sequence cap, the separators string unpack and the indent evaluation. It also accepts host strings that arrive as views. The security review did not run on the first request.

---

Generated by Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d010800e66

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".