Small filter and lexer parity: ordering over None, select/reject binding, fold by choice, \N{name} (0045, 0295, 0291, 0041)

.github/workflows/parity-expectations.yml

@@ -15,10 +15,13 @@ on:
       - 'src/unicode_printable.h'
       - 'src/unicode_tables.h'
       - 'src/unicode_case.h'
+      - 'src/unicode_names.h'
       - 'src/html_entities.h'
       - 'scripts/gen_unicode_printable.py'
       - 'scripts/gen_unicode_tables.py'
       - 'scripts/gen_unicode_case.py'
+      - 'scripts/gen_unicode_names.py'
+      - 'scripts/unicode_name_aliases.txt'
       - 'scripts/gen_html_entities.py'
       - '.github/workflows/parity-expectations.yml'
   pull_request:
@@ -30,10 +33,13 @@ on:
       - 'src/unicode_printable.h'
       - 'src/unicode_tables.h'
       - 'src/unicode_case.h'
+      - 'src/unicode_names.h'
       - 'src/html_entities.h'
       - 'scripts/gen_unicode_printable.py'
       - 'scripts/gen_unicode_tables.py'
       - 'scripts/gen_unicode_case.py'
+      - 'scripts/gen_unicode_names.py'
+      - 'scripts/unicode_name_aliases.txt'
       - 'scripts/gen_html_entities.py'
       - '.github/workflows/parity-expectations.yml'
   workflow_dispatch:
@@ -70,9 +76,11 @@ jobs:
       run: python scripts/gen_unicode_printable.py --check src/unicode_printable.h
 
     # The other generated tables follow this Python too: regex classes and int()/float()
-    # digits (unicode_tables.h), case mapping (unicode_case.h), html.unescape (html_entities.h)
+    # digits (unicode_tables.h), case mapping (unicode_case.h), html.unescape (html_entities.h),
+    # the names of \N{...} escapes (unicode_names.h)
     - name: Check the generated Unicode and HTML tables
       run: |
         python scripts/gen_unicode_tables.py --check
         python scripts/gen_unicode_case.py --check
         python scripts/gen_html_entities.py --check
+        python scripts/gen_unicode_names.py --check src/unicode_names.h

CMakeLists.txt

@@ -37,6 +37,7 @@ option(JINJA2CPP_BUILD_FUZZERS "Build the libFuzzer targets in fuzz/ (clang only
 option(JINJA2CPP_STRICT_WARNINGS "Enable additional warnings and treat them as errors" ON)
 set(JINJA2CPP_NODEREF_CHECKS "" CACHE STRING "Checks of the handles that link a parse tree: OFF, ON (Seal and cross-template links), FULL (every access); empty picks FULL for Debug, sanitizer and fuzz builds, else ON")
 set_property(CACHE JINJA2CPP_NODEREF_CHECKS PROPERTY STRINGS "" ON FULL OFF)
+option(JINJA2CPP_WITH_UNICODE_NAMES "Decode \\N{name} escapes in string literals with the Unicode name table (about 450 KB of data)" ON)
 option(JINJA2CPP_CHECK_SLOTS "Check every name read from a slot against its lookup by name (always on without NDEBUG)" OFF)
 option(JINJA2CPP_BUILD_SHARED "Build shared linkage version of Jinja2Cpp" OFF)
 option(JINJA2CPP_PIC "Control -fPIC option for library build" OFF)
@@ -191,6 +192,7 @@ set(JINJA2CPP_SOURCES
     src/tester_factories.cpp
     src/testers.cpp
     src/undefined.cpp
+    src/unicode_names.cpp
     src/value.cpp
     src/value_methods.cpp
 )
@@ -321,6 +323,9 @@ target_compile_definitions(
 if (JINJA2CPP_CHECK_SLOTS)
     target_compile_definitions(${LIB_TARGET_NAME} PUBLIC JINJA2CPP_CHECK_SLOTS)
 endif ()
+if (NOT JINJA2CPP_WITH_UNICODE_NAMES)
+    set_source_files_properties(src/unicode_names.cpp PROPERTIES COMPILE_DEFINITIONS JINJA2CPP_NO_UNICODE_NAMES)
+endif ()
 # The level is part of the node layout's checks, so every target that includes src/ takes it
 # from the library; installed consumers never include src/ and never see it
 string(TOUPPER "${JINJA2CPP_NODEREF_CHECKS}" _noderef_checks)
@@ -447,6 +452,9 @@ if (JINJA2CPP_BUILD_TESTS)
         nlohmann_json::nlohmann_json ${LIB_TARGET_NAME} ${EXTRA_TEST_LIBS} ${JINJA2CPP_PRIVATE_LIBS})
     # The Boost big-integer adapter is checked in every build; the built-in one only in boost builds
     target_link_libraries(jinja2cpp_tests jinja2cpp_boost_multiprecision)
+    if (NOT JINJA2CPP_WITH_UNICODE_NAMES)
+        target_compile_definitions(jinja2cpp_tests PRIVATE JINJA2CPP_NO_UNICODE_NAMES)
+    endif ()
     if ("${JINJA2CPP_WITH_BIGINT}" STREQUAL "boost")
         target_compile_definitions(jinja2cpp_tests PRIVATE JINJA2CPP_WITH_BIGINT_BOOST)
     endif ()

README.md

@@ -286,6 +286,7 @@ You can define (via -D command-line CMake option) the following build flags:
     -  `external` In this mode all dependencies should be provided externally. Paths to `boost`, `nonstd-*` libs, etc. should be specified via standard CMake variables (like `CMAKE_PREFIX_PATH` or libname_DIR)
     -  `conan-build` Special mode for building Jinja2C++ via conan recipe.
 -  **JINJA2CPP_WITH_BIGINT** (default `none`) - the big-integer backend every `TemplateEnv` starts with. With `none`, templates print, compare and convert integers of any size, and arithmetic whose result leaves `int64_t` fails with `IntegerOutOfRange`. With `boost`, that arithmetic is exact, as in Python, through Boost.Multiprecision's `cpp_int` (header-only, compiled into the library; consumers need nothing). The installed package reports the choice as `jinja2cpp_BIGINT` (CMake) and `bigint` (pkg-config).
+-  **JINJA2CPP_WITH_UNICODE_NAMES** (default TRUE) - decode `\N{name}` escapes in template string literals, as Python does, with a table of the Unicode 15.0 character names and aliases (about 450 KB of read-only data). With FALSE the table is left out and a `\N{...}` escape is a parse error.
 
 ### Big integers
 Python integers have no size limit. Jinja2C++ computes in `int64_t` and calls a big-integer backend only when a result leaves that range. A backend is a table of plain functions (`jinja2cpp/big_int_backend.h`), so one built separately plugs into a prebuilt library:

docs/tasks/0041-string-literal-escapes.md

@@ -1,9 +1,9 @@
 ---
-status: open
+status: done
 priority: low
 area: parity
 depends: [0001]
-touches: [src/helpers.h#CompileEscapes]
+touches: [src/helpers.h#CompileEscapes, src/unicode_names.cpp, scripts/gen_unicode_names.py]
 shares: [test/parity/divergences/]
 ---
 # String literal escape sequences
@@ -34,3 +34,12 @@ this task.
 
 **Remaining.** `\N{name}` is rejected as malformed (case `literals.string_escape_named`,
 kind `rejects`) until a compact Unicode name table exists; the task closes with it.
+
+**Done.** `\N{name}` looks the name up in `src/unicode_names.h`, generated by
+`scripts/gen_unicode_names.py` from the oracle Python's Unicode database (checked in CI by
+parity-expectations.yml) and `scripts/unicode_name_aliases.txt` (NameAliases.txt, which
+Python cannot list). It matches CPython's lookup: stored names and aliases in any ASCII case,
+the computed CJK unified and Hangul names only as spelled. All 430k spellings the generator
+knows (names, lower and title case, aliases, CJK variants) were checked against Python. The
+table is about 450 KB of read-only data; `-DJINJA2CPP_WITH_UNICODE_NAMES=OFF` leaves it out,
+and every `\N{}` is then a parse error.

docs/tasks/0045-ordering-none-and-undefined.md

@@ -1,5 +1,5 @@
 ---
-status: open
+status: done
 priority: low
 area: parity
 depends: [0034]
@@ -30,3 +30,8 @@ which lets an empty operand through; the corpus has no case for them yet, so add
 before closing this task.
 
 **Done when.** No line of `test/parity/divergences/` names task 0045, and `ctest -R parity` passes.
+
+**Done.** `CompareForOrder` no longer lets `None` or an undefined operand through: `None`
+is unordered (TypeError) and an undefined key raises UndefinedError under every Undefined
+class, so `min`, `max`, `dictsort` and `groupby` match Python. Cases in `errors.py`
+(`min_with_none` ... `groupby_undefined_key`) and `filters.min_max_single_none`.

docs/tasks/0291-fold-chosen-by-foldable-test.md

@@ -1,5 +1,5 @@
 ---
-status: open
+status: done
 priority: low
 area: parity
 touches: [src/expression_evaluator.cpp#IfExpression, src/expression_evaluator.cpp#BinaryExpression]
@@ -30,3 +30,10 @@ Errors while evaluating mean "not foldable", as in Jinja2.
 
 **Done when.** `autoescape.env_on_finalize_folded_choice` matches and its line is gone from
 `test/parity/divergences/autoescape.txt`.
+
+**Done.** `Expression::PrepareFold` (one walk at Load) says whether a value folds always or
+only by the side a foldable test or `and`/`or` operand picks, and records it per operand;
+`IsFoldableAt` evaluates those tests at render, where they have the value Jinja2 folds.
+`FinalizedExpressionRenderer` and the volatile `StaticOutputRenderer` use it. Cases:
+`autoescape.env_on_finalize_folded_choice*`, `autoescape.volatile_folded_choice`. A choice
+inside a `~` chain or another composite is still run-time: task 0307.

docs/tasks/0295-selectattr-extra-keyword.md

@@ -1,5 +1,5 @@
 ---
-status: open
+status: done
 priority: low
 area: parity
 depends: []
@@ -22,3 +22,10 @@ cases for `selectattr` and `rejectattr` with no test plus an extra keyword, and
 plus a keyword the test does not accept.
 
 **Done when** those cases match Python in narrow and wide renders.
+
+**Done.** `Tester` binds as `select_or_reject` does: the attribute and the test name only by
+position, every other argument (keywords included) to the test, and none at all without a
+test. A missing attribute fails at the first item ("Missing parameter for attribute name"),
+so `selectattr(attribute='a')`, a Jinja2C++-only spelling, is now an error as in Python; the
+two unit rows that used it pass the attribute by position. Cases: `filters.select*_extra_*`,
+`filters.select_lazy_errors_on_empty` and four in `errors.py`.

docs/tasks/0307-fold-by-choice-nested.md

@@ -0,0 +1,27 @@
+---
+status: open
+priority: low
+area: parity
+depends: [0291]
+touches: [src/expression_evaluator.cpp#ConcatExpression, src/expression_parser.cpp#ParseStringConcat]
+---
+# Fold a choice that sits inside another foldable expression
+
+Found while fixing 0291. Jinja2 folds `value if test else alt` and `and`/`or` by the side a
+foldable test picks, at any depth: `('<' if 1 > 0 else x) ~ ''` folds, because `Concat.as_const`
+folds each operand. 0291 decides that at render time (`Expression::PrepareFold` and
+`IsFoldableAt`) only for the conditional, `and`/`or` and the other binary operators, nested in
+each other. A `~` chain, a unary operator, a subscript, a slice, a comparison, an `is` test, a
+tuple, list or dict literal or a filter whose operand folds only by such a choice is treated as
+run-time, so it is escaped by the runtime setting in volatile code and after finalize. The gap
+only ever escapes more. `~` is the hard one: the parser already decides at Load whether a
+chain is folded (`ConcatExpression::SetFolded`, which drops Markup with `str()`), and a fold by
+choice makes that a render-time decision too.
+
+Case: `autoescape.volatile_folded_choice_nested` (Python `<&lt;-1`, Jinja2C++ `&lt;&lt;-1`).
+
+**Proposal.** Give the composite nodes `PrepareFoldByChoice`/`IsFoldableAt` overrides that
+record and check their operands the way `BinaryExpression` does, and make the concat chain's
+Markup drop follow `IsFoldableAt` when its operands fold by choice.
+
+**Done when.** No line of `test/parity/divergences/` names task 0307, and `ctest -R parity` passes.

docs/tasks/0308-divisibleby-non-number.md

@@ -0,0 +1,20 @@
+---
+status: open
+priority: low
+area: parity
+depends: []
+touches: [src/testers.cpp]
+---
+# `divisibleby` on a value that is not a number
+
+Found by the 0045/0291 verifier. Jinja2's `test_divisibleby(value, num)` is `value % num == 0`,
+so it inherits Python's `%`: `'a' is divisibleby 2` raises `TypeError` (str formatting with
+no format spec), `'%d' is divisibleby 2` formats to `'2'` and is False, and a list or None
+raises. Jinja2C++ answers False for every non-number.
+
+Case: `tests.divisibleby_string`.
+
+**Proposal.** Compute the test with the evaluator's `%` (`BinaryExpression::DivRemainder`,
+which already implements str formatting and the TypeErrors) and compare the result with 0.
+
+**Done when.** No line of `test/parity/divergences/` names task 0308, and `ctest -R parity` passes.