Skip to content

Add template_mismatch_policy for render-time {{field}} mismatches#1958

Merged
dmontagu merged 7 commits into
feature/variable-composition-native-handlebarsfrom
feature/variable-template-mismatch-policy
Jun 6, 2026
Merged

Add template_mismatch_policy for render-time {{field}} mismatches#1958
dmontagu merged 7 commits into
feature/variable-composition-native-handlebarsfrom
feature/variable-template-mismatch-policy

Conversation

@dmontagu
Copy link
Copy Markdown
Contributor

Third in the #1950 series, stacked on #1953. Implements item C: a configurable render-time policy for TemplateVariable.get(inputs) when the resolved template references a {{field}} the inputs don't declare. Closes the asymmetry Alex flagged on templates-and-composition.md:36 (composition warned, render path silently substituted empty).

New knob

template_mismatch_policy: Literal['warn', 'error', 'ignore']:

  • 'warn' (default): emit a RuntimeWarning and render the missing field as the empty string (Handlebars' default behaviour).
  • 'error': raise TemplateInputsMismatchError instead of rendering.
  • 'ignore': render silently, no warning.

Three configuration levels, variable wins

  1. Per-variable via template_var(..., template_mismatch_policy='error'). Variable-level wins, even when relaxing — set this when one specific variable should opt out of (or into) strict enforcement without affecting siblings.
  2. Per-Logfire-instance via VariablesOptions.template_mismatch_policy or LocalVariablesOptions.template_mismatch_policy (the two carry the field independently — they're peers, not subclasses of each other).
  3. Falls back to 'warn' when nothing is set.
# Tighten one variable while the rest of the app stays lenient
strict_prompt = logfire.template_var(
    'prompt',
    type=str,
    default='Hi {{user_name}}',
    inputs_type=Inputs,
    template_mismatch_policy='error',
)

# Or flip the default at the configure() level
logfire.configure(variables=LocalVariablesOptions(config=..., template_mismatch_policy='error'))

Why a dedicated exception type

For 'error', we raise a new TemplateInputsMismatchError(Exception) rather than a HandlebarsError. The SDK's normal composition-failure fallback (Variable._resolve's outer except Exception) catches HandlebarsError and friends to degrade gracefully to the code default — but the 'error' policy is explicitly opt-in loud mode, so we route past that fallback. A dedicated except TemplateInputsMismatchError: raise in _resolve lets the exception escape to the caller.

Tests

Eight new cases in TestTemplateMismatchPolicy:

  • Default policy is 'warn', fires on mismatch, silent when inputs satisfy.
  • Per-variable 'error' raises TemplateInputsMismatchError.
  • Per-variable 'ignore' renders silently.
  • Instance-level 'error' raises.
  • Instance-level 'error' + variable 'ignore' → renders silently (variable-level relaxes).
  • Instance-level 'warn' + variable 'error' → raises (variable-level escalates).
  • Instance-level 'warn' + variable 'ignore' → renders silently.

Series

Next: PR 4 (misrender retention parity — item F), then PR 5 (internal cleanups — items G and H).

Implements item C of #1950: a configurable render-time policy for what
happens when `TemplateVariable.get(inputs)` is called against a template
that references a `{{field}}` not declared in the variable's
`inputs_type`. Closes the asymmetry Alex flagged on
`templates-and-composition.md:36` (composition warned, render path
silently substituted empty).

`template_mismatch_policy: Literal['warn', 'error', 'ignore']`:

- `'warn'` (default): emit a `RuntimeWarning` and render the missing
  field as the empty string (Handlebars' default behaviour).
- `'error'`: raise `TemplateInputsMismatchError` instead of rendering.
- `'ignore'`: render silently, no warning.

Configurable at three levels with explicit precedence:

1. Per-variable via `var()` / `template_var()`'s
   `template_mismatch_policy` argument. **Variable-level wins**, even
   when relaxing — set this when one specific variable should opt out
   of (or into) strict enforcement without affecting siblings.
2. Per-Logfire-instance via `VariablesOptions.template_mismatch_policy`
   or `LocalVariablesOptions.template_mismatch_policy`.
3. Falls back to `'warn'` when nothing is set.

The two options classes carry the field independently (they're peers,
not subclasses of each other), so a `LocalVariablesOptions`-driven
test/dev rig can be strict while production `VariablesOptions` stays
lenient (or vice versa).

`TemplateVariable._check_template_fields` runs every `{{field}}`
reference in the post-composition serialized value through
`pydantic_handlebars.check_template_compatibility` against
`get_template_inputs_schema()`. Any error-severity issue triggers the
policy.

For `'error'`, we raise a new `TemplateInputsMismatchError(Exception)`
rather than a `HandlebarsError`. The SDK's normal composition-failure
fallback (`Variable._resolve`'s outer `except Exception`) catches
HandlebarsError-and-friends to degrade gracefully to the code default;
the `'error'` policy is explicitly opt-in *loud* mode, so we route past
the fallback. A dedicated `except TemplateInputsMismatchError: raise`
in `_resolve` lets it escape to the caller.

`extract_template_strings` (previously `_extract_template_strings` in
`template_validation`) is promoted to public so the runtime check can
share the same JSON-walker the push-time validator uses; the function
is still small and dependency-free.

Eight new cases in `TestTemplateMismatchPolicy`:

- Default policy is `'warn'`, fires on mismatch, silent when inputs
  satisfy.
- Per-variable `'error'` raises `TemplateInputsMismatchError`.
- Per-variable `'ignore'` renders silently.
- Instance-level `'error'` raises.
- Instance-level `'error'` + variable `'ignore'` → renders silently
  (variable-level relaxes).
- Instance-level `'warn'` + variable `'error'` → raises (variable-level
  escalates).
- Instance-level `'warn'` + variable `'ignore'` → renders silently.
Copy link
Copy Markdown
Contributor

@cubic-dev-ai cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

4 issues found across 7 files

Confidence score: 4/5

  • This PR looks safe to merge with minimal risk: all reported findings are documentation mismatches rather than runtime code defects.
  • The most significant issue is in logfire/_internal/main.py, where docs say 'error' mode raises HandlebarsRuntimeError but behavior raises TemplateInputsMismatchError, which could lead callers to catch the wrong exception.
  • Similar exception-type and API-surface doc drift in logfire/variables/variable.py and logfire/_internal/config.py may confuse integration/error-handling expectations, but impact appears limited to guidance text.
  • Pay close attention to logfire/_internal/main.py, logfire/variables/variable.py, logfire/_internal/config.py - align documented exceptions and var() vs template_var() policy support to prevent user confusion.

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread logfire/_internal/main.py Outdated
Comment thread logfire/variables/variable.py Outdated
Comment thread logfire/_internal/config.py Outdated
Comment thread logfire/_internal/config.py Outdated
@alexmojaki
Copy link
Copy Markdown
Collaborator

coverage will likely be resolved by #1957

dmontagu added 6 commits May 24, 2026 13:34
`'error'` mode raises `TemplateInputsMismatchError` (defined in
`logfire/variables/variable.py`), not `HandlebarsRuntimeError`. Three
spots referenced the wrong exception:

- `_internal/main.py` `template_var` `Raises:` clause
- `variables/variable.py` `TemplateVariable.get` `Raises:` clause
- `_internal/config.py` `TemplateMismatchPolicy` type alias docstring

Also clarify in `TemplateMismatchPolicy` and the field-level
docstrings that plain `var()` doesn't accept the policy — only
`template_var()` does. Plain `Variable` doesn't render `{{...}}`
templates so the policy doesn't apply.
dmontagu added a commit that referenced this pull request Jun 6, 2026
Adversarial review of the rolled-in push/sync validation and render-time policy
turned up a few issues, now fixed:

- Robustness: the reference-graph build walked decoded JSON recursively, so a
  deeply-nested server/local value raised RecursionError out of push/validate
  (the cycle and template walks were already guarded, the build path wasn't).
  Made _walk_references iterative, which also hardens the resolution path.
- Docs accuracy: the TemplateMismatchPolicy alias docstring (and template_var's
  param docstring) described a runtime-value check; reworded to match the actual
  static "{{field}} not declared in inputs_type" check.
- Display: the multi-hop composition path in template-field-issue output dropped
  an interior closing '@' (rendered '@{mid} -> @{leaf}@'); now each ref is a
  complete @{...}@ token. Fixed in both _format_diff and ValidationReport.format.
- Messaging: template-field issues were duplicated once per template root that
  composed the same shared fragment; dedup across roots in
  _collect_template_field_issues.
- API consistency: exported TemplateMismatchPolicy from logfire.variables and
  logfire.variables.config (it was public-looking but not importable).
- Docs: documented the render-time mismatch policy and added the
  template_field_issues / reference_errors rows to the ValidationReport table.

Regression tests added (deep nesting, multi-hop chain format, cross-root dedup).
Variables suites (564) + logfire-api shim, ruff, and pyright pass.
Base automatically changed from feature/variable-push-validation to feature/variable-composition-native-handlebars June 6, 2026 00:27
@dmontagu dmontagu merged commit ff54fdb into feature/variable-composition-native-handlebars Jun 6, 2026
14 checks passed
@dmontagu dmontagu deleted the feature/variable-template-mismatch-policy branch June 6, 2026 00:27
@dmontagu
Copy link
Copy Markdown
Contributor Author

dmontagu commented Jun 6, 2026

Consolidated into #1954. This PR's render-time template_mismatch_policy ('warn'/'error'/'ignore') + TemplateInputsMismatchError (plumbed via template_var() / VariablesOptions / LocalVariablesOptions) now lives in #1954. Integrated in ff54fdb (plus a fix keeping the 'warn' policy correct under -W error), with docs/export follow-ups in b11c187. GitHub auto-marked this merged since its commits are now contained in #1954's branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants