Skip to content

Add template_mismatch_policy for render-time {{field}} mismatches#1958

Open
dmontagu wants to merge 2 commits into
feature/variable-push-validationfrom
feature/variable-template-mismatch-policy
Open

Add template_mismatch_policy for render-time {{field}} mismatches#1958
dmontagu wants to merge 2 commits into
feature/variable-push-validationfrom
feature/variable-template-mismatch-policy

Conversation

@dmontagu
Copy link
Copy Markdown
Contributor

Third in the #1950 series, stacked on #1953. Implements item C: a configurable render-time policy for TemplateVariable.get(inputs) when the resolved template references a {{field}} the inputs don't declare. Closes the asymmetry Alex flagged on templates-and-composition.md:36 (composition warned, render path silently substituted empty).

New knob

template_mismatch_policy: Literal['warn', 'error', 'ignore']:

  • 'warn' (default): emit a RuntimeWarning and render the missing field as the empty string (Handlebars' default behaviour).
  • 'error': raise TemplateInputsMismatchError instead of rendering.
  • 'ignore': render silently, no warning.

Three configuration levels, variable wins

  1. Per-variable via template_var(..., template_mismatch_policy='error'). Variable-level wins, even when relaxing — set this when one specific variable should opt out of (or into) strict enforcement without affecting siblings.
  2. Per-Logfire-instance via VariablesOptions.template_mismatch_policy or LocalVariablesOptions.template_mismatch_policy (the two carry the field independently — they're peers, not subclasses of each other).
  3. Falls back to 'warn' when nothing is set.
# Tighten one variable while the rest of the app stays lenient
strict_prompt = logfire.template_var(
    'prompt',
    type=str,
    default='Hi {{user_name}}',
    inputs_type=Inputs,
    template_mismatch_policy='error',
)

# Or flip the default at the configure() level
logfire.configure(variables=LocalVariablesOptions(config=..., template_mismatch_policy='error'))

Why a dedicated exception type

For 'error', we raise a new TemplateInputsMismatchError(Exception) rather than a HandlebarsError. The SDK's normal composition-failure fallback (Variable._resolve's outer except Exception) catches HandlebarsError and friends to degrade gracefully to the code default — but the 'error' policy is explicitly opt-in loud mode, so we route past that fallback. A dedicated except TemplateInputsMismatchError: raise in _resolve lets the exception escape to the caller.

Tests

Eight new cases in TestTemplateMismatchPolicy:

  • Default policy is 'warn', fires on mismatch, silent when inputs satisfy.
  • Per-variable 'error' raises TemplateInputsMismatchError.
  • Per-variable 'ignore' renders silently.
  • Instance-level 'error' raises.
  • Instance-level 'error' + variable 'ignore' → renders silently (variable-level relaxes).
  • Instance-level 'warn' + variable 'error' → raises (variable-level escalates).
  • Instance-level 'warn' + variable 'ignore' → renders silently.

Series

Next: PR 4 (misrender retention parity — item F), then PR 5 (internal cleanups — items G and H).

Implements item C of #1950: a configurable render-time policy for what
happens when `TemplateVariable.get(inputs)` is called against a template
that references a `{{field}}` not declared in the variable's
`inputs_type`. Closes the asymmetry Alex flagged on
`templates-and-composition.md:36` (composition warned, render path
silently substituted empty).

`template_mismatch_policy: Literal['warn', 'error', 'ignore']`:

- `'warn'` (default): emit a `RuntimeWarning` and render the missing
  field as the empty string (Handlebars' default behaviour).
- `'error'`: raise `TemplateInputsMismatchError` instead of rendering.
- `'ignore'`: render silently, no warning.

Configurable at three levels with explicit precedence:

1. Per-variable via `var()` / `template_var()`'s
   `template_mismatch_policy` argument. **Variable-level wins**, even
   when relaxing — set this when one specific variable should opt out
   of (or into) strict enforcement without affecting siblings.
2. Per-Logfire-instance via `VariablesOptions.template_mismatch_policy`
   or `LocalVariablesOptions.template_mismatch_policy`.
3. Falls back to `'warn'` when nothing is set.

The two options classes carry the field independently (they're peers,
not subclasses of each other), so a `LocalVariablesOptions`-driven
test/dev rig can be strict while production `VariablesOptions` stays
lenient (or vice versa).

`TemplateVariable._check_template_fields` runs every `{{field}}`
reference in the post-composition serialized value through
`pydantic_handlebars.check_template_compatibility` against
`get_template_inputs_schema()`. Any error-severity issue triggers the
policy.

For `'error'`, we raise a new `TemplateInputsMismatchError(Exception)`
rather than a `HandlebarsError`. The SDK's normal composition-failure
fallback (`Variable._resolve`'s outer `except Exception`) catches
HandlebarsError-and-friends to degrade gracefully to the code default;
the `'error'` policy is explicitly opt-in *loud* mode, so we route past
the fallback. A dedicated `except TemplateInputsMismatchError: raise`
in `_resolve` lets it escape to the caller.

`extract_template_strings` (previously `_extract_template_strings` in
`template_validation`) is promoted to public so the runtime check can
share the same JSON-walker the push-time validator uses; the function
is still small and dependency-free.

Eight new cases in `TestTemplateMismatchPolicy`:

- Default policy is `'warn'`, fires on mismatch, silent when inputs
  satisfy.
- Per-variable `'error'` raises `TemplateInputsMismatchError`.
- Per-variable `'ignore'` renders silently.
- Instance-level `'error'` raises.
- Instance-level `'error'` + variable `'ignore'` → renders silently
  (variable-level relaxes).
- Instance-level `'warn'` + variable `'error'` → raises (variable-level
  escalates).
- Instance-level `'warn'` + variable `'ignore'` → renders silently.
Copy link
Copy Markdown
Contributor

@cubic-dev-ai cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

4 issues found across 7 files

Confidence score: 4/5

  • This PR looks safe to merge with minimal risk: all reported findings are documentation mismatches rather than runtime code defects.
  • The most significant issue is in logfire/_internal/main.py, where docs say 'error' mode raises HandlebarsRuntimeError but behavior raises TemplateInputsMismatchError, which could lead callers to catch the wrong exception.
  • Similar exception-type and API-surface doc drift in logfire/variables/variable.py and logfire/_internal/config.py may confuse integration/error-handling expectations, but impact appears limited to guidance text.
  • Pay close attention to logfire/_internal/main.py, logfire/variables/variable.py, logfire/_internal/config.py - align documented exceptions and var() vs template_var() policy support to prevent user confusion.

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread logfire/_internal/main.py Outdated
Comment thread logfire/variables/variable.py Outdated
Comment thread logfire/_internal/config.py Outdated
Comment thread logfire/_internal/config.py Outdated
@alexmojaki
Copy link
Copy Markdown
Collaborator

coverage will likely be resolved by #1957

`'error'` mode raises `TemplateInputsMismatchError` (defined in
`logfire/variables/variable.py`), not `HandlebarsRuntimeError`. Three
spots referenced the wrong exception:

- `_internal/main.py` `template_var` `Raises:` clause
- `variables/variable.py` `TemplateVariable.get` `Raises:` clause
- `_internal/config.py` `TemplateMismatchPolicy` type alias docstring

Also clarify in `TemplateMismatchPolicy` and the field-level
docstrings that plain `var()` doesn't accept the policy — only
`template_var()` does. Plain `Variable` doesn't render `{{...}}`
templates so the policy doesn't apply.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants