Skip to content

fix: serialize pydantic models in JSON mode in pipeline snapshots - #13789

Open
emreocell wants to merge 1 commit into
deepset-ai:mainfrom
emreocell:fix/13490-pydantic-datetime-snapshot
Open

emreocell wants to merge 1 commit into
deepset-ai:mainfrom
emreocell:fix/13490-pydantic-datetime-snapshot

Conversation

@emreocell

@emreocell emreocell commented Nov 17, 2026 •

Copy link
Copy Markdown

Related Issues

Proposed Changes:

_serialize_value_with_schema stored pydantic models with model_dump(), which keeps Python objects such as datetime, date, UUID and Decimal. With HAYSTACK_PIPELINE_SNAPSHOT_SAVE_ENABLED set, json.dump in _save_pipeline_snapshot then failed, so a breakpoint raised TypeError: Object of type datetime is not JSON serializable instead of BreakpointException and left a half-written snapshot file.

The fix uses model_dump(mode="json"). Deserialization already goes through model_validate, which parses the JSON-mode values back into the original field types. The same helper is used for the Agent State, which benefits too. A release note is included.

This only covers the pydantic-model path from the issue. It does not touch typed values inside Document.meta (that is the scope of #13538).

How did you test it?

  • New unit test test_serialize_pydantic_model_with_non_json_fields_is_json_safe (datetime/date/UUID/Decimal JSON round trip) and new end-to-end test test_breakpoint_with_pydantic_model_with_datetime_field_saves_resumable_snapshot (the breakpoint writes a snapshot file, load_pipeline_snapshot + resume returns the same model). Both fail without the fix (TypeError: Object of type datetime is not JSON serializable) and pass with it. The script from the issue also went from TypeError to BreakpointException.
  • Full hatch run test:unit: 7568 passed, 49 skipped, 0 failed (before a rebase onto the latest main; after the rebase I re-ran the two touched test files: 102 passed). hatch run test:types (mypy): no issues.
  • ruff check and ruff format run directly on the changed files (hatch run fmt does not work on Windows because of the ; separator in the script). Integration/slow tests and pre-commit hooks were not run; tested with Python 3.13 on Windows 11.

Notes for the reviewer

  • fix: preserve typed Document metadata in pipeline snapshots #13538 is open and touches the same area (typed values in Document.meta); as far as I can tell the two changes are independent. Happy to adjust if you prefer one consolidated change.
  • JSON mode turns bytes into str, set into list and enums into their value, all of which model_validate converts back. For field types pydantic cannot represent as JSON, model_dump(mode="json") now raises PydanticSerializationError during serialization; before, such values reached json.dump and failed there anyway, and _serialize_with_field_fallback handles the failing field.
  • An AI coding assistant (Claude, via Claude Code) helped write this change and also reviewed it and ran the tests and checks listed above; I am the submitter and take responsibility for it. I have signed the CLA.

Checklist

  • I have read the contributors guidelines and the code of conduct.
  • I have updated the related issue with new insights and changes.
  • I have added unit tests and updated the docstrings.
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
  • I have documented my code.
  • I have added a release note file, following the contributors guidelines.
  • I have not run the pre-commit hooks (ruff was run directly on the changed files instead).

`_serialize_value_with_schema` stored pydantic models with `model_dump()`, which keeps
`datetime`, `date`, `UUID` and `Decimal` values. With snapshot saving enabled, `json.dump`
then failed and a breakpoint raised `TypeError` instead of `BreakpointException`, leaving a
half-written snapshot file. Use `model_dump(mode="json")`; deserialization already goes
through `model_validate`, which parses the values back.

Fixes deepset-ai#13490
@emreocell
emreocell requested a review from a team as a code owner November 17, 2026 14:16
@emreocell
emreocell requested review from bogdankostic and a balanced review from Copilot and removed request for a team November 17, 2026 14:16
@vercel

vercel Bot commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

@emreocell is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@github-actions github-actions Bot added topic:tests P2 Medium priority, add to the next sprint if no P1 available labels Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Medium priority, add to the next sprint if no P1 available topic:tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Breakpoint snapshot of a pydantic model with a datetime field raises TypeError

2 participants