Skip to content

fix: attempt remaining pipeline cleanup after component errors - #13520

Closed
DiadkoShmek wants to merge 2 commits into
deepset-ai:mainfrom
DiadkoShmek:lezo2-fix-pipeline-cleanup-drain
Closed

DiadkoShmek wants to merge 2 commits into
deepset-ai:mainfrom
DiadkoShmek:lezo2-fix-pipeline-cleanup-drain

Conversation

@DiadkoShmek

Copy link
Copy Markdown

Related Issues

No linked issue. Reproduced through public Pipeline resource-lifecycle calls with actual local file resources; the related tool-batch cancellation work in #12796 concerns a different owner.

Proposed Changes:

If one component's close() raises, pipeline cleanup currently exits before visiting later components. Repeating cleanup can encounter the same error and leave those later resources open again.

Attempt all remaining cleanup hooks after an ordinary Exception, then re-raise the first exception object. Preserve component order, native async cleanup preference and sync fallback. Cancellation and other BaseException subclasses still interrupt immediately; the pipeline cannot repair resources inside a component whose own cleanup fails. The pipeline guide and Reno release note describe that behavior.

How did you test it?

  • The original owner fails 9 new lifecycle cases while 12 controls pass. The sync/async/lifecycle selection passes all 76 tests after the change, with two existing Document-mutation warnings.
  • A public Pipeline consumer reproduces a genuine buffered-file EFBIG cleanup error; later files remain open on base and close after the patch, while the same error propagates.
  • A separate agent review exercises 42 fresh public cases, including genuine EBADF, exception identity/cause/traceback, repeated calls, mixed async/sync cleanup and real task cancellation. The base misses later cleanup in 12 cases; the patch passes.
  • Scoped Hatch formatting, selected-file pre-commit hooks and mypy for both Python files pass. The branch hook is checked on the publication branch. No full test environment, repository suite, documentation-site build or live integration is claimed.

The main test command in the focused Hatch development environment was:

hatch -e default run pytest test/core/pipeline/test_pipeline.py test/core/pipeline/test_async_pipeline.py test/core/pipeline/test_pipeline_lifecycle.py -q -p no:cacheprovider

Notes for the reviewer

The existing exception type remains catchable, but arrives after later ordinary cleanup attempts. Secondary ordinary errors do not replace it; an intervening cancellation still propagates immediately.

This PR was fully generated with OpenAI Codex. Codex reviewed the changes, ran the relevant local tests and obtained a separate agent review; no human code review is claimed.

Checklist

  • Read the contributor guidelines and code of conduct.
  • Updated a related issue — no linked issue.
  • Added unit tests and updated the docstrings.
  • Used a conventional commit title.
  • Updated the current pipeline documentation.
  • Added a generated Reno release note.
  • Ran applicable selected-file pre-commit hooks.

@DiadkoShmek
DiadkoShmek requested a review from a team as a code owner November 2, 2026 01:13
@DiadkoShmek
DiadkoShmek requested review from davidsbatista and removed request for a team November 2, 2026 01:13
@vercel

vercel Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

@DiadkoShmek is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

CLAassistant commented Nov 2, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@HaystackBot

Copy link
Copy Markdown
Contributor

Hi @DiadkoShmek, thanks a lot for your contribution! 🙏

We noticed that the Contributor License Agreement (CLA) check (license/cla) hasn't passed yet, so we've temporarily moved this PR to draft and paused the review assignment.

To get your PR reviewed, please sign the CLA via the link in the license/cla check below (or in the CLA bot comment). As soon as the check turns green, this PR will automatically be marked ready for review again and a reviewer will be re-assigned.

@HaystackBot
HaystackBot removed the request for review from davidsbatista November 2, 2026 02:25
@HaystackBot HaystackBot added the cla-pending PR is in draft until the contributor signs the CLA label Nov 2, 2026
@HaystackBot
HaystackBot marked this pull request as draft November 2, 2026 02:25
@HaystackBot
HaystackBot marked this pull request as ready for review November 2, 2026 20:50
@HaystackBot HaystackBot removed the cla-pending PR is in draft until the contributor signs the CLA label Nov 2, 2026
@HaystackBot

Copy link
Copy Markdown
Contributor

Thanks for signing the CLA, @DiadkoShmek! 🎉 This PR is now ready for review again and the reviewer has been re-assigned.

@DiadkoShmek

DiadkoShmek commented Nov 2, 2026 •

Copy link
Copy Markdown
Author

@davidsbatista The CLA now passes and the bot has restored the review request. The fork’s 16 workflows still await maintainer approval, and Vercel is waiting for deepset-team authorization before any preview deployment. Could a team member approve the test workflows and authorize the preview when reviewing? No code has changed since f3c958b.

@davidsbatista

Copy link
Copy Markdown
Contributor

Thanks for the PR, @DiadkoShmek!

We're going to close this one for now:

  • There's no reported issue behind it, and close() / close_async() just shipped. We'd rather see how they behave in real use before deciding on how the cleanup should handle component errors, there many approaches to it, and this one doesn't cover all the cases.
  • A component whose close() raises should be fixed in that component. We don't want the pipeline to hide later errors.
  • The current approach drops every cleanup error after the first one without logging it, which would make failures harder to debug.

If you run into this in practice, please open an issue with the scenario and we'll revisit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants