Release driver-side references in PublisherModel before publishing - #261
Merged
Conversation
PublisherModel.__call__ held live references to the source result (data) and the extracted pub_data for the entire duration of the publisher's execution. When a publisher writes a large frame to a sink, it can build a sole-reference ownership carrier internally so the frame is freed before the terminal write drains. The outer method retaining data/pub_data defeated that transfer, leaving peak memory at roughly twice the frame size during writes for return_data=False sink-write publishers. When return_data=False, delete the driver-side references before invoking the publisher so it can own the sole reference and free the frame during the write. When return_data=True the references are retained, since the source result must be returned. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Pascal Tomecek <pascal.tomecek@cubistsystematic.com>
Contributor
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #261 +/- ##
==========================================
+ Coverage 93.48% 93.51% +0.03%
==========================================
Files 176 176
Lines 20327 20361 +34
Branches 1350 1350
==========================================
+ Hits 19002 19040 +38
+ Misses 1052 1050 -2
+ Partials 273 271 -2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PublisherModel.__call__held live references to the source result (data) and the extractedpub_datafor the entire duration of the publisher's execution. When a publisher writes a large frame to a sink, it can build a sole-reference ownership carrier internally so the frame is freed before the terminal write drains — a peak-memory contract. But the outer method retainingdata/pub_datadefeated that ownership transfer from the outside, so the frame's refcount never dropped to zero duringpublisher(). Peak memory was roughly twice the frame size during the terminal write for anyreturn_data=Falsesink-write publisher.Fix
When
return_data=False, the driver-side references are deleted before invoking the publisher, so the publisher can own the sole reference and free the frame during the write:When
return_data=True, references are retained because the source result must be returned. Behavior is otherwise unchanged.Testing
Added regression tests using
weakref+gcthat verify the frame is collected duringpublisher()whenreturn_data=False, and retained whenreturn_data=True. The newreturn_data=Falsetest was confirmed to fail on the pre-fix source. Fullccflow/tests/modelssuite passes (81 tests).