Record task metadata the GCS has not published yet - #183
Merged
Conversation
The actor test I added was unreliable on CI. It failed on main twice with an empty dataframe while every other test passed, including the plain-function equivalent, and there were no scheduling errors or actor tracebacks in the logs. My earlier explanation, that the head-node resource budget was exhausted, was wrong: it did not fix this. The defect was a type declaration, so assert that directly. Building a polars frame from the shipped schema with a real Ray hex id reproduces it exactly — Float32 raises TypeError, Utf8 round-trips — with no Ray, in 0.09s rather than a 120s poll. Reverting the schema turns 4 of the new tests red. Extend the same check to every id column, since task_id, job_id, node_id, worker_id and parent_task_id are all hex strings too. Why actor tasks record no metadata on Linux CI while they do on macOS is still unexplained and is not covered here. Signed-off-by: Tim Paine <3105306+timkpaine@users.noreply.github.com>
The tracker drops task metadata, and that is what has been failing CI. When ray.wait reports an object ready the GCS has not yet published the task's state. Measured on a quiet box, this is not an edge case: all 60 of 60 tasks read back as None at that instant. callback files those tasks in pending_tasks, and only a later callback revisits that list. The processor sends callbacks while it still has unfinished tasks, so whatever lands in the final batch is never looked at again and never reaches the dataframe. Ten tasks that all finish in one wait round produce exactly one callback, so every task can be stranded at once and get_df stays empty forever. That is the empty dataframe CI reported, and it explains why only the two tests that wait on metadata ever failed while the other three passed. Have the processor keep asking until the tracker reports nothing pending. Calling back with no tasks re-resolves the list on its own. This reproduces only under CI timing. On macOS the hop between the two actors is enough for the GCS to catch up, so the suite passes here with and without this change. Signed-off-by: Tim Paine <3105306+timkpaine@users.noreply.github.com>
Contributor
Test Results52 tests 52 ✅ 34s ⏱️ Results for commit a2b4c44. ♻️ This comment has been updated with latest results. |
timkpaine
changed the title
timkpaine
marked this pull request as ready for review
August 22, 2026 01:07
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The tracker drops task metadata. That is what has been failing CI, and it is a product bug, not a flaky test.
I was wrong twice before this
test_construction_and_dataframe— a test that had passed the run before. That killed the actor-specific theory too, and is what finally pointed at the real mechanism.The bug
ray.waitreports an object ready before the GCS has published the task's state. This is not an edge case. Measured on a quiet box, 60 of 60 tasks read back asNoneat that instant:callbackfiles unresolved tasks intoself.pending_tasks. That list is written at task_tracker.py:204 and read only at :191 — insidecallbackitself, which fires only from :56, inside a loop that exits once every ref is done. So whatever lands in the final batch is never revisited and never reaches the dataframe.Ten tasks that all finish in one wait round produce exactly one callback, so every task can be stranded at once and
get_dfstays empty forever. That is the empty dataframe CI kept reporting, and it explains the part of the pattern I could not account for before: only the two tests that wait on metadata ever failed, interchangeably, while the other three always passed.The fix
The processor keeps asking until the tracker reports nothing pending. Calling back with no tasks re-resolves the list on its own, so this needs no new plumbing — just
has_pending_tasks()and a bounded settle loop.On verification
This reproduces only under CI timing. On macOS the hop between the two actors is enough for the GCS to catch up, so the suite passes locally with and without this change — including the new single-task test. I am not claiming a local reproduction.
The evidence is instead: the measured 60/60 lag, the code path showing
pending_taskshas no other reader, and three consecutive green CI runs (32538836488, 32538946200, 32538949011) where the previous two attempts failed.Also here
The id-type regression test now runs without a Ray cluster. Building a polars frame from the shipped schema with a real hex id reproduces that defect exactly —
Float32raises,Utf8round-trips — in 0.09s instead of a 120s poll. Revertingschema.pyturns 4 of those tests red, so it is a real regression test. Extended to every id column, sincetask_id,job_id,node_id,worker_idandparent_task_idare hex strings too.52 tests pass locally; lint clean.