feat: cross-run lineage via reserved run attributes#2153
Conversation
🦋 Changeset detectedLatest commit: 0a03200 The changes in this PR will be included in the next version bump. This PR includes changesets to release 21 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
@rchasman is attempting to deploy a commit to the Vercel Labs Team on Vercel. A member of the Team first needs to authorize it. |
|
since now that #2088 has been shipped in v5 too, you can also use attributes to have parent and child runs link to each other by setting the run IDs as you want. We could have a reserved wdyt @rchasman ? |
bb30012 to
e0b62e9
Compare
|
Re-spun onto attributes per your suggestion. Good call.. Going through attributes turned out cleaner than the column: lineage is preserved across lifecycle updates for free (attributes already carry forward), and the same filter generalizes to any attribute, not just lineage. I record both Also fixed the resilient-start path the review bot flagged: attributes now flow through Proven end-to-end with a real daisy-chain; unit + e2e green, existing suites unaffected. Still world-local only and one |
Completes the attributes feature from vercel#2088. Attributes can now be set when a run is created (start options, CreateWorkflowRunRequest, the run_created and resilient run_started event data, and queue RunInput) and queried via an attributes filter on ListWorkflowRunsParams, so related runs can be tagged and listed as a group. Signed-off-by: Roey D. Chasman <rchasman@gmail.com>
Store attributes passed at run creation and honor the attributes filter in runs.list (match runs holding every requested key/value). Because they ride on the attributes mechanism, they are preserved across lifecycle updates for free. Signed-off-by: Roey D. Chasman <rchasman@gmail.com>
Honor the attributes filter via jsonb containment (@>), matching world-local. Without it the param was silently ignored and the query returned every run. Signed-off-by: Roey D. Chasman <rchasman@gmail.com>
Attribute filtering executes server-side in the platform /v2/runs endpoint, which does not accept it yet. Forwarding the filter would be silently ignored and return every run, so throw instead of returning wrong results. Signed-off-by: Roey D. Chasman <rchasman@gmail.com>
start() records reserved $rootRunId and $parentRunId attributes on each run: a
run with no parent is its own root, and a run started from inside another run
inherits the parent's $rootRunId, so a daisy chain of any depth groups under one
root. Built on the attributes filter above, so the chain is listable and
cancellable as a unit via list({ attributes: { $rootRunId } }).
Signed-off-by: Roey D. Chasman <rchasman@gmail.com>
Prove the full path: a real daisy-chain through the runtime and queue groups under one $rootRunId, listable via the attributes filter, with an unrelated run excluded. The /runs route translates ?rootRunId= into the attribute filter. Signed-off-by: Roey D. Chasman <rchasman@gmail.com>
Signed-off-by: Roey D. Chasman <rchasman@gmail.com>
e0b62e9 to
0a03200
Compare
|
@pranaygp here is where the PR landed and why, since the purpose evolved into two clear layers: Layer 1: make attributes filterable. #2088 lets you set attributes mid-run, but not at creation, and there is no way to query by them. This adds both: set attributes when you start a run, and filter Layer 2: reserved lineage params. The original PR did this with a dedicated I ordered the commits so layer 1 lands first and layer 2 is the last two. If you would rather take the reserved lineage params slower, dropping those two leaves a clean attributes-filter PR. Either way the filter is the foundation and the lineage is a reserved convention on top of it. (world-local and world-postgres implement the filter; world-vercel throws until the hosted runs API accepts attributes at creation and on list, noted in the description.) |
Problem
When one run starts another (a recurring schedule that re-arms itself, a fan-out, any daisy chain), the runs have no relationship to each other, so you cannot list or cancel a chain as a unit. I hit this building a scheduler on Workflow.
Approach
Two layers on the attributes mechanism from #2088:
list({ attributes })returns runs holding every pair. Tag related runs, list them as a group.start()records reserved$rootRunId/$parentRunId, so a daisy chain of any depth groups under one root, listable and cancellable vialist({ attributes: { $rootRunId } }).Scope (commits are ordered so this is easy)
The first four commits are layer 1: attributes-at-creation plus the
listfilter, across core/world/world-local/world-postgres/world-vercel. They are self-contained, complete #2088, and are useful on their own. The last two commits add thestart()lineage layer and its e2e. If you would rather take the lineage auto-recording slower, dropping those two leaves a clean attributes-filter PR. Happy either way.Caveat: the Vercel world needs the runs API to adopt attributes
world-local and world-postgres implement the whole feature. The Vercel world talks to the hosted runs API, where #2088 shipped only a write-only attributes endpoint (set yes, filter no). For this to work end to end on the hosted world, the runs API needs two additions:
start()-set attributes (including$rootRunId) are stored, and/v2/runs).Both are server-side and outside this repo. Until they land, world-vercel throws on an attributes filter rather than silently returning every run, and attributes set at creation take effect only once the create path stores them. The world-testing
/runsroute is a reference for the filter.Testing
core, world-local, and world-vercel suites green; an end-to-end daisy chain groups under one
$rootRunIdthrough the real runtime and queue.