job_runs: make a triggered run idempotent across deploy retries - #5882
Open
radakam wants to merge 1 commit into
Open
job_runs: make a triggered run idempotent across deploy retries#5882radakam wants to merge 1 commit into
radakam wants to merge 1 commit into
Conversation
Collaborator
Integration test reportCommit: 8e63c56
10 interesting tests: 5 RECOVERED, 3 SKIP, 2 flaky
Top 6 slowest tests (at least 2 minutes):
|
radakam
force-pushed
the
job-runs-idempotency-wait-for-completion
branch
from
July 13, 2026 09:13
acf313b to
3a5ea16
Compare
radakam
force-pushed
the
job-runs-idempotency-wait-for-completion
branch
from
July 14, 2026 10:07
51dca78 to
66ca15a
Compare
radakam
marked this pull request as ready for review
July 14, 2026 11:52
Contributor
Approval status: pending
|
radakam
changed the title
radakam
added a commit
that referenced
this pull request
Jul 28, 2026
## Changes The fake workspace reported every run as `TERMINATED SUCCESS`, overwriting the `FAILED` state it had just recorded for a task it executed locally. A run now reports the terminal state its tasks add up to. Tasks whose code the fake workspace does not have are left successful. An immutable deployment, for example, uploads the bundle as a snapshot zip that the fake workspace never unpacks, so there is nothing to execute; that gap is in the fake workspace, not in the job under test. With that, `job_runs/failed_run` drops the `runs/get` and `runs/get-output` stubs from its `test.toml` and fails for real, on a task that exits non-zero. ## Why Follow-up to #5882, where a stub stood in for a failing run because the fake workspace could not produce one. Splitting it out keeps the behaviour change for existing tests out of that PR. The stub could not know the test server's address, so it could not report a run page URL; the golden now covers the run URL and the progress lines too. ## Tests `libs/testserver` unit tests for the rollup, for completing the tasks that jobs/runs/submit leaves running, and for a task whose code is missing not failing its run. `job_runs/failed_run` covers the end-to-end path. The rest of the acceptance suite passes unchanged.
denik
reviewed
Jul 28, 2026
| resources.job_runs.my_run: Run URL: [DATABRICKS_URL]/jobs/[NUMID]/runs/[NUMID]?o=[NUMID] | ||
| resources.job_runs.my_run: [TIMESTAMP] "my-job" RUNNING | ||
| resources.job_runs.my_run: [TIMESTAMP] "my-job" TERMINATED FAILED task main failed | ||
| Error: cannot create resources.job_runs.my_run: job run [NUMID] did not succeed: FAILED: task main failed |
Contributor
There was a problem hiding this comment.
if possible, use replace_ids.py or add_repl.py to record replacements for specific IDs.
denik
reviewed
Jul 28, 2026
|
|
||
| STATE=.databricks/bundle/default/resources.json | ||
| run_id() { | ||
| jq -r '.state["resources.job_runs.my_run"].__id__' "$STATE" |
radakam
force-pushed
the
job-runs-idempotency-wait-for-completion
branch
from
July 28, 2026 22:08
d2a1cc6 to
2d11c4d
Compare
radakam
changed the title
DoCreate derives idempotency_token as a SHA-256 of the run config plus a create identity (workspace root path, resource key, and the prior run id when re-creating a vanished run), so a deploy that fires run-now and then crashes before recording the id rejoins that same run on retry instead of starting a duplicate. The token is computed by the CLI and rejected if set in bundle configuration; the new bundle-only rerun_token field folds into it, so bumping it re-runs a configuration that already ran. DoDelete becomes a noop: the Jobs API keeps an idempotency_token reserved once the run it triggered is deleted, so deleting the run would tombstone the token and break re-running the same configuration. That drops the cancel-before-delete path this branch added earlier: with no delete there is nothing to make room for, and cancelling would settle the run on CANCELED, which no later deploy that dedupes onto it could complete. A run whose wait was interrupted keeps going, and the error that ended the wait links its run page.
radakam
force-pushed
the
job-runs-idempotency-wait-for-completion
branch
from
July 30, 2026 13:19
26d8bc8 to
8e63c56
Compare
radakam
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
DoCreatesends anidempotency_tokenderived from the run config plus a newCreateIdentityframework hook, so a retried create rejoins the run it already triggered. The identity is the workspace root path, the resource key, and the id of the run being replaced when re-creating one whose remote copy is gone.The new bundle-only
rerun_tokenfolds into the token to force a fresh run of an unchanged config. Settingidempotency_tokendirectly is rejected at validate time.DoDeletebecomes a noop, which also drops the cancel-before-delete path from #5881.Why
A deploy that fires
run-nowand then crashes before recording the run id cannot tell, on retry, that the job already started, so it triggers a second run. A token the CLI derives rather than the user sets fixes that, as long as every part of it is reconstructible from config and state — hence the identity, and hence rejecting a user-set token.Delete is a noop because the Jobs API keeps a token reserved once the run it triggered is deleted: deleting a run tombstones its token and a later deploy of the same config errors instead of running. Cancelling is no better, since it settles the run on
CANCELED, which no deploy that dedupes onto it could ever complete. A run whose wait was interrupted keeps going, and the error that ended the wait links its run page.Tests
Six direct-engine acceptance tests: retrying after a lost run id, replacing a run that vanished remotely, bumping
rerun_token, two runs of one job in one bundle, flipping config away and back to dedupe onto the original run, and rejecting a user-set token.idempotency_tokenis masked with aDistinctreplacement so a golden with severalrun-nowcalls shows whether they shared a token or rotated.Unit tests cover token derivation, rejoining an already-triggered run, and
DoDeleteissuing no requests.libs/testservermodels the API's deduplication, including a token staying reserved after its run is deleted.