job_runs: wait for run completion, expose output, dedupe deploy retries

[radakam]

Changes

Milestone 2 of the job_runs resource (direct engine only): make a triggered run idempotent across retries, and wait for it so other resources can use its outcome.

• Idempotent create. DoCreate computes idempotency_token as a SHA-256 of the run config plus a create identity (workspace root path, resource key, and the prior run id when re-creating a vanished run). A deploy that fires run-now and then crashes before recording the id rejoins the same run on retry instead of starting a duplicate. The token is never user-settable; a new validate:job_runs mutator rejects it in bundle config.
• Waiting. The deploy polls until the run is terminal, printing progress lines tagged with the resource key (runs deploy concurrently onto one stream). Only SUCCESS completes the deploy; anything else fails it with the failed task, the message that task reported, and a link to the run page. Bounded by 24h, matching bundle run. There is deliberately no config field to opt out: ignore_local_changes would make it a silent no-op after the first deploy.
• A reused run is reported. run-now returns an existing run whenever its token was already spent, so a deploy could finish green having started nothing. The wait's first poll settles this without any timestamp comparison: a run that was just started is always still pending, so a terminal state on the first poll means run-now deduplicated onto an earlier deploy's run. DoCreate now names the run it reused and points at rerun_token.
• A failed run no longer wedges the resource. A failed run is never written to state, so the next deploy rejoined it and reported the same failure, even after the job was fixed — the only way out was editing rerun_token in tracked source. DoCreate now folds such a run's id into the token, the same rotation already used for a vanished run, and re-issues run-now, which starts a fresh run. A run that this deploy started and that failed is still reported rather than silently retried. The walk is bounded at 10 attempts, since it costs one run-now per earlier consecutive failure.
• Outputs. run_id, run_name, state, run_type and run_page_url are readable by other resources, e.g. ${resources.job_runs.nightly.state.result_state}. run_page_url is normalized to the path form that resolves for non-admins.
• rerun_token. A bundle-only field folded into the token. Bump it to re-run a configuration that already ran successfully; retrying a run that did not succeed needs no change, since the next deploy starts a fresh one.
• DoDelete is now a noop. A run is immutable history, and the Jobs API keeps an idempotency_token reserved after its run is deleted, so deleting would tombstone the token and break re-runs. Destroy and recreate leave the run in place.

Supporting changes

• New framework hook dresources.CreateIdentity, attached by DeploymentUnit.Create, for resources deriving a stable create-time key.
• runPageURL moved from bundle/run to libs/workspaceurls.JobRunPageURL, now shared by bundle run and job_runs.
• The fake workspace models run-now idempotency tokens, including the tombstone left behind by a deleted run, and rolls task outcomes up into the run-level state the way the Jobs API does (one failed task fails the run), so a failing run can be exercised end to end locally.

Tests

Unit tests for token derivation (stability, scoping, rotation), for each wait outcome (success, failed, skipped, internal error, abandoned wait, polling), and for each create path against a fake that models run-now dedupe: starting a fresh run, resuming a run still going, reusing a run that succeeded, rotating off a run that failed, reporting a run it started itself that failed, and the bound on the rotation walk.

Acceptance tests: basic, redeploy, job_parameters, rerun, idempotent_recreate (retry mid-run with and without a recorded id), recreate_gone (run deleted out of band), token_survives_delete (flip-back reuses the original run and says so), distinct_runs (identical config under two keys), wait_output (downstream resource reads result_state), failed_run (a failing run fails the deploy, then a redeploy rotates the token off it and starts a fresh run), validate.

invariant/configs/job_run.yml.tmpl is excluded from cloud runs: deploying it now waits for a real run, a real workspace reports a condition-task-only run as SKIPPED, and giving the job a task that succeeds would add a multi-minute cluster run to every variant in a suite that asserts plan and state invariants. Still covered locally.

[eng-dev-ecosystem-bot]

Integration test report

Commit: 2d11c4d

Run: 30403499029

	Env	❌​FAIL	🟨​KNOWN	💚​RECOVERED	🙈​SKIP	✅​pass	🙈​skip	Time
❌​	aws linux	3	1	4	3	323	1334	6:24
❌​	aws windows	3	1	4	3	325	1332	8:51
❌​	azure linux	3	1	4	3	323	1333	6:01
❌​	azure windows	3	1	4	3	325	1331	8:40
❌​	gcp linux	3	1	1	4	322	1335	5:58
❌​	gcp windows	3	1	1	4	324	1333	9:51

11 interesting tests: 4 RECOVERED, 3 FAIL, 3 SKIP, 1 KNOWN

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
🟨​	TestAccept	🟨​K	🟨​K	🟨​K	🟨​K	🟨​K	🟨​K
❌​	TestAccept/bundle/deploy/mlops-stacks	❌​F	❌​F	❌​F	❌​F	❌​F	❌​F
❌​	TestAccept/bundle/deploy/mlops-stacks/DATABRICKS_BUNDLE_ENGINE=direct	❌​F	❌​F	❌​F	❌​F	❌​F	❌​F
❌​	TestAccept/bundle/deploy/mlops-stacks/DATABRICKS_BUNDLE_ENGINE=terraform	❌​F	❌​F	❌​F	❌​F	❌​F	❌​F
💚​	TestAccept/bundle/invariant/no_drift	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R
🙈​	TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
🙈​	TestAccept/ssh/connection	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S	🙈​S
💚​	TestFetchRepositoryInfoAPI_FromRepo	💚​R	💚​R	💚​R	💚​R	🙈​S	🙈​S
💚​	TestFetchRepositoryInfoAPI_FromRepo/root	💚​R	💚​R	💚​R	💚​R
💚​	TestFetchRepositoryInfoAPI_FromRepo/subdir	💚​R	💚​R	💚​R	💚​R

[github-actions]

Approval status: pending

/acceptance/bundle/ - needs approval

46 files changed
Suggested: @denik
Also eligible: @pietern, @janniklasrose, @shreyas-goenka, @andrewnester, @lennartkats-db, @anton-107

/bundle/ - needs approval

17 files changed
Suggested: @denik
Also eligible: @pietern, @janniklasrose, @shreyas-goenka, @andrewnester, @lennartkats-db, @anton-107

General files (require maintainer)

7 files changed
Based on git history:

• @denik -- recent work in bundle/direct/dresources/, bundle/direct/, bundle/phases/

_{Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @janniklasrose, @lennartkats-db) can approve all areas.
See OWNERS for ownership rules.}

[denik]

if possible, use replace_ids.py or add_repl.py to record replacements for specific IDs.

[denik]

there is read_id.py