v1.12: Linux run never returns after the turn completes; job dies on timeout with the output file already written

[wjarka]

Summary

Since v1.12, a sufficiently heavy Run Codex review step never completes on Linux. (See the control run in the comments: light runs on v1.12 finish fine, and v1.11 handles the same heavy workload cleanly.) Codex finishes the turn, writes its output file, prints its final message and token count — and then the step sits idle until the job's timeout-minutes kills it. The run is billed and discarded even though the result was already on disk.

The same workflow on v1.11 (52fe01ec70a42f454c9d2ebd47598f9fd6893d56) completes cleanly in ~87s.

Timeline from one affected run

ubuntu-latest, GitHub-hosted, runner 2.336.0, openai/codex-action@v1 resolving to 86365089eb2b84e0a8fb0717b304f8bdcb13b20e (v1.12):

Time (UTC)	Event
06:40:48	job starts
06:40:56	Run Codex review starts
06:47:45	last exec completes
06:47:51	final assistant message, then tokens used / 270,199
06:47:51 → 06:55:48	nothing at all
06:55:48	job timeout-minutes: 15 deadline
07:00:49	force-killed, job cancelled

The step produced its full result in 6m55s, then idled for 7m57s.

The tell is the missing end marker. On v1.11 the log reads:

<token count>
##[end-action id=__openai_codex-action.run_codex;outcome=success;conclusion=success;duration_ms=86604]

On v1.12 the token count is the last line the step ever emits. No end-action, no outcome.

Configuration

      - uses: openai/codex-action@v1
        with:
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
          prompt-file: <path>
          output-file: <path>
          working-directory: <path>
          model: gpt-5.6-luna
          effort: xhigh
          sandbox: read-only
          output-schema-file: <path>

Raising timeout-minutes does not help — the step idles longer. Lowering effort does not help either; the turn was never the slow part.

Why this loses completed work

With an explicit output-file, runCodexExec passes --output-last-message <file> straight to codex exec, so the CLI writes the result itself as soon as the turn ends. In the run above the file existed at 06:47:51.

But because the step never returns, every later step in the job is skipped — including the artifact upload that hands the result to the next job. A finished, valid, schema-conforming result is thrown away. Consumers that treat step success as "a result exists" will silently produce nothing.

Suspected area

v1.12 is the first release to route Codex through the rewritten privilege-isolation path (src/dropSudo.ts +419/-20, src/runCodexExec.ts +285/-6, new src/linuxCredentials.ts), launching via:

/usr/bin/setpriv --reuid="$uid" --regid="$nobody_gid" --clear-groups \
  --no-new-privs --bounding-set=-all --inh-caps=-all --ambient-caps=-all \
  -- /usr/bin/env ... "$@"
exec "$@"

A plausible mechanism is a descendant process surviving the turn and holding the inherited stdio pipe open, so the wrapper never observes stream close. Supporting detail: even on a clean v1.11 run, the runner's own cleanup reports leftovers —

Cleaning up orphan processes
Terminate orphan process: pid (2460) (bash)
Terminate orphan process: pid (2462) (MainThread)

That run was 87s and spawned few children, so it exited regardless. The affected run executed many subprocesses during the turn.

A second symptom from the same run may share a cause: subprocesses failing to create a temporary directory. Retracted — a v1.11 control run produces the same errors and finishes normally, so that is ordinary sandbox: read-only behaviour and unrelated to this issue. See the comment below.

Possibly related open issues: #103 (output-schema with runAsUser fails to create temporary file), #137 (failed runs leave temporary output directories behind), #108 (proxy hangs).

Two things that would help regardless of the root cause

1. Bound the wait. If the wrapper waited on process exit with a timeout rather than on stdio close, a lingering descendant would cost seconds, not the whole job budget.
2. Don't hide a written result. The output file is complete before the hang. Failing the step with a message naming the written file — instead of never returning — would let callers publish the result they already paid for.

Workaround

Pin to v1.11:

uses: openai/codex-action@52fe01ec70a42f454c9d2ebd47598f9fd6893d56  # v1.11

Note that v1 currently points at v1.12, so anyone on the floating tag picked this up automatically.

[wjarka]

Follow-up with a control run, which both narrows the report and retracts part of it.

The failure needs a heavy run, not just v1.12

I originally had no v1.11 run comparable to the failing one, so "v1.12 hangs" was confounded with "that particular review did a lot of work". I found the missing cell:

	light run	heavy run (many Python subprocesses, ~240–270k tokens)
v1.11 52fe01ec	clean, 87s	clean, 6m21s
v1.12 86365089	clean, 2m14s	hangs after the turn ends

The v1.11 heavy run executed pytest, uv run pytest, python -m unittest, and python -m compileall, spent 237,369 tokens over 48 tool calls, and exited cleanly — including its post step. The v1.12 failure spent 270,199 tokens on the same class of work and never returned.

So v1.12 is not broken across the board. Light runs on v1.12 finish fine, which is presumably why this got through. Something about the rewritten launch path only fails to return once the turn has spawned enough subprocess work.

Caveat: n=1 in the failing cell. I have one hung run, not a reproduction I can re-trigger on demand.

Retracting the tempfile speculation

I suggested the new --regid=<nobody> --clear-groups launch might explain subprocesses failing with

FileNotFoundError: [Errno 2] No usable temporary directory found in
['/tmp', '/var/tmp', '/usr/tmp', '<working-directory>']

That is wrong, and worth striking so nobody chases it. The v1.11 control run produced 32 of the same errors and finished normally. It is ordinary sandbox: read-only behaviour, unrelated to this issue and unchanged by v1.12.

What still stands

The orphan-process theory in the original report is also weaker than I implied: all three clean runs — v1.11 light, v1.11 heavy, v1.12 light — report a leftover process at runner cleanup and exit anyway.

Cleaning up orphan processes
Terminate orphan process: pid (....) (MainThread)

So a lingering child is normal here and is not sufficient on its own. What distinguishes the failing run is the volume of subprocess work, not the mere presence of a leftover process.

The rest of the original report is unaffected: the turn completes, the output file is already written via --output-last-message, no end-action is ever emitted, and the job dies on its timeout with every later step skipped.

[islamJamer]

This happened to me as well.

[rockwotj]

We're hitting this too

[uilton-oliveira]

We are hitting the same thing on v1.12, in the "runner lost communication" variant from the previous comment. We have a before/after that separates two variables currently confounded in this thread: the v1.12 upgrade and the model / reasoning effort.

Setup

A PR-review workflow that fans out into parallel matrix jobs — one codex-action call per slice of the changed files, each slice on its own GitHub-hosted ubuntu-latest runner. A second, independent codex-action call runs later in the finalize job (a coverage gap-fill). Private repo, so I can only share shape and numbers.

uses: openai/codex-action@v1        # floating -> 86365089 (v1.12)
with:
  sandbox: read-only
  safety-strategy: drop-sudo
  output-schema-file: <path>
  output-file: <path>
  # model / effort: see windows below

No timeout-minutes on these jobs, so the GitHub default of 360 applied.

Three windows, same workflow, 260 pass jobs

We changed model/effort at 2026-08-21T11:02Z, which happens to be ~11h after the v1 tag moved to v1.12 (2026-08-20T23:38Z). That leaves a clean middle window where v1.12 was already live and our config had not changed yet.

window	codex-action	model / effort	pass jobs	completed the Codex step
A	v1.11 52fe01ec	gpt-5.6-terra / high	145	145 (100%)
B	v1.12 86365089	gpt-5.6-terra / high	74	69 (93.2%)
C	v1.12 86365089	gpt-5.6-luna / xhigh	41	11 (26.8%)

Two things fall out of that:

1. v1.12 alone is enough to break it. Window A is 145/145. Window B breaks with nothing changed on our side. Our first "lost communication" was 2026-08-21T02:52Z, ~3h after the tag moved — consistent with the 01:35Z first failure reported above.
2. Effort/model is a strong modulator, not the cause. 93% → 27% on the same action SHA. This is the cell the OP's "lowering effort does not help" (n=1) doesn't cover. Lowering it is not a fix, but it moves the failure rate by ~3.5x, so any repro attempt should hold it fixed.

Failure rate tracks how much work the turn does

Within window C, re-sampled slightly later (66 pass jobs, hence the different total), survival correlates with how many files the slice carries:

slices in the run	Codex step completed
1	11/12
2	1/8
3	0/9
4	0/16
6	1/12
9	0/9

In the original sample every pass with index ≥ 2 failed (0/21). The only survivors are the smallest slices — one had 1 changed file and 21 diff anchors. That matches "light runs on v1.12 finish fine" exactly.

Two details that may help narrow it

• sandbox: workspace-write is not required for the harsher variant. The previous comment suggested the escalation from "step never returns" to "runner killed" might come from workspace-write plus very long turns. We get the runner-killed variant with sandbox: read-only, on workloads that ran at p50 ~2 min, max 4.4 min across 145 jobs under v1.11. So it is not exclusive to long or write-sandboxed turns — the same work that took 2 minutes on v1.11 now kills the runner.
• Time-to-death is ~64 min and unrelated to the job budget. With no timeout-minutes set (default 360), GitHub declared the runner dead a very consistent ~64 min after the step started: 64m02s, 64m04s, 63m02s and ~64m on separate runs. The previous comment saw 237 and 363 min against a 120 min limit. Whatever that interval is, it does not track timeout-minutes.
• Confirming the logging symptom: the job log blob 404s on every dead job (BlobNotFound from the REST API, log not found from gh run view --log-failed), so there is never step output to attach.

Both the matrix passes and the independent gap-fill call in the later job die the same way, so it reproduces at two unrelated points of the same workflow.

Why it costs more than the run

Because the runner dies rather than the step failing, continue-on-error: true on our gap-fill step does not help: the whole job goes to failure and the step that publishes the review never runs. We had 4 PRs where the review was fully computed and then discarded, one of them after 2h09 of wall clock. Same "finished work thrown away" as the OP, one job further downstream.

We are pinning to 52fe01ec (v1.11). Happy to run an instrumented comparison if it would help — we have a workload that fails ~73% of the time on v1.12 within a couple of minutes of a trigger, which should make a bisect or a strace/ps capture cheap.

[viyatb-oai]

this has been fixed in bf3eb2e - it will be in a mainline release this week!

[mzdm]

please escalate the release, this is consuming lots of github actions minutes

[viyatb-oai]

The fix is now available in Codex CLI 0.150.0, released today.
No action version change is needed: openai/codex-action@v1 installs the latest CLI by default. If your workflow pins codex-version, update it to "0.150.0".
Please rerun the affected workflows and let us know if you still see the hang after the turn completes.