UI fetches container_state/container_log during PENDING → backend 500s and error UI

[morgan-wowk]

Summary

The UI issues /container_state and /container_log requests for nodes that legitimately have no
container yet, which the backend answers as "not ready." Two problems: (1) the UI makes requests it
should be able to skip, and (2) it renders the not-ready response as a user-facing error.

Status update: the backend side (TangleML/tangle#297) now returns 409 container_not_ready
(no Bugsnag) instead of a 500, so the Bugsnag storm is resolved. What remains is UI-side:
unnecessary requests, a retry storm, and an error box for an expected state.

Corrected root cause

The original framing ("PENDING is missing from CONTAINER_STATUSES_PRE_LAUNCH, so PENDING fetches
race the container row") was investigated at the backend code level and is not accurate:

• PENDING is a red herring. The orchestrator writes container_execution_status = PENDING and
  the container link in the same transaction (atomic), so a PENDING node effectively always has a
  container row. Fetching during PENDING is correct and desirable (early logs). Do not add PENDING
  to the skip set — it would suppress logs for containers that are about to run.

The requests that actually hit the not-ready path are for nodes where container_execution IS NULL:

• Graph / sub-pipeline nodes — never launch a container; their status is permanently None.
  This is the dominant, guaranteed source.
• Pre-launch container nodes — QUEUED / WAITING_FOR_UPSTREAM before the orchestrator launches.

The two guards in the codebase are inconsistent, which is why these slip through:

• useFetchContainerExecutionState (src/services/executionService.ts ~L108):
  !!executionId && (!status || !CONTAINER_STATUSES_PRE_LAUNCH.has(status)) — the !status ||
  clause fetches when status is unknown/None, i.e. for graph nodes.
• Logs/shouldStatusHaveLogs (src/.../TaskOverview/logs.tsx ~L75): if (!status) return false
  — correctly skips when status is unknown. (Different behavior from the state hook.)

Where the unnecessary requests come from

• LogsEventsOverlaySection.tsx (~L33) — biggest offender: calls
  useFetchContainerExecutionState(executionId, backendUrl) with no status argument and no
  subgraph guard. So status is undefined → the !status clause fires → it fetches for every
  node, including graph nodes and pre-launch container nodes. The component even receives status
  as a prop (~L25) but doesn't pass it through.
• ExecutionDetails.tsx (~L40) — already correct: guards subgraph nodes
  (isSubgraph ? undefined : executionId) and passes status. Keep as the reference pattern. Its
  only remaining issue is the error box below.
• AiChat bridges — runBridge.ts (~L183) and debugBridge.ts (~L46) call
  fetchContainerExecutionState directly with no status guard (try/catch swallows the result, but
  the request still hits the backend).

Other gaps

• Retry storm — retryUnlessAuth (src/services/executionService.ts ~L30) retries up to 3× on
  any non-auth error → one not-ready response = 4 requests. It should not retry on 409/404.
• ExecutionDetails.tsx (~L140) renders containerStateError as a red "Failed to load container
  state" InfoBox → an expected not-ready state shows as a user-facing error.
• fetchContainerLog (src/services/executionService.ts ~L93) does a raw fetch(...).json() with
  no status/error handling, so a 409/500 body is parsed as if it were a log response.

Extra findings

https://docs.google.com/document/d/1HlZRkwKZ5KFHE_HvMPOQA7Atkj3fYhqfjQ4DkHT_D4E/edit?usp=sharing

Suggested solution (in priority order)

1. Stop fetching for nodes that can't have a container. Make LogsEventsOverlaySection pass
   status and skip subgraph nodes, matching ExecutionDetails. Unify the two guards so
   container-state and container-log fetches share one rule that skips when status is unknown/None
   or in CONTAINER_STATUSES_PRE_LAUNCH. (Do not add PENDING to the skip set.)
2. Treat 409 container_not_ready as a non-error. Don't set error state; keep polling and show
   a "waiting for container" state instead of the red InfoBox in ExecutionDetails.tsx.
3. Don't retry on not-ready. Update retryUnlessAuth to skip 409/404; only retry
   network/5xx.
4. Gate the AiChat bridges on status the same way (skip subgraph / unknown / pre-launch).
5. Harden fetchContainerLog to use fetchWithErrorHandling and handle the not-ready response
   explicitly.

Acceptance

• Selecting/polling a graph node or a pre-launch node makes no container_state/container_log
  request (or, if one slips through, the 409 is handled silently).
• "Not ready" never surfaces as an error box; it shows a pending/waiting state.
• No retry storm on not-ready responses.
• AiChat bridges gate their fetches on status/node type.

Backend context

Full analysis of why the columns can legitimately be out of sync (graph nodes, pre-launch nodes,
admin override) and why the orchestrator never produces a PENDING-without-container window is in
the investigation doc backing TangleML/tangle#297. The backend fix keys off the container link
(not the status), so it already returns 409 for every container-less case regardless of status.