Skip to content

Keep startup workflow binds recoverable - #3172

Open
louis4li wants to merge 2 commits into
feature/integratefrom
fix/2026-08-03_workflow-startup-bind-recovery
Open

Keep startup workflow binds recoverable#3172
louis4li wants to merge 2 commits into
feature/integratefrom
fix/2026-08-03_workflow-startup-bind-recovery

Conversation

@louis4li

@louis4li louis4li commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Convert startup workflow bind observation timeout into a recoverable materialization failure even when the host startup token races the bind timeout.
  • Let workflow definition bootstrap continue host startup when a bind commit is not observed, and continue remaining startup materialization in a background retry.
  • Cover the hosted-service path where a startup bind is dispatched but no committed bind is observed.

Test plan

  • dotnet test test/Aevatar.Workflow.Host.Api.Tests/Aevatar.Workflow.Host.Api.Tests.csproj --nologo --filter \"FullyQualifiedName~FileBackedWorkflowCatalogAdmissionTests\"
  • dotnet test test/Aevatar.Workflow.Host.Api.Tests/Aevatar.Workflow.Host.Api.Tests.csproj --nologo (after stopping stale local Aevatar host on port 5000)
  • bash tools/ci/workflow_binding_boundary_guard.sh
  • bash tools/ci/test_stability_guards.sh could not complete locally: Python 3.12 pyexpat fails to load _XML_SetAllocTrackerActivationThreshold; the guard's first two checks passed before the local Python XML parser failure.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@louis4li
louis4li requested a review from jason-aelf as a code owner August 3, 2026 11:29

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5c0f4c6e66

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

"Workflow definition bind commit retry delay must be positive.");
}

var remainingDefinitions = GetDefinitionsAfter(definitions, timedOutWorkflowName);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep the timed-out workflow in the retry set

When an accepted dispatch never commits, GetDefinitionsAfter removes the failed workflow from the background retry set. For a single or last configured workflow this returns an empty list and starts no retry task; the same removal happens on each later timeout, so the host can run indefinitely without materializing that definition. Dispatch admission is explicitly accepted-only, so the timed-out definition must remain pending, with a committed-binding check before redispatch, rather than being treated as complete.

AGENTS.md reference: AGENTS.md:L57-L57

Useful? React with 👍 / 👎.

Comment on lines +49 to +51
catch (WorkflowDefinitionMaterializationException ex)
when (ex.Code == WorkflowDefinitionMaterializationException.BindNotCommittedCode &&
!cancellationToken.IsCancellationRequested)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Recover the bind timeout when the startup token races it

When the host startup token and the private bind timeout are both requested, WaitForCommittedBindAsync now converts the cancellation into BindNotCommitted, but this filter immediately rejects that exception because the startup token is canceled. The exception therefore still escapes StartAsync and aborts host startup in the exact race this change is intended to recover; the added test uses CancellationToken.None and does not exercise this path.

Useful? React with 👍 / 👎.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant