Skip to content

QFw: install the service-plane manager's stop handlers before it starts - #137

Merged
DougSO merged 1 commit into
mainfrom
service-plane-sigterm-race
Oct 10, 2026
Merged

DougSO merged 1 commit into
mainfrom
service-plane-sigterm-race

Conversation

@DougSO

@DougSO DougSO commented Oct 10, 2026

Copy link
Copy Markdown
Collaborator

Fixes #135.

The foreground service-plane manager wrote manager_pid to its state file and printed its state, and only then installed its SIGTERM and SIGINT handlers. A caller that sees manager_pid takes the manager as up, as test_foreground_run_stops_on_sigterm does when it sends SIGTERM. A signal in between took the default action:

  • the manager died with rc -15;
  • stop() never ran, so its components were left running;
  • its buffered stdout was lost, hence the empty output in the CI failures.

That is the test's intermittent CI failure, 4 of the 15 failed QFw runs since 2026-09-15.

The fix

run() installs the handlers first, and restores them in a finally that covers start() too. The handlers only set the stop flag:

  • A signal after the plane is up stops it, as before.
  • A signal during start-up is honoured once start() returns, and start()'s own timeouts bound that wait. The plane is then stopped cleanly.

Before, a SIGINT during start-up raised KeyboardInterrupt. That went past start()'s cleanup, which catches Exception only, so the components it had already started were left behind.

Tests

Two new tests wrap a module function so the manager sends SIGTERM to itself as the function returns:

  • test_a_sigterm_just_after_ready_still_stops_the_plane wraps _print_state, which is the race behind the CI failures;
  • test_a_sigterm_during_start_stops_the_plane_once_it_is_up wraps start.

The window is hit every time, so these are deterministic. Without the fix, both fail 3 of 3 runs with assert -15 == 0, the CI signature. With it, both pass, and so does test_foreground_run_stops_on_sigterm.

In the container, on main f2c8e93: mock 619 passed and 1 skipped, unit 130 passed, tests/qiskit 35 passed, ci-syntax clean. That holds with and without the container's telemetry environment.

The foreground manager wrote manager_pid to its state file and printed
its state, and only then installed its SIGTERM and SIGINT handlers. A
caller that sees manager_pid takes the manager as up, as
test_foreground_run_stops_on_sigterm does when it sends SIGTERM. A
signal in between took the default action: the manager died with
rc -15, stop() never ran, and its components were left running. That
is the test's intermittent CI failure, 4 of the 15 failed runs since
2026-09-15.

run() now installs the handlers first and restores them in a finally
that covers start() too. The handlers only set the stop flag, so a
signal during start-up is honoured once start() returns, which its own
timeouts bound, and the plane is stopped rather than cut off partway.
Before, a SIGINT there raised KeyboardInterrupt past start()'s
cleanup, which catches Exception only.

Two tests wrap a module function to send SIGTERM to the manager as it
returns, once just after the ready report and once at the end of
start(), so the window is hit every time. Both fail with rc -15
without the fix.

Fixes #135

Signed-off-by: Doug Oucharek <dougso@me.com>
@DougSO
DougSO merged commit e132fb2 into main Oct 10, 2026
6 checks passed
@DougSO
DougSO deleted the service-plane-sigterm-race branch October 10, 2026 00:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Service-plane foreground manager dies on an early SIGTERM: it reports ready before it installs its handler

1 participant