Skip to content

common.sh: Make die in a subshell actually stop everything immediately - #4211

Draft
chewi wants to merge 2 commits into
mainfrom
chewi/die-faster
Draft

common.sh: Make die in a subshell actually stop everything immediately#4211
chewi wants to merge 2 commits into
mainfrom
chewi/die-faster

Conversation

@chewi

@chewi chewi commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Currently, it is more like a slow motion car crash where parent processes continue to execute, usually with bad data that triggers yet more die calls.

How to use

Run some image build command that fails.

Testing done

Before:

sdk@flatcar-sdk ~/trunk/src/scripts $ ./build_image --board=../../etc --output_root=/var/tmp --group=dev --getbinpkg
ERROR   setup_board: script called: setup_board '--board=../../etc' '--getbinpkgver=' '--regen_configs_only'
ERROR   setup_board: Backtrace:  (most recent call is last)
ERROR   setup_board:   file setup_board, line 190, called: get_board_arch '../../etc'
ERROR   setup_board:   file toolchain_util.sh, line 105, called: get_board_chost '../../etc'
ERROR   setup_board:   file toolchain_util.sh, line 116, called: die 'Unknown board '../../etc''
ERROR   setup_board: 
ERROR   setup_board: Error was:
ERROR   setup_board:   Unknown board '../../etc'
ERROR   setup_board: script called: setup_board '--board=../../etc' '--getbinpkgver=' '--regen_configs_only'
ERROR   setup_board: Backtrace:  (most recent call is last)
ERROR   setup_board:   file setup_board, line 190, called: get_board_arch '../../etc'
ERROR   setup_board:   file toolchain_util.sh, line 105, called: get_portage_arch 
ERROR   setup_board:   file toolchain_util.sh, line 60, called: die 'Unknown CHOST '''
ERROR   setup_board: 
ERROR   setup_board: Error was:
ERROR   setup_board:   Unknown CHOST ''
ERROR   setup_board: script called: setup_board '--board=../../etc' '--getbinpkgver=' '--regen_configs_only'
ERROR   setup_board: Backtrace:  (most recent call is last)
ERROR   setup_board:   file setup_board, line 190, called: die_err_trap 'BOARD_ARCH=$(get_board_arch "$BOARD")' '1'
ERROR   setup_board: 
ERROR   setup_board: Command failed:
ERROR   setup_board:   Command 'BOARD_ARCH=$(get_board_arch "$BOARD")' exited with nonzero code: 1
ERROR   setup_board:   (Note bash sometimes misreports "command not found" as exit code 1 instead of 127)
ERROR   build_image: script called: build_image '--board=../../etc' '--output_root=/var/tmp' '--group=dev' '--getbinpkg'
ERROR   build_image: Backtrace:  (most recent call is last)
ERROR   build_image:   file build_image, line 102, called: die_err_trap '"${SRC_ROOT}/scripts/setup_board" --board="${FLAGS_board}" --getbinpkgver="${FLAGS_getbinpkgver}" --regen_configs_only' '1'
ERROR   build_image: 
ERROR   build_image: Command failed:
ERROR   build_image:   Command '"${SRC_ROOT}/scripts/setup_board" --board="${FLAGS_board}" --getbinpkgver="${FLAGS_getbinpkgver}" --regen_configs_only' exited with nonzero code: 1
ERROR   build_image:   (Note bash sometimes misreports "command not found" as exit code 1 instead of 127)

After:

sdk@flatcar-sdk ~/trunk/src/scripts $ ./build_image --board=../../etc --output_root=/var/tmp --group=dev --getbinpkg
ERROR   setup_board: script called: setup_board '--board=../../etc' '--getbinpkgver=' '--regen_configs_only'
ERROR   setup_board: Backtrace:  (most recent call is last)
ERROR   setup_board:   file setup_board, line 190, called: get_board_arch '../../etc'
ERROR   setup_board:   file toolchain_util.sh, line 105, called: get_board_chost '../../etc'
ERROR   setup_board:   file toolchain_util.sh, line 116, called: die 'Unknown board '../../etc''
ERROR   setup_board: 
ERROR   setup_board: Error was:
ERROR   setup_board:   Unknown board '../../etc'

I've also run an SDK build under Jenkins...

  • Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update) -- N/A
  • Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.

@chewi chewi self-assigned this Aug 12, 2026
Copilot AI lite review requested due to automatic review settings August 12, 2026 16:45

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates common.sh’s fatal error handling so that when die is triggered inside a subshell (e.g., $(...)), it terminates the overall script run immediately rather than allowing the parent shell to continue executing with invalid state.

Changes:

  • Extend die_notrace to detect subshell execution and send a terminating signal to the current process group.
  • Add a SIGTERM trap so the top-level script exits with status code 1 (instead of 143) when terminated via SIGTERM.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread common.sh Outdated
Copilot AI review requested due to automatic review settings August 12, 2026 17:02
@chewi
chewi force-pushed the chewi/die-faster branch from a90f825 to a020d1f Compare August 12, 2026 17:02

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (1)

common.sh:186

  • Using SIGTERM as the “abort whole run” signal is fragile because many scripts legitimately install/override TERM traps for cleanup after sourcing common.sh (e.g. core_sign_update:42 trap cleanup INT TERM EXIT). In that case, kill -s TERM 0 may not actually stop the top-level script (the TERM trap runs and execution resumes), reintroducing the repeated die behavior this PR is trying to eliminate. Consider switching this internal mechanism to a dedicated signal (e.g. USR1) that is unlikely to be trapped elsewhere, and trap that signal in the top-level shell to exit 1.
  # `exit` only leaves the current shell. When die is reached inside a $(...)
  # command substitution or other subshell, the parent keeps running with bad
  # data and usually hits another die, so the same failure gets reported several
  # times. BASHPID (unlike $$, which stays the top-level PID even in subshells)
  # lets us detect that case and signal the main script so the whole run stops

@chewi

chewi commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

Using SIGTERM as the “abort whole run” signal is fragile because many scripts legitimately install/override TERM traps for cleanup after sourcing common.sh (e.g. core_sign_update:42 trap cleanup INT TERM EXIT). In that case, kill -s TERM 0 may not actually stop the top-level script (the TERM trap runs and execution resumes), reintroducing the repeated die behavior this PR is trying to eliminate. Consider switching this internal mechanism to a dedicated signal (e.g. USR1) that is unlikely to be trapped elsewhere, and trap that signal in the top-level shell to exit 1.

core_sign_update was the only case we could find, and that case isn't important here. I think USR1 could cause strange behaviour from other processes in the process group.

Copilot AI review requested due to automatic review settings August 13, 2026 13:42
@chewi
chewi force-pushed the chewi/die-faster branch from a020d1f to c6e3881 Compare August 13, 2026 13:42

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (1)

common.sh:170

  • The comment says "Passing 0 to kill terminates the whole process group", but kill ... 0 only sends the signal to every process in the current process group; processes can still ignore/trap SIGTERM. Tweaking the wording avoids implying a guarantee that isn’t true (and is relevant here since SIGTERM is commonly trapped).
  # lets us detect that case and signal the main script so the whole run stops
  # immediately. Passing 0 to kill terminates the whole process group.

Copilot AI review requested due to automatic review settings August 14, 2026 11:55
chewi and others added 2 commits August 14, 2026 12:55
Currently, it is more like a slow motion car crash where parent
processes continue to execute, usually with bad data that triggers yet
more `die` calls.

Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>
…rocess/audit

The package installed it with a 640 mode, which breaks our tmpfiles
post-processing.

Signed-off-by: Krzesimir Nowak <knowak@microsoft.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (4)

common.sh:186

  • The new subshell-fast-fail relies on SIGTERM to stop the top-level shell, but the current logic explicitly avoids touching an existing TERM trap. If a caller already has a TERM trap that does not exit (e.g. core_sign_update traps TERM for cleanup only), kill -s TERM 0 won’t actually stop the script, so the “stop everything immediately” goal is not guaranteed. Consider chaining any existing TERM handler with an exit 1 (and update the comment that claims the process group is “terminated” by kill 0).
# Only arm this in the real top-level shell (BASHPID == $$), and never clobber
# an existing TERM trap. The latter keeps re-sourcing common.sh idempotent and
# defers to any caller that installed its own handler first. Note that `trap -p`
# reports the parent's traps even from a command substitution.
[[ ${BASHPID:-$$} == $$ && -z $(trap -p TERM) ]] && trap 'exit 1' TERM

sdk_container/src/third_party/coreos-overlay/coreos/config/env/sys-process/audit:4

  • This used to append the mask to both INSTALL_MASK and PKG_INSTALL_MASK. Dropping PKG_INSTALL_MASK means the masked paths can still be installed when emerging from a binary package (the profile explicitly uses PKG_INSTALL_MASK to block files from binpkgs). Unless the intent is to only affect source builds, keep PKG_INSTALL_MASK in sync with INSTALL_MASK here.
INSTALL_MASK+=" /etc/audit/audit.rules* /usr/libexec "

sdk_container/src/third_party/coreos-overlay/coreos/config/env/sys-process/audit:7

  • Typo in comment: “unnecessarilly” → “unnecessarily”.
    # Upstream installs its tmpfiles config file with unnecessarilly

sdk_container/src/third_party/coreos-overlay/coreos/config/env/sys-fs/lvm2:2

  • Typo in comment: “unnecessarilly” → “unnecessarily”.
    # Upstream installs its tmpfiles config file with unnecessarilly

@chewi
chewi force-pushed the chewi/die-faster branch from 5574248 to cadc674 Compare August 14, 2026 13:58
Copilot AI review requested due to automatic review settings August 14, 2026 13:58

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (2)

sdk_container/src/third_party/coreos-overlay/coreos/config/env/sys-process/audit:5

  • PKG_INSTALL_MASK is no longer updated for audit, so the masked files can still be installed into the rootfs when installing from binary packages. Other env overrides update both INSTALL_MASK and PKG_INSTALL_MASK for the same paths, and profiles/coreos/base/make.defaults describes PKG_INSTALL_MASK as controlling binpkg installs.
INSTALL_MASK+=" /etc/audit/audit.rules* /usr/libexec "

sdk_container/src/third_party/coreos-overlay/coreos/config/env/sys-process/audit:7

  • Spelling: “unnecessarilly” -> “unnecessarily”.
    # Upstream installs its tmpfiles config file with unnecessarilly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants