Skip to content

ci: serialize make distcheck - #8983

Merged
alexey-tikhonov merged 1 commit into
SSSD:masterfrom
pbrezina:parallel-distcheck
Jul 24, 2026
Merged

ci: serialize make distcheck#8983
alexey-tikhonov merged 1 commit into
SSSD:masterfrom
pbrezina:parallel-distcheck

Conversation

@pbrezina

@pbrezina pbrezina commented Jul 21, 2026

Copy link
Copy Markdown
Member

I saw make distcheck fail recently with linker due to a race condition when libsss_util.so was not available for libsss_ldap.

I let AI handle it:

  Root cause: Parallel libtool relink race condition during make distcheck install phase

  The failure is on line 27479-27480:
  libsss_util.so: file not recognized: file format not recognized
  collect2: error: ld returned 1 exit status

  What happened — two libtool relink commands ran in parallel during make install:

  1. Line 27474 (at 27.133s): libtool starts relinking libsss_util.so, writing to _build/sub/.libs/libsss_util.so
  2. Line 27478 (at 27.229s, ~100ms later): libtool starts relinking libsss_ldap.so, which links against -lsss_util via -L_build/sub/.libs
  3. Line 27479 (at 27.240s, ~10ms later): the libsss_ldap linker tries to read libsss_util.so from .libs/ — but that file is still being written by the concurrent libsss_util relink, so the linker sees a partially-written
  (truncated/corrupt) .so file


Libtool relinks shared libraries during install. With parallel make,
install-sssdlibLTLIBRARIES and install-pkglibLTLIBRARIES run
concurrently, causing provider plugin relinks to read half-written
base library .so files (e.g. libsss_ldap reads libsss_util.so
mid-relink). Serialize the install targets to prevent this.

Assisted-By: Claude Code (Opus 4.6)

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request modifies Makefile.am to serialize install-sssdlibLTLIBRARIES after install-pkglibLTLIBRARIES and install-libLTLIBRARIES to prevent parallel installation race conditions during Libtool relinking. The reviewer suggests extending this serialization to other library groups (such as ldblib, py3exec, cifsplugin, and winbindplugin) that also link against these base libraries and are susceptible to the same race condition.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread Makefile.am Outdated
@alexey-tikhonov

Copy link
Copy Markdown
Member

I let AI handle it:

I tried this in the past and gave up after several iterations. But maybe it got better.

There is also bcf84e1 that I was going to extract from #8847 to see if it helps.

@pbrezina
pbrezina force-pushed the parallel-distcheck branch from 16fae36 to 1a219cb Compare July 21, 2026 21:47
@pbrezina

Copy link
Copy Markdown
Member Author

I let AI handle it:

I tried this in the past and gave up after several iterations. But maybe it got better.

There is also bcf84e1 that I was going to extract from #8847 to see if it helps.

It's basically the same fix, this PR is just more complete. I also addressed gemini notes, let's see what CI will look like.

@alexey-tikhonov

Copy link
Copy Markdown
Member

Btw, for some (unknown to me) reason, branches other than 'master' fail distcheck much more often.

@pbrezina
pbrezina force-pushed the parallel-distcheck branch from acc32f8 to 38e9719 Compare July 22, 2026 08:54
@pbrezina pbrezina changed the title distcheck: fix parallel install race between library groups ci: serialize make distcheck Jul 22, 2026
@pbrezina

pbrezina commented Jul 22, 2026

Copy link
Copy Markdown
Member Author

Well, fixing one problem revealed another one. The first version where only install-sssdlibLTLIBRARIES: install-pkglibLTLIBRARIES install-libLTLIBRARIES seem to work, but since there are other dependencies that needs to be ordered it is not complete. All other fixes just revelead another issues and eventually AI started guessing and looping. So serializing the distcheck is the easiest fix for CI.

@pbrezina
pbrezina marked this pull request as ready for review July 22, 2026 09:48
@gemini-code-assist

Copy link
Copy Markdown

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@thalman thalman added Waiting for review Trivial A single reviewer is sufficient to review the Pull Request labels Jul 23, 2026
@ikerexxe

Copy link
Copy Markdown
Contributor

@pbrezina can you set the backport labels? This problem has been hitting heavily sssd-2-9 and I'd like to see it ported there

Comment thread .github/workflows/build.yml
The parallel distcheck is full of potentiall race conditions because
there is no defined ordering on the libraries. There were attempts to
fix this using CI, but fixed one problem just revealed another one and
ultimately the fix was becomming more complex and magical but still
incomplete.

Reviewed-by: Alexey Tikhonov <atikhono@redhat.com>
@sssd-bot

Copy link
Copy Markdown
Contributor

The pull request was accepted by @alexey-tikhonov with the following PR CI status:


🟢 CodeQL (success)
🟢 osh-diff-scan:fedora-rawhide-x86_64:upstream (success)
🟢 rpm-build:centos-stream-10-x86_64:upstream (success)
🟢 rpm-build:fedora-43-x86_64:upstream (success)
🟢 rpm-build:fedora-44-x86_64:upstream (success)
🟢 rpm-build:fedora-rawhide-x86_64:upstream (success)
🟢 testing-farm:centos-stream-10-x86_64:upstream (success)
🔴 testing-farm:fedora-43-x86_64:upstream (failure)
🔴 testing-farm:fedora-44-x86_64:upstream (failure)
🔴 testing-farm:fedora-rawhide-x86_64:upstream (failure)
🟢 Build / freebsd (success)
🟢 Build / make-distcheck (success)
🟢 ci / intgcheck (centos-10) (success)
🟢 ci / intgcheck (fedora-43) (success)
🟢 ci / intgcheck (fedora-44) (success)
🟢 ci / intgcheck (fedora-45) (success)
🟢 ci / prepare (success)
🟢 ci / system (centos-10) (success)
🟢 ci / system (fedora-43) (success)
🔴 ci / system (fedora-44) (failure)
🔴 ci / system (fedora-45) (failure)
➖ Coverity scan / coverity (skipped)
🟢 Static code analysis / codeql (success)
🟢 Static code analysis / pre-commit (success)
🟢 Static code analysis / python-system-tests (success)


There are unsuccessful or unfinished checks. Make sure that the failures are not related to this pull request before merging.

@sssd-bot
sssd-bot force-pushed the parallel-distcheck branch from 38e9719 to 1c17ad9 Compare July 24, 2026 14:36
@alexey-tikhonov
alexey-tikhonov merged commit df13bd9 into SSSD:master Jul 24, 2026
9 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants