Repository navigation
feat(libc): record traced system library calls on Linux - #7411
3 commits merged into
Conversation
|
Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.
|
| } | ||
|
|
||
| void ReportOnAbort(int number, siginfo_t* info, void* context) { | ||
| std::fprintf(stderr, "[libc] SIGABRT\n"); |
There was a problem hiding this comment.
Follow-up to the same signal-context concern I raised on stacked #7417: this PR introduces the handler that #7417 exercises. ReportOnAbort calls fprintf/fflush, then ReportCallTrace uses dladdr/snprintf and reads the live trace state. These are not async-signal-safe. If SIGABRT arrives while a thread holds a stdio, loader, or trace-related lock, the handler can deadlock before the previous/default action runs, turning the abort into a hang. POSIX restricts signal handlers to async-signal-safe functions (signal actions). Could the handler hand off to normal context or use only a signal-safe, preformatted write path? The end-to-end abort test confirms the report is printed on its normal path, but not when the signal interrupts one of these routines.
There was a problem hiding this comment.
Cheers, good spot. You're right, and I managed to reproduce it: with another thread holding the stderr lock, the old handler just sat there until the test timed out.
Sorted in f670141:
- the handler now only uses
write(2)on a stack buffer, with no stdio; - callers are named from a module list taken outside the handler, so there's no
dladdr; - counts are read without a lock.
I've also added a regression test that holds the stderr lock from another thread during the abort. It timed out before and passes now. #7417 is rebased on top.
f670141 to
ecf408c
Compare
jotitadev
left a comment
There was a problem hiding this comment.
Linux run of ecf408c7c, with two abort races on top of the existing tests. With one abort the report holds up under load. With two threads aborting at the same time, the report is cut off or missing in every run.
Machine: Linux x86_64, i7-8700, Ubuntu 26.04, GCC 15.2, Release, BUILD_TESTING=ON, full build.
Head ecf408c7c merged onto main a373e85e5 (clean merge).
-
Builds.
guest_call_traceandguest_call_trace_abortpass 50/50 with--repeat until-fail:50, and 20/20 pinned to one core withtaskset -c 0. The non-GPU suite (ctest -E "^agc|gpu|vulkan|video_out" -j7) is 747/747.mainis 745/745, and the difference is the two tests this PR registers on Linux. -
Probe, added to
GuestCallTrace.cppas aracemode: install aSIGABRThandler that_exit(3)s, as the abort test does, and traceIntegers. Then 4 threads call the traced import in a loop, and after 20 ms the main thread callsabort(). Each variant ran 300 times with a 10 s timeout:exit 3 report header "most called" section 64 entries entries out of order or with a wrong name hangs one abort()300 300 300 151 (the rest 60–63) 0 0 two threads call abort()together300 35 0 0 0 0 One abort while other threads keep writing the ring is fine. A few slots are skipped because they are mid-write, which is what the sequence check is for. With two aborts, the thread that loses
g_reporting.exchange(true)goes straight to the previous handler while the winner is still writing. Here that handler is_exit(3). In a real run it is usuallySIG_DFL, and thesigaction(SIG_DFL)plusraise()kills the process the same way. So the report is lost exactly when a crash spreads to a second thread. One way to fix it is to have the losing thread wait until the winner has finished writing before it chains, for example with a secondg_reportedflag andnanosleep, which is async-signal-safe. A bounded wait would avoid hanging if the winner never finishes. I tried exactly that: the winner setsg_reportedafterReportCallTrace(), and the loser polls it with a 1 msnanosleepfor at most 2 s before chaining. With that change the two-abort variant printed the full report in 300/300 runs (header and "most called" section in every run, 60–64 entries). The one-abort variant stayed at 300/300, and both call trace tests still pass.
#7417 carries the same CallTrace.cpp, so this applies there too.
Written with AI assistance (Claude Code). I ran the builds and tests above on my machine.
|
@jotitadev Cheers, brilliant testing. You're spot on: the thread that lost the exchange chained straight away and took the report down with it. Sorted in 6c7bfaa, essentially your fix. The winner sets |
jotitadev
left a comment
There was a problem hiding this comment.
Linux rerun of 6c7bfaa29 with the same race probe as my earlier report on ecf408c7c (4 threads calling the traced Integers in a loop, then abort() after 20 ms, SIGABRT handler that _exit(3)s, 300 runs per variant).
Machine: Linux x86_64, i7-8700, Ubuntu 26.04, GCC 15.2, Release, BUILD_TESTING=ON, full build. Head 6c7bfaa29 merged onto main 8816fd772 (clean merge).
ecf408c7c |
6c7bfaa29 |
|
|---|---|---|
one abort() |
header and "most called" 300/300, 60-64 entries | header and "most called" 300/300, 60-64 entries |
two threads call abort() together |
header 35/300, "most called" 0/300 | header and "most called" 300/300, 60-64 entries |
No hangs, no entries out of order or with a wrong name, exit 3 in every run. The call trace tests pass (3/3, including the new guest_call_trace_abort_race), and the non-GPU suite is 778/778 (main 775/775 plus the tests this PR registers on Linux).
Written with AI assistance (Claude Code). I ran the builds and tests above on my machine.
Build the call tracer for Linux as well as Windows. The thunk area is a memfd mapped twice, writable and executable, and threads are named by gettid. Linux has no crash report, so the first traced import installs a SIGABRT handler that prints the last calls and the most called imports, then passes the signal on to the previous handler. guest_call_trace and guest_call_trace_abort now run on Linux too.
The SIGABRT handler used stdio and dladdr, so an abort raised while another thread held the stderr lock or the loader lock hung instead of ending. The report is now formatted into a stack buffer and written with write(2). Callers are named from a module list taken with dl_iterate_phdr when imports are traced, and import counts are read without a lock from a fixed record array. The abort test now holds the stderr lock in another thread, which timed out before this change.
A thread that lost the race to write the report passed SIGABRT on straight away, and the default action ended the process while the first thread was still writing. It now waits for the report to finish, polling with nanosleep for at most 2 seconds, before it chains. guest_call_trace_abort_race aborts from two threads at once and checks the whole report; it failed 35 times in 50 without the wait. Reported by jotitadev.
6c7bfaa to
0bad015
Compare
29e4c46
What
The call tracer behind
--trace-importsandAPS5_TRACE_CALLS(#4546) was built for Windows only. This makes the libc side work on Linux too, as the first half of #3815's Linux support. The relinker change that emits theTraceImportcalls into Linux executables follows in a separate PR that depends on this one.CallTrace.cppmoves fromspecifics/windowstospecifics/x86_64, since the thunk and the register-saving entry are x86-64 code rather than Windows code. It builds on Windows and Linux; macOS is left out.memfdmapped twice, once writable and once executable, matching the two views of the Windows file mapping. Threads are identified bygettid.SIGABRThandler. It prints the same two sections as the Windows report (last 64 calls with the caller asmodule+offsetfromdladdr, then the 32 most called imports) and passes the signal on to the previous handler, or re-raises it with the default action. This covers both guestabort()and hoststd::abort().SIGSEGVdo not print the trace on Linux. libc's guest write watch handlesSIGSEGVandSIGBUSitself, so a handler in front of it would report faults it recovers from. This is recorded indocs/dev/debt/libc.md.Tested
Linux x86-64, GCC 13, full Release build with
BUILD_TESTING=ON.guest_call_traceandguest_call_trace_abortare now registered on Linux as well as Windows, and both pass. They passed 20 runs in a row, including the 4-thread × 20000-call ring check.SIGABRThandler, which exits with status 3 like the Windows CRT. The test therefore also checks that the report chains to the previous handler, and CTest judges the output against the existing regular expression instead of treating the signal as a crash. The executable suffix in that expression now comes fromCMAKE_EXECUTABLE_SUFFIX.Windows and macOS were not built. The Windows code paths are unchanged apart from being moved behind
#ifdef _WIN32.python3 tools/check_conventions.py --base origin/mainpasses.Checklist
main; no other open PR implements the same functionsNotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file.mdfiles or images added to the repository (attach them to this PR)--trace-importsin the relinker (follow-up PR)