{tools}[foss/2023b] ADIOS2 v2.10.2#24316
Conversation
|
I just couldn't get a bunch of tests to pass, lots of flakyness and there was always 1 or 2 that fails, typically timing out waiting for a file or something |
Updated software
|
|
@boegelbot please test @ jsc-zen3 |
|
@Crivella: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de PR test command '
Test results coming soon (I hope)... Details- notification for comment with ID 3467260871 processed Message to humans: this is just bookkeeping information for me, |
|
Test report by @boegelbot |
|
For the test suite I am trying to do everything that is being done upstream in their CI:
@Micket Was able to get the tests to pass on my WS with this (now redoing a run with Some notes on the test suiteConcurrency failuresI would say the test suite is not meant to be ran multiple times manually. EG i was testing with: which would work fine on a first run but than give the before mentioned errors after. Deleting the UCXWas not able to get tests to pass with Example of hanging tests
NOTE: this test is supposed to pass in a few seconds when working (atleast without UCX) I am not an expert either with using
Running tests through
|
|
PS: this is my local test report https://gist.github.com/Crivella/b50b7ee64b26caa565adab68546daf76 |
|
Test report by @Crivella |
|
I don't know anything about ADIOS2 either, so I can't have much opinions on what is supposed to be built or not.
After all, the whole purpose with ADIOS2 is scalng IO across many nodes right? https://arxiv.org/pdf/2304.06603 speaks highly of UCX in ADIOS2.
This is not at all what I was doing, nor the type of issues I saw. If i ran the entire test suite, a simple Regarding MPMD tests, i left the config option in that I took from the original ADIOS2 PR I have not seen any difference if i trigger the ctest from ninja or directly. Test that randomly fail typically involve something seemingly waiting for a file that doesn't get created, like (I did also see some MPI issue in some tests now in my latest test builds that I don't recall seeing earlier. Not sure what changed. ) |
Yeah that was one of the thing i was trying, wanted to have it documented so others could avoid wasting time in case they have to debug this. Also not 100% sure UCX is the main culprit, but in the 100% sucess run i had no In the failed report #24316 (comment) i do have the, but not all UCX usage leads to failures grep -i ucx out.txt
grep -i ucx out.txt | grep ': D' | cut -d: -f1 |sort | uniq
for num in `grep -i ucx out.txt | grep ': D' | cut -d: -f1 |sort | uniq`; do grep $num out.txt | grep ' sec'; done
Going to try and dig a little more to see what is happening. |
|
Test report by @Crivella NOTESRun with a temporary
These tests made use of the No instances of UCX appearing in the test logs libtree on adios_core
|
|
Test report by @Crivella |
|
@boegelbot please test @ jsc-zen3 |
|
@Crivella: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de PR test command '
Test results coming soon (I hope)... Details- notification for comment with ID 3473071697 processed Message to humans: this is just bookkeeping information for me, |
|
Test report by @boegelbot |
|
@boegelbot please test @ jsc-zen3 |
|
@Crivella: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de PR test command '
Test results coming soon (I hope)... Details- notification for comment with ID 3473407465 processed Message to humans: this is just bookkeeping information for me, |
|
Test report by @Crivella |
|
Test report by @boegelbot |
|
Build with EESSI + EESSI-extend Test report by @Crivella |
|
Not sure why using Anyway if it works with CMakeMake i'll take it... |
Micket
left a comment
Micket
left a comment
There was a problem hiding this comment.
Maybe we should ask the developers for their input on this.
| configopts = ' '.join([ | ||
| # '-DCTEST_USE_LAUNCHERS=TRUE', # Testing from CI | ||
| '-DBUILD_TESTING=ON', | ||
| '-DADIOS2_USE_UCX=OFF', # Needed to avoid failing/timedout tests |
There was a problem hiding this comment.
I would rather not test at all rather then turning off what seems to be a pretty essential feature
There was a problem hiding this comment.
Got some feedback in my PR to their repo ornladios/ADIOS2#4697.
There was a problem hiding this comment.
Been exploring compiling this with LLVM + MPICH (most of the deps are still taken from GCCcore) since we are working on that toolchain.
Used 2025b as we already have Scipy there
- {toolchain}[system/system] lompi v2025b, lmpich v2025b, llvm-compilers v20.1.8 #24481
- {lang,toolchain}[lfbf/2025b,system/system] lfbf v2025b, SciPy-bundle v2025.07 #24482
Was able to include UCX there and run the test successfully.
Only side note while inspecting the output there are several warnings of the type
[1762940715.192463] [crivella-desktop:2096740:0] mpool.c:54 UCX WARN object 0x55ae5ecb4a40 was not returned to mpool ucp_rkeys
Guess this gives us a direction to look into
4 participants
Using #24194 as a starting point
Needs testing and more third-party dependencies from a dependency