Rework of the JuliaPackage easyblock

[Crivella]

Summary of changes

• Adds compile_cache check to sanity checks to ensure that the compiled version of the package can actually be found/re-used
• Allow specifying test-only dependencies with the is_test_dependency EC param
• Allow turning on Julia's verbose output with the julia_debug EC param
• Allow reusing packages from a dependency (fix JuliaPackage can't really work with depedencies #4123)
  • The environments of the dependencies are merged into the one of the package being installed
• Use Pkg.instantiate() to install all extensions instead of installing them one by one with Pkg.develop
  • Remove need for deps to be ordered in the EC files
  • Makes use of Julia's capability to install extension in parallel (still respecting the max_parallel option)
• Allows running package tests in a dedicated step (setting runtest to True for every extension that needs testing)
  • The test are both skippable (--skip-test-step) and ignorable (--ignore-test-failure)
  • Tests are ran in offline mode by default unless test_online is set to true
  • If both offline and online test are to be performed, the offline ones go first as they should not modify the test environment that we are setting up
• Switch to using an EB specific environment variable to keep track of the LOAD/DEPOT PATHS and use them to correctly set them for Julia in a site-specific startup script

Maybe TODO

• Improve Lua/Tcl syntax in module footer
• Modify Julia itself similar to https://github.com/easybuilders/JSC/blob/6285a066b6debed28211730afb8ffc5eb2430c2f/Custom_EasyBlocks/julia.py to allow the interpreter to pick up the DEPOT/LOAD path directly from the new env variables
• Test how reusable deps really are. The original approach aimed at copying all packages from the deps into the current package and re-install everything. Here we actually re-use them but the results seems to be dependent on the load order (see discussion in JuliaPackage can't really work with depedencies #4123) so if the dep order is switched w.r.t. the one at build time slow re-compilation might happen
  • PoC in {ai}[system/system] Flux v0.16.10 w/ Julia 1.12.2 easybuild-easyconfigs#26214 at-least on package used at the same time should work, more investigation might be needed on loading multiple packages that re-compiles the same deps at the same time

Notes

This PRs includes the changes from:

• Fix for running --sanity-check-only for a JuliaBundle #4122
• JuliaPackage: Allow running tests in a dedicated test_step #4124

Fixes:

• JuliaPackage can't really work with depedencies #4123

AI usage

• GitHub CoPilot FIM auto-completion is active in my IDE and has been used in this PR to sporadically auto-complete code/docstrings

[Crivella]

Moved the final setting of LOAD_PATH and DEPOT_PATH to Julia's site-specific startup script.
This resolves:

• having to write custom module code
• the reversing setenv problem
• Lua-vs-Tcl syntax

[Crivella]

Concerning the points of "re-usability" of deps I am unsure what is the best way.
It is possible to do so, but if we wanted to do things properly, every bundle should never define more than 1 package (or 1 package + its optional subpackages if those are never used independently).

The reason being that if 2 packages define the same extension, the compile cache will could work or be broken depending on the load order of the packages from what i've seen (this is why i check for conflicts when rebuilding the manifest).
I think this mainly happens if one package could use the other as an optional dependency, eg: I am working on a package that depends on JLD2 and Flux as deps.
Flux also has JLD2 as an optional dependency. With this approach if JLD2 is not built into Flux (what i am doing right now) if JLD2 is loaded first everything works, but if not i got into an infinite compilation loop which exited after some max-try limit and ended up working in what i assume is an interpreted-only mode

The old attempted route of just using the dependencies just as a source of information for the packages to install (but build a completely new and isolated environment) also has downsides:

• Bundle should still be "1-package" only or anything using it as a dependency will also bring in potentially un-needed extra packages
• I think weird think could still happen also in this scenario with compile caches depending on load order

EDIT: added a PR for what i have concerning the aforementioned packages

• {ai}[system/system] Flux v0.16.10 w/ Julia 1.12.2 easybuild-easyconfigs#26214

Pinging also @lexming since he was the original author of the easyblock to get feedback on these points and the new EB in general

[Flamefire]

every bundle should never define more than 1 package

Why would we need a bundle then? Isn't this effectively saying: Use JuliaPackage?

[Crivella]

Why would we need a bundle then? Isn't this effectively saying: Use JuliaPackage?

There are some packages that have optional features that are enabled by installing a subpackage. In that case i expected them to always be bundable without problems.

Indeed i think the proper approach would be to only have Package, even in Python we sometime encounter trouble when extensions of one package shadow one of another which depending on version might or might not work

[Flamefire]

Indeed i think the proper approach would be to only have Package, even in Python we sometime encounter trouble when extensions of one package shadow one of another which depending on version might or might not work

I mentioned something like that in Slack recently as it seems like PythonBundle is/should be preferred but IMO it makes it too easy to add additional packages that might silently conflict with others. The suggestion was to enhance the CI checks to catch that.
So might be worth to discuss this in a larger group