Skip to content

improve model debugging workflow - #611

Open
ethernet8023 wants to merge 8 commits into
mainfrom
train-refactor
Open

improve model debugging workflow#611
ethernet8023 wants to merge 8 commits into
mainfrom
train-refactor

modeling: work around noexec tmp dirs for triton cache

2be4669
Select commit
Loading
Failed to load commit list.
Garnix CI / check validate-psyche-http-docs-caddyfile [x86_64-linux] succeeded Mar 18, 2026 in 5m 22s

check validate-psyche-http-docs-caddyfile [x86_64-linux]

check validate-psyche-http-docs-caddyfile [x86_64-linux] succeeded

Details

Last 100 lines of logs:

psyche-sidecar-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-sidecar-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-sidecar-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-sidecar-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-sidecar-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-sidecar-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> +++ command cargo --version
psyche-centralized-server-0.2.0> cargo 1.95.0-nightly (85eff7c80 2026-01-15)
psyche-centralized-server-0.2.0> +++ command cargo build --release --message-format json-render-diagnostics --features parallelism --bin psyche-centralized-server
psyche-sidecar-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-sidecar-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-sidecar-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-sidecar-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-sidecar-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-sidecar-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-sidecar-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-sidecar-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-sidecar-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-sidecar-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-sidecar-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-sidecar-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-sidecar-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-sidecar-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-sidecar-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-sidecar-0.2.0> +++ command cargo --version
psyche-sidecar-0.2.0> cargo 1.95.0-nightly (85eff7c80 2026-01-15)
psyche-sidecar-0.2.0> +++ command cargo build --release --message-format json-render-diagnostics --features python,parallelism --bin psyche-sidecar
psyche-centralized-server-0.2.0>    Compiling pyo3-build-config v0.24.2
psyche-centralized-server-0.2.0>    Compiling psyche-core v0.2.0 (/build/source/shared/core)
psyche-centralized-server-0.2.0>    Compiling psyche-tui v0.2.0 (/build/source/shared/tui)
psyche-centralized-server-0.2.0>    Compiling pyo3-ffi v0.24.2
psyche-centralized-server-0.2.0>    Compiling pyo3-macros-backend v0.24.2
psyche-centralized-server-0.2.0>    Compiling pyo3 v0.24.2
psyche-sidecar-0.2.0>    Compiling bzip2-sys v0.1.13+1.0.8
psyche-sidecar-0.2.0>    Compiling bzip2 v0.4.4
psyche-sidecar-0.2.0>    Compiling zip v0.6.6
psyche-centralized-server-0.2.0>    Compiling psyche-coordinator v0.2.0 (/build/source/shared/coordinator)
psyche-centralized-server-0.2.0>    Compiling psyche-metrics v0.2.0 (/build/source/shared/metrics)
psyche-sidecar-0.2.0>    Compiling torch-sys v0.22.0 (https://github.com/NousResearch/tch-rs.git?rev=dda507e05a776547a112b6854d1e611684f8c729#dda507e0)
psyche-centralized-server-0.2.0>    Compiling psyche-watcher v0.2.0 (/build/source/shared/watcher)
psyche-centralized-server-0.2.0>    Compiling pyo3-macros v0.24.2
psyche-centralized-server-0.2.0>    Compiling pyo3-tch v0.22.0 (https://github.com/NousResearch/tch-rs.git?rev=dda507e05a776547a112b6854d1e611684f8c729#dda507e0)
psyche-centralized-server-0.2.0>    Compiling psyche-modeling v0.2.0 (/build/source/shared/modeling)
psyche-centralized-server-0.2.0>    Compiling psyche-network v0.2.0 (/build/source/shared/network)
psyche-centralized-server-0.2.0>    Compiling psyche-data-provider v0.2.0 (/build/source/shared/data-provider)
psyche-centralized-server-0.2.0>    Compiling psyche-eval v0.2.0 (/build/source/shared/eval)
psyche-centralized-server-0.2.0>    Compiling psyche-client v0.2.0 (/build/source/shared/client)
psyche-centralized-server-0.2.0>    Compiling psyche-centralized-shared v0.2.0 (/build/source/architectures/centralized/shared)
psyche-centralized-server-0.2.0>    Compiling psyche-centralized-server v0.2.0 (/build/source/architectures/centralized/server)
psyche-centralized-server-0.2.0>     Finished `release` profile [optimized] target(s) in 25.13s
psyche-centralized-server-0.2.0> searching for bins/libs to install from cargo build log at cargoBuildLogx12C.json
psyche-centralized-server-0.2.0> installing /build/source/target/release/psyche-centralized-server in postBuildInstallFromCargoBuildLogOutTempQbt/bin
psyche-centralized-server-0.2.0> searching for bins/libs complete
psyche-centralized-server-0.2.0> buildPhase completed in 30 seconds
psyche-centralized-server-0.2.0> Running phase: installPhase
psyche-centralized-server-0.2.0> stripping references to Rust toolchain
psyche-centralized-server-0.2.0> Rust toolchain at: /nix/store/g8lx95m2mcshkbshdhzyfddak8zh774y-rust-default-1.95.0-nightly-2026-01-22
psyche-centralized-server-0.2.0> stripping Rust toolchain references done
psyche-centralized-server-0.2.0> stripping references to cargoVendorDir from:
psyche-centralized-server-0.2.0> /nix/store/c0pgjdhbgaxpi59xxp6ybqwvwzgq5qw7-psyche-centralized-server-0.2.0/bin/psyche-centralized-server
psyche-centralized-server-0.2.0> stripping references done
psyche-centralized-server-0.2.0> Running phase: fixupPhase
psyche-centralized-server-0.2.0> shrinking RPATHs of ELF executables and libraries in /nix/store/c0pgjdhbgaxpi59xxp6ybqwvwzgq5qw7-psyche-centralized-server-0.2.0
psyche-centralized-server-0.2.0> shrinking /nix/store/c0pgjdhbgaxpi59xxp6ybqwvwzgq5qw7-psyche-centralized-server-0.2.0/bin/psyche-centralized-server
psyche-centralized-server-0.2.0> checking for references to /build/ in /nix/store/c0pgjdhbgaxpi59xxp6ybqwvwzgq5qw7-psyche-centralized-server-0.2.0...
psyche-centralized-server-0.2.0> patching script interpreter paths in /nix/store/c0pgjdhbgaxpi59xxp6ybqwvwzgq5qw7-psyche-centralized-server-0.2.0
psyche-centralized-server-0.2.0> stripping (with command strip and flags -S -p) in  /nix/store/c0pgjdhbgaxpi59xxp6ybqwvwzgq5qw7-psyche-centralized-server-0.2.0/bin
psyche-centralized-server-0.2.0> shrinking RPATHs of ELF executables and libraries in /nix/store/c0pgjdhbgaxpi59xxp6ybqwvwzgq5qw7-psyche-centralized-server-0.2.0
psyche-centralized-server-0.2.0> shrinking /nix/store/c0pgjdhbgaxpi59xxp6ybqwvwzgq5qw7-psyche-centralized-server-0.2.0/bin/psyche-centralized-server
psyche-centralized-server-0.2.0> autoFixElfFiles: using removeStubsFromRunpath to fix /nix/store/c0pgjdhbgaxpi59xxp6ybqwvwzgq5qw7-psyche-centralized-server-0.2.0/bin/psyche-centralized-server
psyche-sidecar-0.2.0>    Compiling tch v0.22.0 (https://github.com/NousResearch/tch-rs.git?rev=dda507e05a776547a112b6854d1e611684f8c729#dda507e0)
psyche-sidecar-0.2.0>    Compiling psyche-sidecar v0.2.0 (/build/source/tools/rust-tools/psyche-sidecar)
psyche-sidecar-0.2.0>     Finished `release` profile [optimized] target(s) in 59.05s
psyche-sidecar-0.2.0> searching for bins/libs to install from cargo build log at cargoBuildLogDGYb.json
psyche-sidecar-0.2.0> installing /build/source/target/release/psyche-sidecar in postBuildInstallFromCargoBuildLogOutTempPre/bin
psyche-sidecar-0.2.0> searching for bins/libs complete
psyche-sidecar-0.2.0> buildPhase completed in 1 minutes 3 seconds
psyche-sidecar-0.2.0> Running phase: installPhase
psyche-sidecar-0.2.0> stripping references to Rust toolchain
psyche-sidecar-0.2.0> Rust toolchain at: /nix/store/g8lx95m2mcshkbshdhzyfddak8zh774y-rust-default-1.95.0-nightly-2026-01-22
psyche-sidecar-0.2.0> stripping Rust toolchain references done
psyche-sidecar-0.2.0> stripping references to cargoVendorDir from:
psyche-sidecar-0.2.0> /nix/store/bqlfa00c5279hrlyyw9h4s0mqn3r1rxy-psyche-sidecar-0.2.0/bin/psyche-sidecar
psyche-sidecar-0.2.0> stripping references done
psyche-sidecar-0.2.0> Running phase: fixupPhase
psyche-sidecar-0.2.0> shrinking RPATHs of ELF executables and libraries in /nix/store/bqlfa00c5279hrlyyw9h4s0mqn3r1rxy-psyche-sidecar-0.2.0
psyche-sidecar-0.2.0> shrinking /nix/store/bqlfa00c5279hrlyyw9h4s0mqn3r1rxy-psyche-sidecar-0.2.0/bin/psyche-sidecar
psyche-sidecar-0.2.0> checking for references to /build/ in /nix/store/bqlfa00c5279hrlyyw9h4s0mqn3r1rxy-psyche-sidecar-0.2.0...
psyche-sidecar-0.2.0> patching script interpreter paths in /nix/store/bqlfa00c5279hrlyyw9h4s0mqn3r1rxy-psyche-sidecar-0.2.0
psyche-sidecar-0.2.0> stripping (with command strip and flags -S -p) in  /nix/store/bqlfa00c5279hrlyyw9h4s0mqn3r1rxy-psyche-sidecar-0.2.0/bin
psyche-sidecar-0.2.0> shrinking RPATHs of ELF executables and libraries in /nix/store/bqlfa00c5279hrlyyw9h4s0mqn3r1rxy-psyche-sidecar-0.2.0
psyche-sidecar-0.2.0> shrinking /nix/store/bqlfa00c5279hrlyyw9h4s0mqn3r1rxy-psyche-sidecar-0.2.0/bin/psyche-sidecar
psyche-sidecar-0.2.0> autoFixElfFiles: using removeStubsFromRunpath to fix /nix/store/bqlfa00c5279hrlyyw9h4s0mqn3r1rxy-psyche-sidecar-0.2.0/bin/psyche-sidecar
validate-psyche-http-docs-caddyfile> 2026/03/18 22:11:57.765	INFO	using config from file	{"file": "./Caddyfile"}
validate-psyche-http-docs-caddyfile> 2026/03/18 22:11:57.766	INFO	adapted config to JSON	{"adapter": "caddyfile"}
validate-psyche-http-docs-caddyfile> Valid configuration