Skip to content

Preserve Hugging Face metadata when checkpointing - #647

Open
dylantirandaz wants to merge 1 commit into
PsycheFoundation:mainfrom
dylantirandaz:fix/631-checkpoint-metadata
Open

Preserve Hugging Face metadata when checkpointing#647
dylantirandaz wants to merge 1 commit into
PsycheFoundation:mainfrom
dylantirandaz:fix/631-checkpoint-metadata

Preserve Hugging Face metadata in checkpoints

f3dc6cb
Select commit
Loading
Failed to load commit list.
Garnix CI / check validate-all-configs [x86_64-linux] succeeded Jul 13, 2026 in 22m 29s

check validate-all-configs [x86_64-linux]

check validate-all-configs [x86_64-linux] succeeded

Details

Last 100 lines of logs:

psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> ++++++ command cargo metadata --format-version 1 --no-deps
psyche-centralized-server-0.2.0> +++ command cargo --version
psyche-centralized-server-0.2.0> cargo 1.95.0-nightly (85eff7c80 2026-01-15)
psyche-centralized-server-0.2.0> +++ command cargo build --release --message-format json-render-diagnostics --features parallelism --bin psyche-centralized-server
psyche-centralized-server-0.2.0>    Compiling pyo3-build-config v0.24.2
psyche-centralized-server-0.2.0>    Compiling psyche-core v0.2.0 (/build/source/shared/core)
psyche-centralized-server-0.2.0>    Compiling pyo3-macros-backend v0.24.2
psyche-centralized-server-0.2.0>    Compiling pyo3-ffi v0.24.2
psyche-centralized-server-0.2.0>    Compiling pyo3 v0.24.2
psyche-centralized-server-0.2.0>    Compiling psyche-coordinator v0.2.0 (/build/source/shared/coordinator)
psyche-centralized-server-0.2.0>    Compiling psyche-metrics v0.2.0 (/build/source/shared/metrics)
psyche-centralized-server-0.2.0>    Compiling psyche-event-sourcing v0.2.0 (/build/source/shared/event-sourcing)
psyche-centralized-server-0.2.0>    Compiling pyo3-macros v0.24.2
psyche-centralized-server-0.2.0>    Compiling psyche-tui v0.2.0 (/build/source/shared/tui)
psyche-centralized-server-0.2.0>    Compiling psyche-watcher v0.2.0 (/build/source/shared/watcher)
psyche-centralized-server-0.2.0>    Compiling psyche-centralized-shared v0.2.0 (/build/source/architectures/centralized/shared)
psyche-centralized-server-0.2.0>    Compiling pyo3-tch v0.22.0 (https://github.com/NousResearch/tch-rs.git?rev=dda507e05a776547a112b6854d1e611684f8c729#dda507e0)
psyche-centralized-server-0.2.0>    Compiling psyche-modeling v0.2.0 (/build/source/shared/modeling)
psyche-centralized-server-0.2.0>    Compiling psyche-network v0.2.0 (/build/source/shared/network)
psyche-centralized-server-0.2.0>    Compiling psyche-data-provider v0.2.0 (/build/source/shared/data-provider)
psyche-centralized-server-0.2.0>    Compiling psyche-eval v0.2.0 (/build/source/shared/eval)
psyche-centralized-server-0.2.0>    Compiling psyche-client v0.2.0 (/build/source/shared/client)
psyche-centralized-server-0.2.0>    Compiling psyche-centralized-server v0.2.0 (/build/source/architectures/centralized/server)
psyche-centralized-server-0.2.0>     Finished `release` profile [optimized] target(s) in 26.96s
psyche-centralized-server-0.2.0> searching for bins/libs to install from cargo build log at cargoBuildLogb1uR.json
psyche-centralized-server-0.2.0> installing /build/source/target/release/psyche-centralized-server in postBuildInstallFromCargoBuildLogOutTempiG6/bin
psyche-centralized-server-0.2.0> searching for bins/libs complete
psyche-centralized-server-0.2.0> buildPhase completed in 31 seconds
psyche-centralized-server-0.2.0> Running phase: installPhase
psyche-centralized-server-0.2.0> stripping references to Rust toolchain
psyche-centralized-server-0.2.0> Rust toolchain at: /nix/store/g8lx95m2mcshkbshdhzyfddak8zh774y-rust-default-1.95.0-nightly-2026-01-22
psyche-centralized-server-0.2.0> stripping Rust toolchain references done
psyche-centralized-server-0.2.0> stripping references to cargoVendorDir from:
psyche-centralized-server-0.2.0> /nix/store/4jph73a0gkd714sjsp1vc57crs325jxb-psyche-centralized-server-0.2.0/bin/psyche-centralized-server
psyche-centralized-server-0.2.0> stripping references done
psyche-centralized-server-0.2.0> Running phase: fixupPhase
psyche-centralized-server-0.2.0> shrinking RPATHs of ELF executables and libraries in /nix/store/4jph73a0gkd714sjsp1vc57crs325jxb-psyche-centralized-server-0.2.0
psyche-centralized-server-0.2.0> shrinking /nix/store/4jph73a0gkd714sjsp1vc57crs325jxb-psyche-centralized-server-0.2.0/bin/psyche-centralized-server
psyche-centralized-server-0.2.0> checking for references to /build/ in /nix/store/4jph73a0gkd714sjsp1vc57crs325jxb-psyche-centralized-server-0.2.0...
psyche-centralized-server-0.2.0> patching script interpreter paths in /nix/store/4jph73a0gkd714sjsp1vc57crs325jxb-psyche-centralized-server-0.2.0
psyche-centralized-server-0.2.0> stripping (with command strip and flags -S -p) in  /nix/store/4jph73a0gkd714sjsp1vc57crs325jxb-psyche-centralized-server-0.2.0/bin
psyche-centralized-server-0.2.0> shrinking RPATHs of ELF executables and libraries in /nix/store/4jph73a0gkd714sjsp1vc57crs325jxb-psyche-centralized-server-0.2.0
psyche-centralized-server-0.2.0> shrinking /nix/store/4jph73a0gkd714sjsp1vc57crs325jxb-psyche-centralized-server-0.2.0/bin/psyche-centralized-server
psyche-centralized-server-0.2.0> autoFixElfFiles: using removeStubsFromRunpath to fix /nix/store/4jph73a0gkd714sjsp1vc57crs325jxb-psyche-centralized-server-0.2.0/bin/psyche-centralized-server
validate-configs> Note: client has no state.toml, skipping validation
validate-configs> checking consilience/state.toml
validate-configs> 2026-07-13T15:32:47.327116Z  INFO psyche_centralized_server: Configs are OK!
validate-configs> ok!
validate-configs> checking consilience-match-llama2-20m-fineweb-pretrain-dev/data.toml and consilience-match-llama2-20m-fineweb-pretrain-dev/state.toml
validate-configs> 2026-07-13T15:32:48.295816Z  INFO psyche_centralized_server: Configs are OK!
validate-configs> ok!
validate-configs> checking kill-test-short-epoch-checkpoint/data.toml and kill-test-short-epoch-checkpoint/state.toml
validate-configs> 2026-07-13T15:32:48.804699Z  INFO psyche_centralized_server: Configs are OK!
validate-configs> ok!
validate-configs> checking llama2-20m-dolma-noverify-first-checkpointer/data.toml and llama2-20m-dolma-noverify-first-checkpointer/state.toml
validate-configs> 2026-07-13T15:32:49.303488Z  INFO psyche_centralized_server: Configs are OK!
validate-configs> ok!
validate-configs> checking llama2-20m-dolma-noverify-no-checkpointer/data.toml and llama2-20m-dolma-noverify-no-checkpointer/state.toml
validate-configs> 2026-07-13T15:32:49.754460Z  INFO psyche_centralized_server: Configs are OK!
validate-configs> ok!
validate-configs> checking llama2-20m-dolma-verify-first-checkpointer/data.toml and llama2-20m-dolma-verify-first-checkpointer/state.toml
validate-configs> 2026-07-13T15:32:50.028119Z  INFO psyche_centralized_server: Configs are OK!
validate-configs> ok!
validate-configs> Note: solana-test has no state.toml, skipping validation
validate-configs> checking test/state.toml
validate-configs> 2026-07-13T15:32:50.300874Z  INFO psyche_centralized_server: Configs are OK!
validate-configs> ok!
validate-configs> all configs ok!