Skip to content

Preserve Hugging Face metadata when checkpointing - #647

Open
dylantirandaz wants to merge 1 commit into
PsycheFoundation:mainfrom
dylantirandaz:fix/631-checkpoint-metadata
Open

Preserve Hugging Face metadata when checkpointing#647
dylantirandaz wants to merge 1 commit into
PsycheFoundation:mainfrom
dylantirandaz:fix/631-checkpoint-metadata

Preserve Hugging Face metadata in checkpoints

f3dc6cb
Select commit
Loading
Failed to load commit list.
Garnix CI / check workspace-test-ci [x86_64-linux] succeeded Jul 13, 2026 in 27m 11s

check workspace-test-ci [x86_64-linux]

check workspace-test-ci [x86_64-linux] succeeded

Details

Last 100 lines of logs:

psyche-workspace-nextest-0.2.0>         PASS [   0.585s] (157/243) psyche-modeling distro::tests::test_compress_2d
psyche-workspace-nextest-0.2.0>         PASS [   0.728s] (158/243) psyche-data-provider::weighted test_weighted_data_provider_auto_weights
psyche-workspace-nextest-0.2.0>         PASS [   0.626s] (159/243) psyche-modeling distro::tests::test_dct
psyche-workspace-nextest-0.2.0>         PASS [   0.767s] (160/243) psyche-data-provider::weighted test_weighted_data_provider_exhaustive
psyche-workspace-nextest-0.2.0>         PASS [   0.579s] (161/243) psyche-modeling distro::tests::test_decompress_2d
psyche-workspace-nextest-0.2.0>         PASS [   0.581s] (162/243) psyche-modeling distro::tests::test_encode_1d
psyche-workspace-nextest-0.2.0>         PASS [   0.679s] (163/243) psyche-modeling distro::tests::test_decompress_1d
psyche-workspace-nextest-0.2.0>         PASS [   0.628s] (164/243) psyche-modeling distro::tests::test_encode_2d
psyche-workspace-nextest-0.2.0>         PASS [   0.872s] (165/243) psyche-modeling device_utils::tests::test_device_functionality
psyche-workspace-nextest-0.2.0>         PASS [   0.826s] (166/243) psyche-modeling distro::tests::test_compress_1d
psyche-workspace-nextest-0.2.0>         PASS [   0.949s] (167/243) psyche-data-provider::local loads_fineweb_subset
psyche-workspace-nextest-0.2.0>         PASS [   0.956s] (168/243) psyche-data-provider::http test_http_data_provider
psyche-workspace-nextest-0.2.0>         PASS [   1.035s] (169/243) psyche-data-provider weighted::http::tests::test_http_multi_data_provider
psyche-workspace-nextest-0.2.0>         PASS [   1.054s] (170/243) psyche-modeling distro::tests::test_1bit_matches_non_quant
psyche-workspace-nextest-0.2.0>         PASS [   0.632s] (171/243) psyche-modeling distro::tests::test_get_divisors
psyche-workspace-nextest-0.2.0>         PASS [   0.931s] (172/243) psyche-modeling distro::tests::test_edge_cases
psyche-workspace-nextest-0.2.0>         PASS [   0.610s] (173/243) psyche-modeling distro::tests::test_idct
psyche-workspace-nextest-0.2.0>         PASS [   0.696s] (174/243) psyche-modeling distro::tests::test_get_smaller_split
psyche-workspace-nextest-0.2.0>         PASS [   0.621s] (175/243) psyche-network download::scheduler::tests::test_distro_retries_returned_and_removed
psyche-workspace-nextest-0.2.0>         PASS [   0.642s] (176/243) psyche-network download::scheduler::tests::test_distro_max_retries_exceeded
psyche-workspace-nextest-0.2.0>         PASS [   0.679s] (177/243) psyche-modeling distro::tests::test_signed_vals_reconstructs_original_sign
psyche-workspace-nextest-0.2.0>         PASS [   0.005s] (178/243) psyche-solana-authorizer test_id
psyche-workspace-nextest-0.2.0>         PASS [   0.005s] (179/243) psyche-solana-coordinator test_id
psyche-workspace-nextest-0.2.0>         PASS [   0.005s] (180/243) psyche-solana-distributor test_id
psyche-workspace-nextest-0.2.0>         PASS [   0.005s] (181/243) psyche-solana-mining-pool test_id
psyche-workspace-nextest-0.2.0>         PASS [   0.786s] (182/243) psyche-modeling distro::tests::test_get_prime_divisors
psyche-workspace-nextest-0.2.0>         PASS [   0.016s] (183/243) psyche-solana-tooling::mod suites::memnet_coordinator_data_layout::run
psyche-workspace-nextest-0.2.0>         PASS [   0.615s] (184/243) psyche-network download::scheduler::tests::test_wait_for_capacity_errors_on_actor_shutdown
psyche-workspace-nextest-0.2.0>         PASS [   0.722s] (185/243) psyche-network download::scheduler::tests::test_config_retries_dont_consume_capacity
psyche-workspace-nextest-0.2.0>         PASS [   0.760s] (186/243) psyche-modeling distro::tests::test_large_numbers
psyche-workspace-nextest-0.2.0>         PASS [   0.682s] (187/243) psyche-network download::scheduler::tests::test_remove_retry
psyche-workspace-nextest-0.2.0>         PASS [   1.411s] (188/243) psyche-data-provider::http test_http_data_provider_shuffled
psyche-workspace-nextest-0.2.0>         PASS [   0.567s] (189/243) psyche-network serializable_tensor::tests::test_roundtrip_bool_tensor_single_element
psyche-workspace-nextest-0.2.0>         PASS [   0.765s] (190/243) psyche-network download::scheduler::tests::test_parameter_retries_are_immediate
psyche-workspace-nextest-0.2.0>         PASS [   0.684s] (191/243) psyche-network serializable_tensor::tests::test_roundtrip_bool_tensor2d
psyche-workspace-nextest-0.2.0>         PASS [   0.846s] (192/243) psyche-network download::scheduler::tests::test_model_sharing_retry_respects_capacity
psyche-workspace-nextest-0.2.0>         PASS [   0.863s] (193/243) psyche-network download::scheduler::tests::test_release_unblocks_waiter
psyche-workspace-nextest-0.2.0>         PASS [   0.791s] (194/243) psyche-network router::tests::test_shutdown
psyche-workspace-nextest-0.2.0>         PASS [   0.968s] (195/243) psyche-network download::scheduler::tests::test_capacity_grants_up_to_max
psyche-workspace-nextest-0.2.0>         PASS [   0.891s] (196/243) psyche-network download::scheduler::tests::test_model_sharing_never_exceeds_max_retries
psyche-workspace-nextest-0.2.0>         PASS [   0.005s] (197/243) psyche-solana-treasurer test_id
psyche-workspace-nextest-0.2.0>         PASS [   0.874s] (198/243) psyche-network download::scheduler::tests::test_waiters_are_served_fifo
psyche-workspace-nextest-0.2.0>         PASS [   0.703s] (199/243) psyche-network serializable_tensor::tests::test_roundtrip_bool_tensor_unusual_shape
psyche-workspace-nextest-0.2.0>         PASS [   0.776s] (200/243) psyche-network serializable_tensor::tests::test_roundtrip_bool_tensor_non_divisible_by_8
psyche-workspace-nextest-0.2.0>         PASS [   0.851s] (201/243) psyche-network serializable_tensor::tests::test_roundtrip_bool_tensor1d
psyche-workspace-nextest-0.2.0>         PASS [   0.180s] (202/243) psyche-solana-tooling::mod suites::memnet_mining_pool_live_freeze::run
psyche-workspace-nextest-0.2.0>         PASS [   0.223s] (203/243) psyche-solana-tooling::mod suites::memnet_mining_pool_cap_deposits::run
psyche-workspace-nextest-0.2.0>         PASS [   1.056s] (204/243) psyche-network download::scheduler::tests::test_distro_retries_dont_consume_capacity
psyche-workspace-nextest-0.2.0>         PASS [   0.412s] (205/243) psyche-solana-tooling::mod suites::memnet_authorizer_full_cycle::run
psyche-workspace-nextest-0.2.0>         PASS [   0.382s] (206/243) psyche-solana-tooling::mod suites::memnet_coordinator_init_free::run
psyche-workspace-nextest-0.2.0>         PASS [   1.737s] (207/243) psyche-centralized-testing::integration_tests connect_single_node
psyche-workspace-nextest-0.2.0>         PASS [   0.194s] (208/243) psyche-solana-tooling::mod suites::memnet_treasurer_create_claim::run
psyche-workspace-nextest-0.2.0>         PASS [   0.266s] (209/243) psyche-solana-tooling::mod suites::memnet_mining_pool_full_cycle::run
psyche-workspace-nextest-0.2.0>         PASS [   0.648s] (210/243) psyche-network serialized_distro::tests::test_roundtrip_distro_result_1bit
psyche-workspace-nextest-0.2.0>         PASS [   0.187s] (211/243) psyche-solana-tooling::mod suites::memnet_treasurer_create_update::run
psyche-workspace-nextest-0.2.0>         PASS [   1.122s] (212/243) psyche-network download::scheduler::tests::test_distro_retry_not_immediately_due
psyche-workspace-nextest-0.2.0>         PASS [   0.863s] (213/243) psyche-network serializable_tensor::tests::test_roundtrip_tensor2d
psyche-workspace-nextest-0.2.0>         PASS [   0.999s] (214/243) psyche-network serializable_tensor::tests::test_roundtrip_bool_tensor_manyd
psyche-workspace-nextest-0.2.0>         PASS [   0.695s] (215/243) psyche-network tcp::tests::test_challenge_rejects_invalid_signature
psyche-workspace-nextest-0.2.0>         PASS [   0.895s] (216/243) psyche-network serializable_tensor::tests::test_roundtrip_tensor1d
psyche-workspace-nextest-0.2.0>         PASS [   0.562s] (217/243) psyche-solana-tooling::mod suites::memnet_distributor_claim_vesting::run
psyche-workspace-nextest-0.2.0>         PASS [   0.711s] (218/243) psyche-network tcp::tests::test_tcp_handshake_uses_only_iroh_key
psyche-workspace-nextest-0.2.0>         PASS [   0.828s] (219/243) psyche-network tcp::tests::test_multiple_clients
psyche-workspace-nextest-0.2.0>         PASS [   1.002s] (220/243) psyche-network serializable_tensor::tests::test_roundtrip_tensor_manyd
psyche-workspace-nextest-0.2.0>         PASS [   0.690s] (221/243) psyche-solana-tooling::mod suites::memnet_distributor_many_claims::run
psyche-workspace-nextest-0.2.0>         PASS [   2.072s] (222/243) psyche-modeling distro::tests::test_artifical_distro_results_roundtrip
psyche-workspace-nextest-0.2.0>         PASS [   0.949s] (223/243) psyche-network tcp::tests::test_message_exchange_after_iroh_handshake
psyche-workspace-nextest-0.2.0>         PASS [   0.786s] (224/243) psyche-solana-tooling::mod suites::memnet_coordinator_full_round::run
psyche-workspace-nextest-0.2.0>         PASS [   0.671s] (225/243) psyche-solana-tooling::mod suites::memnet_treasurer_full_epoch::run
psyche-workspace-nextest-0.2.0>         PASS [   2.288s] (226/243) psyche-centralized-testing::integration_tests state_change_waiting_for_members_to_warmup
psyche-workspace-nextest-0.2.0>         PASS [   1.005s] (227/243) psyche-solana-tooling::mod suites::memnet_distributor_single_claim::run
psyche-workspace-nextest-0.2.0>         PASS [   2.589s] (228/243) psyche-centralized-testing::integration_tests state_change_shutdown_node_in_warmup
psyche-workspace-nextest-0.2.0>         PASS [   2.561s] (229/243) psyche-inference vllm::tests::test_list_engines
psyche-workspace-nextest-0.2.0>         PASS [   1.276s] (230/243) psyche-solana-tooling::mod suites::memnet_coordinator_rewards::run
psyche-workspace-nextest-0.2.0>         PASS [   3.120s] (231/243) psyche-centralized-testing::integration_tests connect_multiple_nodes
psyche-workspace-nextest-0.2.0>         PASS [   2.794s] (232/243) psyche-network router::tests::test_allowlist
psyche-workspace-nextest-0.2.0>         PASS [  16.651s] (233/243) psyche-centralized-testing::integration_tests state_change_waiting_for_members_to_round_witness
psyche-workspace-nextest-0.2.0>         PASS [  17.483s] (234/243) psyche-centralized-testing::integration_tests replace_node_and_complete_round
psyche-workspace-nextest-0.2.0>         PASS [  21.686s] (235/243) psyche-centralized-testing::integration_tests state_change_waiting_for_members_to_round_train
psyche-workspace-nextest-0.2.0>         PASS [  22.609s] (236/243) psyche-network test::test_retry_connection_mid_download
psyche-workspace-nextest-0.2.0>         PASS [  23.257s] (237/243) psyche-network test::test_retry_connection
psyche-workspace-nextest-0.2.0>         PASS [  24.727s] (238/243) psyche-centralized-testing::integration_tests shutdown_node_in_training_and_complete_round
psyche-workspace-nextest-0.2.0>         PASS [  31.212s] (239/243) psyche-centralized-testing::integration_tests validate_all_clients_participate_in_witness_bloom
psyche-workspace-nextest-0.2.0>         PASS [  42.649s] (240/243) psyche-centralized-testing::integration_tests finish_epoch
psyche-workspace-nextest-0.2.0>         PASS [  42.662s] (241/243) psyche-centralized-testing::integration_tests client_join_in_training
psyche-workspace-nextest-0.2.0>         SLOW [> 60.000s] (-------) psyche-centralized-testing::integration_tests client_join_in_training_and_get_model_using_p2p
psyche-workspace-nextest-0.2.0>         SLOW [> 60.000s] (-------) psyche-centralized-testing::integration_tests two_clients_join_in_training_and_get_model_using_p2p
psyche-workspace-nextest-0.2.0>         PASS [  82.201s] (242/243) psyche-centralized-testing::integration_tests client_join_in_training_and_get_model_using_p2p
psyche-workspace-nextest-0.2.0>         PASS [  82.941s] (243/243) psyche-centralized-testing::integration_tests two_clients_join_in_training_and_get_model_using_p2p
psyche-workspace-nextest-0.2.0> ------------
psyche-workspace-nextest-0.2.0>      Summary [  82.943s] 243 tests run: 243 passed (2 slow), 2 skipped
psyche-workspace-nextest-0.2.0> checkPhase completed in 3 minutes 9 seconds
psyche-workspace-nextest-0.2.0> Running phase: installPhase
psyche-workspace-nextest-0.2.0> linking previous artifacts /nix/store/8d858ylzfi37d47q12f6w2s0ywlfkysb-psyche-workspace-deps-0.2.0/target.tar.zst to /nix/store/zbhh2w5zi2ggrmags3anxypdicsk9fbi-psyche-workspace-nextest-0.2.0/target.tar.zst
psyche-workspace-nextest-0.2.0> /*stdin*\            : 28.79%   (   873 MiB =>    251 MiB, /nix/store/zbhh2w5zi2ggrmags3anxypdicsk9fbi-psyche-workspace-nextest-0.2.0/target.tar.zst) 
psyche-workspace-nextest-0.2.0> Running phase: fixupPhase
psyche-workspace-nextest-0.2.0> shrinking RPATHs of ELF executables and libraries in /nix/store/zbhh2w5zi2ggrmags3anxypdicsk9fbi-psyche-workspace-nextest-0.2.0
psyche-workspace-nextest-0.2.0> checking for references to /build/ in /nix/store/zbhh2w5zi2ggrmags3anxypdicsk9fbi-psyche-workspace-nextest-0.2.0...
psyche-workspace-nextest-0.2.0> patching script interpreter paths in /nix/store/zbhh2w5zi2ggrmags3anxypdicsk9fbi-psyche-workspace-nextest-0.2.0
psyche-workspace-nextest-0.2.0> shrinking RPATHs of ELF executables and libraries in /nix/store/zbhh2w5zi2ggrmags3anxypdicsk9fbi-psyche-workspace-nextest-0.2.0