Skip to content

Add parallelism auto-detection from GCS via signed URLs - #620

Draft
pefontana wants to merge 2 commits into
run-down-signed-urlsfrom
gcs-parallelism_data
Draft

pefontana wants to merge 2 commits into
run-down-signed-urlsfrom
gcs-parallelism_data

Simplify parallelism auto-detection to GCS-only and add docs

380a6e4
Select commit
Loading
Failed to load commit list.
Garnix CI / check workspace-test-ci [x86_64-linux] succeeded Mar 6, 2026 in 9m 27s

check workspace-test-ci [x86_64-linux]

check workspace-test-ci [x86_64-linux] succeeded

Details

Last 100 lines of logs:

psyche-workspace-nextest-0.2.0>         PASS [   0.660s] (136/222) psyche-data-provider::weighted test_weighted_data_provider_consistency
psyche-workspace-nextest-0.2.0>         PASS [   0.644s] (137/222) psyche-modeling distro::tests::test_encode_1d
psyche-workspace-nextest-0.2.0>         PASS [   0.676s] (138/222) psyche-modeling distro::tests::test_decode_1d
psyche-workspace-nextest-0.2.0>         PASS [   0.678s] (139/222) psyche-modeling distro::tests::test_decode_2d
psyche-workspace-nextest-0.2.0>         PASS [   0.691s] (140/222) psyche-data-provider::preprocessed loads_hermes3_subset
psyche-workspace-nextest-0.2.0>         PASS [   0.691s] (141/222) psyche-data-provider::weighted test_weighted_data_provider_exhausts_small_dataset_before_repeat
psyche-workspace-nextest-0.2.0>         PASS [   0.745s] (142/222) psyche-modeling distro::tests::test_compress_1d
psyche-workspace-nextest-0.2.0>         PASS [   0.641s] (143/222) psyche-modeling distro::tests::test_get_divisors
psyche-workspace-nextest-0.2.0>         PASS [   0.579s] (144/222) psyche-modeling distro::tests::test_get_smaller_split
psyche-workspace-nextest-0.2.0>         PASS [   0.798s] (145/222) psyche-modeling device_utils::tests::test_parse_device
psyche-workspace-nextest-0.2.0>         PASS [   0.808s] (146/222) psyche-data-provider::local loads_dolma_subset
psyche-workspace-nextest-0.2.0>         PASS [   0.594s] (147/222) psyche-modeling distro::tests::test_get_prime_divisors
psyche-workspace-nextest-0.2.0>         PASS [   0.712s] (148/222) psyche-modeling distro::tests::test_encode_2d
psyche-workspace-nextest-0.2.0>         PASS [   0.825s] (149/222) psyche-data-provider::local loads_fineweb_subset
psyche-workspace-nextest-0.2.0>         PASS [   0.839s] (150/222) psyche-data-provider::weighted test_weighted_data_provider_multi_batch
psyche-workspace-nextest-0.2.0>         PASS [   1.045s] (151/222) psyche-data-provider weighted::http::tests::test_http_multi_data_provider
psyche-workspace-nextest-0.2.0>         PASS [   0.658s] (152/222) psyche-modeling distro::tests::test_large_numbers
psyche-workspace-nextest-0.2.0>         PASS [   0.010s] (153/222) psyche-solana-authorizer test_id
psyche-workspace-nextest-0.2.0>         PASS [   1.253s] (154/222) psyche-data-provider::http test_http_data_provider
psyche-workspace-nextest-0.2.0>         PASS [   0.658s] (155/222) psyche-network download::scheduler::tests::test_distro_retry_not_immediately_due
psyche-workspace-nextest-0.2.0>         PASS [   0.648s] (156/222) psyche-network serializable_tensor::tests::test_roundtrip_bool_tensor2d
psyche-workspace-nextest-0.2.0>         PASS [   0.762s] (157/222) psyche-network download::scheduler::tests::test_distro_retries_returned_and_removed
psyche-workspace-nextest-0.2.0>         PASS [   0.016s] (158/222) psyche-solana-coordinator test_id
psyche-workspace-nextest-0.2.0>         PASS [   0.028s] (159/222) psyche-solana-distributor test_id
psyche-workspace-nextest-0.2.0>         PASS [   0.825s] (160/222) psyche-network download::scheduler::tests::test_config_retries_dont_consume_capacity
psyche-workspace-nextest-0.2.0>         PASS [   0.009s] (161/222) psyche-solana-mining-pool test_id
psyche-workspace-nextest-0.2.0>         PASS [   0.803s] (162/222) psyche-network download::scheduler::tests::test_parameter_retries_are_immediate
psyche-workspace-nextest-0.2.0>         PASS [   0.021s] (163/222) psyche-solana-tooling::mod suites::memnet_coordinator_data_layout::run
psyche-workspace-nextest-0.2.0>         PASS [   0.772s] (164/222) psyche-network serializable_tensor::tests::test_roundtrip_bool_tensor1d
psyche-workspace-nextest-0.2.0>         PASS [   0.927s] (165/222) psyche-modeling distro::tests::test_idct
psyche-workspace-nextest-0.2.0>         PASS [   0.672s] (166/222) psyche-network serialized_distro::tests::test_roundtrip_distro_result_1bit
psyche-workspace-nextest-0.2.0>         PASS [   0.871s] (167/222) psyche-network download::scheduler::tests::test_remove_retry
psyche-workspace-nextest-0.2.0>         PASS [   0.717s] (168/222) psyche-network serializable_tensor::tests::test_roundtrip_tensor1d
psyche-workspace-nextest-0.2.0>         PASS [   0.842s] (169/222) psyche-network router::tests::test_shutdown
psyche-workspace-nextest-0.2.0>         PASS [   0.949s] (170/222) psyche-network download::scheduler::tests::test_capacity_grants_up_to_max
psyche-workspace-nextest-0.2.0>         PASS [   0.949s] (171/222) psyche-network download::scheduler::tests::test_distro_retries_dont_consume_capacity
psyche-workspace-nextest-0.2.0>         PASS [   1.562s] (172/222) psyche-data-provider::http test_http_data_provider_shuffled
psyche-workspace-nextest-0.2.0>         PASS [   1.583s] (173/222) psyche-modeling distro::tests::test_1bit_matches_non_quant
psyche-workspace-nextest-0.2.0>         PASS [   0.799s] (174/222) psyche-network serializable_tensor::tests::test_roundtrip_bool_tensor_single_element
psyche-workspace-nextest-0.2.0>         PASS [   0.791s] (175/222) psyche-network serializable_tensor::tests::test_roundtrip_tensor2d
psyche-workspace-nextest-0.2.0>         PASS [   0.006s] (176/222) psyche-solana-treasurer test_id
psyche-workspace-nextest-0.2.0>         PASS [   0.981s] (177/222) psyche-network download::scheduler::tests::test_model_sharing_retry_respects_capacity
psyche-workspace-nextest-0.2.0>         PASS [   0.904s] (178/222) psyche-network serializable_tensor::tests::test_roundtrip_bool_tensor_non_divisible_by_8
psyche-workspace-nextest-0.2.0>         PASS [   1.043s] (179/222) psyche-network download::scheduler::tests::test_release_unblocks_waiter
psyche-workspace-nextest-0.2.0>         PASS [   0.280s] (180/222) psyche-solana-tooling::mod suites::memnet_authorizer_full_cycle::run
psyche-workspace-nextest-0.2.0>         PASS [   1.131s] (181/222) psyche-modeling distro::tests::test_signed_vals_reconstructs_original_sign
psyche-workspace-nextest-0.2.0>         PASS [   0.279s] (182/222) psyche-solana-tooling::mod suites::memnet_coordinator_init_free::run
psyche-workspace-nextest-0.2.0>         PASS [   0.234s] (183/222) psyche-solana-tooling::mod suites::memnet_mining_pool_cap_deposits::run
psyche-workspace-nextest-0.2.0>         PASS [   0.233s] (184/222) psyche-solana-tooling::mod suites::memnet_mining_pool_live_freeze::run
psyche-workspace-nextest-0.2.0>         PASS [   1.172s] (185/222) psyche-network download::scheduler::tests::test_distro_max_retries_exceeded
psyche-workspace-nextest-0.2.0>         PASS [   1.115s] (186/222) psyche-network download::scheduler::tests::test_waiters_are_served_fifo
psyche-workspace-nextest-0.2.0>         PASS [   0.973s] (187/222) psyche-network serializable_tensor::tests::test_roundtrip_tensor_manyd
psyche-workspace-nextest-0.2.0>         PASS [   1.168s] (188/222) psyche-network download::scheduler::tests::test_model_sharing_never_exceeds_max_retries
psyche-workspace-nextest-0.2.0>         PASS [   0.246s] (189/222) psyche-solana-tooling::mod suites::memnet_treasurer_create_claim::run
psyche-workspace-nextest-0.2.0>         PASS [   0.282s] (190/222) psyche-solana-tooling::mod suites::memnet_treasurer_create_update::run
psyche-workspace-nextest-0.2.0>         PASS [   1.105s] (191/222) psyche-network serializable_tensor::tests::test_roundtrip_bool_tensor_manyd
psyche-workspace-nextest-0.2.0>         PASS [   0.331s] (192/222) psyche-solana-tooling::mod suites::memnet_mining_pool_full_cycle::run
psyche-workspace-nextest-0.2.0>         PASS [   1.242s] (193/222) psyche-network download::scheduler::tests::test_wait_for_capacity_errors_on_actor_shutdown
psyche-workspace-nextest-0.2.0>         PASS [   1.101s] (194/222) psyche-network serializable_tensor::tests::test_roundtrip_bool_tensor_unusual_shape
psyche-workspace-nextest-0.2.0>         PASS [   0.633s] (195/222) psyche-solana-client::bin/psyche-solana-client network_identity::tests::test_network_identity_invalid_challenge
psyche-workspace-nextest-0.2.0>         PASS [   0.608s] (196/222) psyche-solana-client::bin/psyche-solana-client network_identity::tests::test_network_identity_roundtrip
psyche-workspace-nextest-0.2.0>         PASS [   0.738s] (197/222) psyche-solana-client::bin/psyche-solana-client network_identity::tests::test_network_identity_hash
psyche-workspace-nextest-0.2.0>         PASS [   2.005s] (198/222) psyche-centralized-testing::integration_tests connect_single_node
psyche-workspace-nextest-0.2.0>         PASS [   0.755s] (199/222) psyche-solana-client::bin/psyche-solana-client network_identity::tests::test_network_identity_display
psyche-workspace-nextest-0.2.0>         PASS [   0.563s] (200/222) psyche-solana-tooling::mod suites::memnet_distributor_claim_vesting::run
psyche-workspace-nextest-0.2.0>         PASS [   0.611s] (201/222) psyche-solana-tooling::mod suites::memnet_distributor_single_claim::run
psyche-workspace-nextest-0.2.0>         PASS [   0.558s] (202/222) psyche-solana-tooling::mod suites::memnet_treasurer_full_epoch::run
psyche-workspace-nextest-0.2.0>         PASS [   0.782s] (203/222) psyche-solana-tooling::mod suites::memnet_coordinator_full_round::run
psyche-workspace-nextest-0.2.0>         PASS [   0.755s] (204/222) psyche-solana-tooling::mod suites::memnet_distributor_many_claims::run
psyche-workspace-nextest-0.2.0>         PASS [   2.546s] (205/222) psyche-modeling distro::tests::test_artifical_distro_results_roundtrip
psyche-workspace-nextest-0.2.0>         PASS [   1.262s] (206/222) psyche-solana-tooling::mod suites::memnet_coordinator_rewards::run
psyche-workspace-nextest-0.2.0>         PASS [   2.740s] (207/222) psyche-centralized-testing::integration_tests state_change_waiting_for_members_to_warmup
psyche-workspace-nextest-0.2.0>         PASS [   2.805s] (208/222) psyche-inference vllm::tests::test_list_engines
psyche-workspace-nextest-0.2.0>         PASS [   3.291s] (209/222) psyche-centralized-testing::integration_tests state_change_shutdown_node_in_warmup
psyche-workspace-nextest-0.2.0>         PASS [   3.387s] (210/222) psyche-centralized-testing::integration_tests connect_multiple_nodes
psyche-workspace-nextest-0.2.0>         PASS [   4.422s] (211/222) psyche-network router::tests::test_allowlist
psyche-workspace-nextest-0.2.0>         PASS [  16.745s] (212/222) psyche-centralized-testing::integration_tests state_change_waiting_for_members_to_round_witness
psyche-workspace-nextest-0.2.0>         PASS [  19.087s] (213/222) psyche-centralized-testing::integration_tests replace_node_and_complete_round
psyche-workspace-nextest-0.2.0>         PASS [  19.673s] (214/222) psyche-network test::test_retry_connection
psyche-workspace-nextest-0.2.0>         PASS [  21.152s] (215/222) psyche-centralized-testing::integration_tests shutdown_node_in_training_and_complete_round
psyche-workspace-nextest-0.2.0>         PASS [  21.718s] (216/222) psyche-centralized-testing::integration_tests state_change_waiting_for_members_to_round_train
psyche-workspace-nextest-0.2.0>         PASS [  23.127s] (217/222) psyche-network test::test_retry_connection_mid_download
psyche-workspace-nextest-0.2.0>         PASS [  31.144s] (218/222) psyche-centralized-testing::integration_tests validate_all_clients_participate_in_witness_bloom
psyche-workspace-nextest-0.2.0>         PASS [  38.697s] (219/222) psyche-centralized-testing::integration_tests client_join_in_training
psyche-workspace-nextest-0.2.0>         PASS [  46.442s] (220/222) psyche-centralized-testing::integration_tests finish_epoch
psyche-workspace-nextest-0.2.0>         SLOW [> 60.000s] (-------) psyche-centralized-testing::integration_tests client_join_in_training_and_get_model_using_p2p
psyche-workspace-nextest-0.2.0>         SLOW [> 60.000s] (-------) psyche-centralized-testing::integration_tests two_clients_join_in_training_and_get_model_using_p2p
psyche-workspace-nextest-0.2.0>         PASS [  74.057s] (221/222) psyche-centralized-testing::integration_tests client_join_in_training_and_get_model_using_p2p
psyche-workspace-nextest-0.2.0>         PASS [  79.808s] (222/222) psyche-centralized-testing::integration_tests two_clients_join_in_training_and_get_model_using_p2p
psyche-workspace-nextest-0.2.0> ------------
psyche-workspace-nextest-0.2.0>      Summary [  79.809s] 222 tests run: 222 passed (2 slow), 2 skipped
psyche-workspace-nextest-0.2.0> checkPhase completed in 3 minutes 21 seconds
psyche-workspace-nextest-0.2.0> Running phase: installPhase
psyche-workspace-nextest-0.2.0> linking previous artifacts /nix/store/nf4lg0yn4ncakk7qkxcrwai8nxs2agnj-psyche-workspace-deps-0.2.0/target.tar.zst to /nix/store/54q293hk5r2rcl8hds6j29mqvh06z3lc-psyche-workspace-nextest-0.2.0/target.tar.zst
psyche-workspace-nextest-0.2.0> /*stdin*\            : 30.04%   (   848 MiB =>    255 MiB, /nix/store/54q293hk5r2rcl8hds6j29mqvh06z3lc-psyche-workspace-nextest-0.2.0/target.tar.zst) 
psyche-workspace-nextest-0.2.0> Running phase: fixupPhase
psyche-workspace-nextest-0.2.0> shrinking RPATHs of ELF executables and libraries in /nix/store/54q293hk5r2rcl8hds6j29mqvh06z3lc-psyche-workspace-nextest-0.2.0
psyche-workspace-nextest-0.2.0> checking for references to /build/ in /nix/store/54q293hk5r2rcl8hds6j29mqvh06z3lc-psyche-workspace-nextest-0.2.0...
psyche-workspace-nextest-0.2.0> patching script interpreter paths in /nix/store/54q293hk5r2rcl8hds6j29mqvh06z3lc-psyche-workspace-nextest-0.2.0
psyche-workspace-nextest-0.2.0> shrinking RPATHs of ELF executables and libraries in /nix/store/54q293hk5r2rcl8hds6j29mqvh06z3lc-psyche-workspace-nextest-0.2.0