GCS integration: run down signed urls - #615

psyche-workspace-nextest-0.2.0>         PASS [   2.494s] (203/222) psyche-modeling distro::tests::test_artifical_distro_results_roundtrip
psyche-workspace-nextest-0.2.0>         PASS [   0.852s] (204/222) psyche-solana-tooling::mod suites::memnet_treasurer_full_epoch::run
psyche-workspace-nextest-0.2.0>         PASS [   1.043s] (205/222) psyche-solana-tooling::mod suites::memnet_distributor_single_claim::run
psyche-workspace-nextest-0.2.0>         PASS [   2.735s] (206/222) psyche-centralized-testing::integration_tests state_change_waiting_for_members_to_warmup
psyche-workspace-nextest-0.2.0>         PASS [   2.799s] (207/222) psyche-inference vllm::tests::test_list_engines
psyche-workspace-nextest-0.2.0>         PASS [   1.313s] (208/222) psyche-solana-tooling::mod suites::memnet_coordinator_rewards::run
psyche-workspace-nextest-0.2.0>         PASS [   3.185s] (209/222) psyche-centralized-testing::integration_tests state_change_shutdown_node_in_warmup
psyche-workspace-nextest-0.2.0>         PASS [   3.416s] (210/222) psyche-centralized-testing::integration_tests connect_multiple_nodes
psyche-workspace-nextest-0.2.0>         PASS [  17.227s] (211/222) psyche-centralized-testing::integration_tests state_change_waiting_for_members_to_round_witness
psyche-workspace-nextest-0.2.0>         PASS [  19.137s] (212/222) psyche-centralized-testing::integration_tests replace_node_and_complete_round
psyche-workspace-nextest-0.2.0>         PASS [  19.397s] (213/222) psyche-network test::test_retry_connection
psyche-workspace-nextest-0.2.0>         PASS [  22.118s] (214/222) psyche-centralized-testing::integration_tests state_change_waiting_for_members_to_round_train
psyche-workspace-nextest-0.2.0>         PASS [  22.275s] (215/222) psyche-centralized-testing::integration_tests shutdown_node_in_training_and_complete_round
psyche-workspace-nextest-0.2.0>         PASS [  21.754s] (216/222) psyche-network test::test_retry_connection_mid_download
psyche-workspace-nextest-0.2.0>         PASS [  32.161s] (217/222) psyche-centralized-testing::integration_tests validate_all_clients_participate_in_witness_bloom
psyche-workspace-nextest-0.2.0>         PASS [  39.624s] (218/222) psyche-centralized-testing::integration_tests client_join_in_training
psyche-workspace-nextest-0.2.0>         PASS [  41.673s] (219/222) psyche-centralized-testing::integration_tests finish_epoch
psyche-workspace-nextest-0.2.0>         SLOW [> 60.000s] (-------) psyche-centralized-testing::integration_tests client_join_in_training_and_get_model_using_p2p
psyche-workspace-nextest-0.2.0>         SLOW [> 60.000s] (-------) psyche-centralized-testing::integration_tests two_clients_join_in_training_and_get_model_using_p2p
psyche-workspace-nextest-0.2.0>         SLOW [> 60.000s] (-------) psyche-network router::tests::test_allowlist
psyche-workspace-nextest-0.2.0>         PASS [  75.933s] (220/222) psyche-centralized-testing::integration_tests client_join_in_training_and_get_model_using_p2p
psyche-workspace-nextest-0.2.0>         PASS [  82.419s] (221/222) psyche-centralized-testing::integration_tests two_clients_join_in_training_and_get_model_using_p2p
psyche-workspace-nextest-0.2.0>         SLOW [>120.000s] (-------) psyche-network router::tests::test_allowlist
psyche-workspace-nextest-0.2.0>  TERMINATING [>180.000s] (-------) psyche-network router::tests::test_allowlist
psyche-workspace-nextest-0.2.0>      TIMEOUT [ 180.034s] (222/222) psyche-network router::tests::test_allowlist
psyche-workspace-nextest-0.2.0>   stdout ---
psyche-workspace-nextest-0.2.0> 
psyche-workspace-nextest-0.2.0>     running 1 test
psyche-workspace-nextest-0.2.0>     subscribing 0 (e10ec7905b15add24af906ecf1e0feb337c4051c49147f64dcaed926d3ab63f9) to topic..
psyche-workspace-nextest-0.2.0>     subscribing 1 (8e09ba140e416e137f979dc6eb7da87e2027b3d3c1a92cd42f3ec60ffa286710) to topic..
psyche-workspace-nextest-0.2.0>     subscribing 2 (3b310dffb78054176e749e91c6b60f2571be5014e8a936edaa748bdf7d3194f4) to topic..
psyche-workspace-nextest-0.2.0>     subscribing 3 (5692d1472c78ea6d63abe2d5dd11cbd2b81a9e117f4581ff858654d5236e3d60) to topic..
psyche-workspace-nextest-0.2.0>     waiting for gossip connections..
psyche-workspace-nextest-0.2.0>     waiting for 0 (e10ec7905b15add24af906ecf1e0feb337c4051c49147f64dcaed926d3ab63f9) to connect to 2 neighbors..
psyche-workspace-nextest-0.2.0>     waiting for 1 (8e09ba140e416e137f979dc6eb7da87e2027b3d3c1a92cd42f3ec60ffa286710) to connect to 2 neighbors..
psyche-workspace-nextest-0.2.0>     waiting for 2 (3b310dffb78054176e749e91c6b60f2571be5014e8a936edaa748bdf7d3194f4) to connect to 2 neighbors..
psyche-workspace-nextest-0.2.0>     gossip connections 2 ready (2 neighbors)
psyche-workspace-nextest-0.2.0>     gossip connections 0 ready (2 neighbors)
psyche-workspace-nextest-0.2.0>     gossip connections 1 ready (2 neighbors)
psyche-workspace-nextest-0.2.0>     all gossip connections set up.
psyche-workspace-nextest-0.2.0>     broadcasting Message from client 0
psyche-workspace-nextest-0.2.0>     broadcasting Message from client 1
psyche-workspace-nextest-0.2.0>     broadcasting Message from client 2
psyche-workspace-nextest-0.2.0>     broadcasting Message from client 3
psyche-workspace-nextest-0.2.0>     checking for recv'd messages..
psyche-workspace-nextest-0.2.0>     Router 2 received all messages
psyche-workspace-nextest-0.2.0>     Router 0 received all messages
psyche-workspace-nextest-0.2.0>     Router 1 received all messages
psyche-workspace-nextest-0.2.0>     test router::tests::test_allowlist has been running for over 60 seconds
psyche-workspace-nextest-0.2.0>   stderr ---
psyche-workspace-nextest-0.2.0>     2026-03-17T18:39:17.529165Z  WARN gossip{me=e10ec7905b}:connect{me=e10ec7905b alpn="/iroh-gossip/1" remote=3b310dffb7}: iroh_quinn_proto::connection: remote server configuration might cause nat traversal issues max_local_addresses=12 remote_cid_limit=5
psyche-workspace-nextest-0.2.0>     2026-03-17T18:39:17.535080Z  WARN gossip{me=5692d1472c}:connect{me=5692d1472c alpn="/iroh-gossip/1" remote=3b310dffb7}: iroh_quinn_proto::connection: remote server configuration might cause nat traversal issues max_local_addresses=12 remote_cid_limit=5
psyche-workspace-nextest-0.2.0>     2026-03-17T18:39:17.535078Z  WARN gossip{me=3b310dffb7}:connect{me=3b310dffb7 alpn="/iroh-gossip/1" remote=e10ec7905b}: iroh_quinn_proto::connection: remote server configuration might cause nat traversal issues max_local_addresses=12 remote_cid_limit=5
psyche-workspace-nextest-0.2.0>     2026-03-17T18:39:17.536153Z  WARN gossip{me=5692d1472c}:connect{me=5692d1472c alpn="/iroh-gossip/1" remote=8e09ba140e}: iroh_quinn_proto::connection: remote server configuration might cause nat traversal issues max_local_addresses=12 remote_cid_limit=5
psyche-workspace-nextest-0.2.0>     2026-03-17T18:39:17.535589Z  WARN gossip{me=5692d1472c}:connect{me=5692d1472c alpn="/iroh-gossip/1" remote=e10ec7905b}: iroh_quinn_proto::connection: remote server configuration might cause nat traversal issues max_local_addresses=12 remote_cid_limit=5
psyche-workspace-nextest-0.2.0>     2026-03-17T18:39:17.546093Z  WARN router.accept{me=8e09ba140e alpn="/iroh-gossip/1"}: iroh::protocol: Accepting incoming connection ended with error: Connection was rejected locally
psyche-workspace-nextest-0.2.0>     2026-03-17T18:39:17.546303Z  WARN gossip{me=8e09ba140e}:connect{me=8e09ba140e alpn="/iroh-gossip/1" remote=e10ec7905b}: iroh_quinn_proto::connection: remote server configuration might cause nat traversal issues max_local_addresses=12 remote_cid_limit=5
psyche-workspace-nextest-0.2.0>     2026-03-17T18:39:17.546963Z  WARN gossip{me=8e09ba140e}:connect{me=8e09ba140e alpn="/iroh-gossip/1" remote=3b310dffb7}: iroh_quinn_proto::connection: remote server configuration might cause nat traversal issues max_local_addresses=12 remote_cid_limit=5
psyche-workspace-nextest-0.2.0>     2026-03-17T18:39:17.556092Z  WARN router.accept{me=3b310dffb7 alpn="/iroh-gossip/1"}: iroh::protocol: Accepting incoming connection ended with error: Connection was rejected locally
psyche-workspace-nextest-0.2.0>     2026-03-17T18:39:17.560443Z  WARN router.accept{me=e10ec7905b alpn="/iroh-gossip/1"}: iroh::protocol: Accepting incoming connection ended with error: Connection was rejected locally
psyche-workspace-nextest-0.2.0>     2026-03-17T18:39:17.560799Z  WARN gossip{me=3b310dffb7}:connect{me=3b310dffb7 alpn="/iroh-gossip/1" remote=8e09ba140e}: iroh_quinn_proto::connection: remote server configuration might cause nat traversal issues max_local_addresses=12 remote_cid_limit=5
psyche-workspace-nextest-0.2.0>     2026-03-17T18:39:17.564170Z  WARN gossip{me=e10ec7905b}:connect{me=e10ec7905b alpn="/iroh-gossip/1" remote=8e09ba140e}: iroh_quinn_proto::connection: remote server configuration might cause nat traversal issues max_local_addresses=12 remote_cid_limit=5
psyche-workspace-nextest-0.2.0>     2026-03-17T18:39:24.070850Z  WARN router.accept{me=3b310dffb7 alpn="/iroh-gossip/1" remote=8e09ba140e}: iroh_quinn_proto::connection: failed closing path err=LastOpenPath
psyche-workspace-nextest-0.2.0>     2026-03-17T18:39:24.075104Z  WARN router.accept{me=8e09ba140e alpn="/iroh-gossip/1" remote=e10ec7905b}: iroh_quinn_proto::connection: failed closing path err=LastOpenPath
psyche-workspace-nextest-0.2.0> 
psyche-workspace-nextest-0.2.0>     (test timed out)
psyche-workspace-nextest-0.2.0> 
psyche-workspace-nextest-0.2.0> ------------
psyche-workspace-nextest-0.2.0>      Summary [ 180.808s] 222 tests run: 221 passed (2 slow), 1 timed out, 2 skipped
psyche-workspace-nextest-0.2.0>      TIMEOUT [ 180.034s] (222/222) psyche-network router::tests::test_allowlist
psyche-workspace-nextest-0.2.0> error: test run failed
error: build of '/nix/store/qd9lbg1pg4bsydjmb3kmwbq37f3z07g9-psyche-workspace-nextest-0.2.0.drv' on 'ssh-ng://nix-ssh@garnix5' failed: builder for '/nix/store/qd9lbg1pg4bsydjmb3kmwbq37f3z07g9-psyche-workspace-nextest-0.2.0.drv' failed with exit code 100;
       last 25 log lines:
       >     Router 0 received all messages
       >     Router 1 received all messages
       >     test router::tests::test_allowlist has been running for over 60 seconds
       >   stderr ---
       >     2026-03-17T18:39:17.529165Z  WARN gossip{me=e10ec7905b}:connect{me=e10ec7905b alpn="/iroh-gossip/1" remote=3b310dffb7}: iroh_quinn_proto::connection: remote server configuration might cause nat traversal issues max_local_addresses=12 remote_cid_limit=5
       >     2026-03-17T18:39:17.535080Z  WARN gossip{me=5692d1472c}:connect{me=5692d1472c alpn="/iroh-gossip/1" remote=3b310dffb7}: iroh_quinn_proto::connection: remote server configuration might cause nat traversal issues max_local_addresses=12 remote_cid_limit=5
       >     2026-03-17T18:39:17.535078Z  WARN gossip{me=3b310dffb7}:connect{me=3b310dffb7 alpn="/iroh-gossip/1" remote=e10ec7905b}: iroh_quinn_proto::connection: remote server configuration might cause nat traversal issues max_local_addresses=12 remote_cid_limit=5
       >     2026-03-17T18:39:17.536153Z  WARN gossip{me=5692d1472c}:connect{me=5692d1472c alpn="/iroh-gossip/1" remote=8e09ba140e}: iroh_quinn_proto::connection: remote server configuration might cause nat traversal issues max_local_addresses=12 remote_cid_limit=5
       >     2026-03-17T18:39:17.535589Z  WARN gossip{me=5692d1472c}:connect{me=5692d1472c alpn="/iroh-gossip/1" remote=e10ec7905b}: iroh_quinn_proto::connection: remote server configuration might cause nat traversal issues max_local_addresses=12 remote_cid_limit=5
       >     2026-03-17T18:39:17.546093Z  WARN router.accept{me=8e09ba140e alpn="/iroh-gossip/1"}: iroh::protocol: Accepting incoming connection ended with error: Connection was rejected locally
       >     2026-03-17T18:39:17.546303Z  WARN gossip{me=8e09ba140e}:connect{me=8e09ba140e alpn="/iroh-gossip/1" remote=e10ec7905b}: iroh_quinn_proto::connection: remote server configuration might cause nat traversal issues max_local_addresses=12 remote_cid_limit=5
       >     2026-03-17T18:39:17.546963Z  WARN gossip{me=8e09ba140e}:connect{me=8e09ba140e alpn="/iroh-gossip/1" remote=3b310dffb7}: iroh_quinn_proto::connection: remote server configuration might cause nat traversal issues max_local_addresses=12 remote_cid_limit=5
       >     2026-03-17T18:39:17.556092Z  WARN router.accept{me=3b310dffb7 alpn="/iroh-gossip/1"}: iroh::protocol: Accepting incoming connection ended with error: Connection was rejected locally
       >     2026-03-17T18:39:17.560443Z  WARN router.accept{me=e10ec7905b alpn="/iroh-gossip/1"}: iroh::protocol: Accepting incoming connection ended with error: Connection was rejected locally
       >     2026-03-17T18:39:17.560799Z  WARN gossip{me=3b310dffb7}:connect{me=3b310dffb7 alpn="/iroh-gossip/1" remote=8e09ba140e}: iroh_quinn_proto::connection: remote server configuration might cause nat traversal issues max_local_addresses=12 remote_cid_limit=5
       >     2026-03-17T18:39:17.564170Z  WARN gossip{me=e10ec7905b}:connect{me=e10ec7905b alpn="/iroh-gossip/1" remote=8e09ba140e}: iroh_quinn_proto::connection: remote server configuration might cause nat traversal issues max_local_addresses=12 remote_cid_limit=5
       >     2026-03-17T18:39:24.070850Z  WARN router.accept{me=3b310dffb7 alpn="/iroh-gossip/1" remote=8e09ba140e}: iroh_quinn_proto::connection: failed closing path err=LastOpenPath
       >     2026-03-17T18:39:24.075104Z  WARN router.accept{me=8e09ba140e alpn="/iroh-gossip/1" remote=e10ec7905b}: iroh_quinn_proto::connection: failed closing path err=LastOpenPath
       >
       >     (test timed out)
       >
       > ------------
       >      Summary [ 180.808s] 222 tests run: 221 passed (2 slow), 1 timed out, 2 skipped
       >      TIMEOUT [ 180.034s] (222/222) psyche-network router::tests::test_allowlist
       > error: test run failed
       For full logs, run 'nix log /nix/store/qd9lbg1pg4bsydjmb3kmwbq37f3z07g9-psyche-workspace-nextest-0.2.0.drv'.
error: builder for '/nix/store/qd9lbg1pg4bsydjmb3kmwbq37f3z07g9-psyche-workspace-nextest-0.2.0.drv' failed with exit code 1