Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
217 commits
Select commit Hold shift + click to select a range
0836df8
Make cooldown opportunistic
IAvecilla Dec 23, 2025
80550e6
Fix some things
IAvecilla Dec 23, 2025
be24bda
Fix compilation
IAvecilla Dec 23, 2025
20ac8a2
End epoch after checkpointer
IAvecilla Dec 23, 2025
514c993
Make hub_repo checkpoint mandatory
IAvecilla Dec 23, 2025
1679fcc
wip
pefontana Dec 28, 2025
472d2d6
crendential wip
pefontana Dec 28, 2025
3553f7d
Use client as checkpointer and pick random
IAvecilla Jan 5, 2026
fa3da9c
Merge remote-tracking branch 'origin/main' into gcs-2
pefontana Jan 5, 2026
00b97f1
Reuse opportunistic logic
IAvecilla Jan 5, 2026
0566a4a
Add anyhow to cargo lock
IAvecilla Jan 5, 2026
203d0c4
Fix opportunistic cooldown message
IAvecilla Jan 6, 2026
e8b1dd8
Add random index for client
IAvecilla Jan 6, 2026
0b6e9f5
Remove rand and get checkpointers at cooldown state
IAvecilla Jan 6, 2026
ff4ea2d
implement Checkpoint::Gcs(gcs_repo)
pefontana Jan 6, 2026
5ea32b1
light-config-gcs.toml
pefontana Jan 6, 2026
f8b193a
can_join command for gcs
pefontana Jan 6, 2026
c1f0e1f
gcs centralized version
pefontana Jan 7, 2026
c001fa5
Rework on cooldown to select pseudo random
IAvecilla Jan 7, 2026
6ddfe52
remove fn download_model_from_gcs_sync
pefontana Jan 7, 2026
c0aa10f
Revert "remove fn download_model_from_gcs_sync"
pefontana Jan 7, 2026
3b707ac
handle errors
pefontana Jan 7, 2026
12af6c3
Merge branch 'main' into gcs-2
pefontana Jan 7, 2026
ef3b468
Fix clippy
IAvecilla Jan 7, 2026
4b4a96d
remove progress_bar
pefontana Jan 7, 2026
6162937
use default cache dir
pefontana Jan 7, 2026
1bf03ab
refactor loop
pefontana Jan 7, 2026
8d8c3f4
Add support for multiple checkpointers
IAvecilla Jan 8, 2026
410c192
Fix lint
IAvecilla Jan 8, 2026
3ee7cb6
Update nano config
IAvecilla Jan 8, 2026
2b07c7d
Remove send_checkpoint function from backend
IAvecilla Jan 8, 2026
dcebbde
Merge branch 'main' into gcs-2
pefontana Jan 8, 2026
39fd70e
rm custom PSYCHE_CACHE_DIR
pefontana Jan 8, 2026
7fb8add
Reduce total amount of checkpointers
IAvecilla Jan 8, 2026
712f7cb
GcsRepo prefix: Option<FixedString<{ SOLANA_MAX_STRING_LEN }>>
pefontana Jan 8, 2026
e7421ed
Remove check for permissions on hf repo
IAvecilla Jan 8, 2026
a215881
Merge branch 'main' into cooldown-rework
IAvecilla Jan 8, 2026
a66f089
Fix compilation error after merge
IAvecilla Jan 8, 2026
5d99bf0
Use convert function only on python trainer
IAvecilla Jan 8, 2026
2f77f46
Add conditional import for python feature
IAvecilla Jan 8, 2026
284ae5d
Fix decentralized integration test config and entrypoints
IAvecilla Jan 8, 2026
2296e3f
implement P2PGCS checkpoint
pefontana Jan 8, 2026
d066a87
Merge branch 'main' into gcs-2
pefontana Jan 8, 2026
d1324e0
Add tracing to evaluate crate
pefontana Jan 8, 2026
85ed271
Merge branch 'main' into gcs-2
pefontana Jan 9, 2026
ba70ece
fix bug and remove debug prints
Jan 9, 2026
f82e387
remove unnecesary code
Jan 9, 2026
10e81f3
Add GCS checkpoint variant
IAvecilla Jan 9, 2026
f43c18e
Merge branch 'main' into gcs-2
IAvecilla Jan 12, 2026
0cb4ba0
Merge branch 'gcs-2' into gcs-upload-model
IAvecilla Jan 12, 2026
9fcf808
polish nits on coordinator code
Jan 12, 2026
9209a86
Merge branch 'main' into cooldown-rework
Jan 12, 2026
e893c7c
Fix convert call
IAvecilla Jan 12, 2026
df99802
Refactor on model download and upload
IAvecilla Jan 12, 2026
a41c457
Fix import with python feature
IAvecilla Jan 12, 2026
b19bc90
Remove vllm from nix
IAvecilla Jan 12, 2026
f02fd59
Merge branch 'gcs-upload-model' into cooldown-rework
IAvecilla Jan 12, 2026
9daad93
Add cancellation process after one client checkpoints
IAvecilla Jan 12, 2026
4511ae4
Fix cancellation for upload task
IAvecilla Jan 13, 2026
64ace6f
General refactor and cleanup for new checkpointers logic
IAvecilla Jan 13, 2026
d19df41
Remove comments
IAvecilla Jan 13, 2026
86ba846
Remove hub-repo and gcs-bucket from train args
IAvecilla Jan 14, 2026
607335c
Merge branch 'main' into gcs-2
pefontana Jan 14, 2026
4482623
Fix prefix
IAvecilla Jan 14, 2026
5ea5564
Fix tcp example
IAvecilla Jan 14, 2026
8da9c66
Calculate checkpointers instead of adding new config
IAvecilla Jan 14, 2026
037cc08
Model checkpoint upload with GCS (#476)
IAvecilla Jan 14, 2026
8d6749b
Merge branch 'main' into gcs-2
IAvecilla Jan 14, 2026
6f7aaf2
fixing centralized tests
IAvecilla Jan 14, 2026
860f50d
Merge branch 'gcs-2' into cooldown-rework
IAvecilla Jan 14, 2026
3471706
Fix tcp example compilation
IAvecilla Jan 14, 2026
e8a4e50
Fix centralized tests avoiding uploading checks
IAvecilla Jan 14, 2026
1f545f9
Add test mode cli arg for training
IAvecilla Jan 14, 2026
5706ab7
Fix flag
IAvecilla Jan 14, 2026
5f0edb4
Lower cooldown time for centralized tests
IAvecilla Jan 15, 2026
8b0608e
Merge branch 'cooldown-rework' into remove-checkpointers-config
IAvecilla Jan 15, 2026
48c1bd0
Merge branch 'main' into gcs-2
pefontana Jan 15, 2026
b4db519
generation: Some(file_entry.generation)
pefontana Jan 15, 2026
d53ff7b
update gcp crate to 1.5.x version
dsocolobsky Jan 14, 2026
978b63d
Merge branch 'cooldown-rework' into dy/gcp-new-version
dsocolobsky Jan 15, 2026
29dab79
Merge branch 'main' into gcs-2
pefontana Jan 15, 2026
15b2d5e
Add docs with new cooldown behavior
IAvecilla Jan 15, 2026
634d926
Merge branch 'cooldown-rework' into dy/gcp-new-version
entropidelic Jan 16, 2026
78516e2
Merge branch 'main' into gcs-2
pefontana Jan 16, 2026
172293d
Fix extra docs
IAvecilla Jan 16, 2026
d2ca5cb
Merge branch 'cooldown-rework' into dy/gcp-new-version
dsocolobsky Jan 16, 2026
90339fe
Merge branch 'gcs-2' into cooldown-rework
IAvecilla Jan 16, 2026
6254788
fix google cloud storage code
dsocolobsky Jan 16, 2026
64f683b
Merge branch 'cooldown-rework' into dy/gcp-new-version
dsocolobsky Jan 16, 2026
e2d87ef
fix tcp send_checkpoint
pefontana Jan 16, 2026
3bf0e42
Merge branch 'cooldown-rework' into remove-checkpoint-repo-arg
IAvecilla Jan 16, 2026
45864ca
HF_TOKEN: secrets.HF_TOKEN
pefontana Jan 16, 2026
5355592
Remove hub-repo flag from test
IAvecilla Jan 16, 2026
e99a049
Merge branch 'gcs-2' into cooldown-rework
IAvecilla Jan 16, 2026
cb5b265
Add check for permissions before joining the run
IAvecilla Jan 16, 2026
1b0c1a6
Remove `--hub-repo` and `--gcs-bucket`from train args (#490)
IAvecilla Jan 16, 2026
f155690
add GCS credentials to scratch dir used in run manager
Jan 16, 2026
329c604
Merge branch 'cooldown-rework' into run-manager/gcs-credentials
entropidelic Jan 16, 2026
f361b93
Remove google-cloud-storage from coordinator toml
IAvecilla Jan 16, 2026
611f322
Remove hub repo arguments in test
IAvecilla Jan 16, 2026
dd55838
Merge branch 'cooldown-rework' into run-manager/gcs-credentials
entropidelic Jan 16, 2026
2809135
Update to version 1.6
IAvecilla Jan 16, 2026
089ca65
Fix extra args in script
IAvecilla Jan 16, 2026
2402b4e
Remove sanity check for permissions
IAvecilla Jan 16, 2026
ad5ba5e
Merge branch 'cooldown-rework' into dy/gcp-new-version
IAvecilla Jan 16, 2026
6983523
Check bucket and repo permissions before joining run
IAvecilla Jan 19, 2026
de7e19d
Fix centralized config
IAvecilla Jan 19, 2026
95c8a83
Add new NoUpload to avoid checkpointing in tests
IAvecilla Jan 19, 2026
84c02b3
Merge branch 'dy/gcp-new-version' into cooldown-rework
IAvecilla Jan 19, 2026
49669ff
Fix train solana test script
IAvecilla Jan 19, 2026
e048b25
CooldownStep.checkpoint_complete
pefontana Jan 19, 2026
581d87f
Fix crash after credential errors
IAvecilla Jan 20, 2026
5ff3184
Update only safetensors for cehckpointing
IAvecilla Jan 20, 2026
a8444b2
Refactor hub repo upload
IAvecilla Jan 20, 2026
b6802b5
Uncomment docker cleanup
IAvecilla Jan 20, 2026
da2fab6
Remove NoCheckpoint variant and use a client flag instead
IAvecilla Jan 20, 2026
d6a1201
Send cooldown witness on skip checkpoint
IAvecilla Jan 20, 2026
816ba5b
Skip local save with skip upload flag
IAvecilla Jan 20, 2026
3b83b2c
Merge branch 'cooldown-rework' into run-manager/gcs-credentials
IAvecilla Jan 20, 2026
6e7ecff
Merge branch 'main' into gcs-2
pefontana Jan 21, 2026
f4987c7
Merge branch 'main' into gcs-2
entropidelic Jan 21, 2026
150ce35
Merge branch 'main' into gcs-2
pefontana Jan 22, 2026
c6016b7
Remove comment and add early return on skip upload check
IAvecilla Jan 26, 2026
98c1abe
Fix centralized permission check to upload
IAvecilla Jan 26, 2026
7d8bf2c
Fix cooldown checks and update comment on test flag
IAvecilla Jan 26, 2026
34198e1
Merge branch 'main' into gcs-2
entropidelic Jan 26, 2026
eb043ba
Merge branch 'main' into gcs-2
pefontana Jan 27, 2026
fb4810d
Merge branch 'gcs-2' into cooldown-rework
IAvecilla Jan 27, 2026
68e0252
Merge branch 'main' into gcs-2
pefontana Jan 28, 2026
916d3ae
add button for GCS checkpoint if bucket is public
Jan 28, 2026
2d1e527
fix fakedata.ts
Jan 28, 2026
0134d01
Merge branch 'gcs-2' into cooldown-rework
IAvecilla Jan 29, 2026
4437389
Merge branch 'main' into gcs-2
pefontana Feb 2, 2026
699c4fc
clippy
pefontana Feb 2, 2026
39d5a4b
add GCS documentation for bucket creation and manifest file explanation
entropidelic Feb 2, 2026
92097ba
Refactor credentials validation to checkpoint destination
IAvecilla Feb 4, 2026
a1cfcf4
Merge branch 'main' into gcs-2
pefontana Feb 4, 2026
a0bd207
Refactor credentials management for external sources
IAvecilla Feb 4, 2026
83a3ed8
Remove is_test flag for testing
IAvecilla Feb 4, 2026
c6017de
Refactor hub repo and gcs bucket type
IAvecilla Feb 4, 2026
9ddffcf
Save local checkpoints inside scratch dir for docker containers
IAvecilla Feb 4, 2026
1108f4a
Fix checkpoint enum order
IAvecilla Feb 4, 2026
19db71d
Avoid checking for gcs credentials in run manager everytime
IAvecilla Feb 4, 2026
fa96d31
Merge branch 'main' into cooldown-rework
IAvecilla Feb 4, 2026
2d025cc
Merge branch 'gcs-2' into cooldown-rework
IAvecilla Feb 4, 2026
c13d521
Fix import error
IAvecilla Feb 4, 2026
ad8276d
Fix clippy
IAvecilla Feb 4, 2026
01d196d
Refactor send opportunistic witness function
IAvecilla Feb 4, 2026
540d352
Use Google credentials path in env file to mount in run manager
IAvecilla Feb 5, 2026
9cdf792
Fix get data from bucket without credentials
IAvecilla Feb 5, 2026
8923d3e
Fix credentials check in testing run
IAvecilla Feb 5, 2026
571f7ea
Merge branch 'main' into cooldown-rework
IAvecilla Feb 27, 2026
7d3fcc5
Merge branch 'main' into cooldown-rework
IAvecilla Feb 27, 2026
45842b1
Fix clippy
IAvecilla Feb 27, 2026
b5d7945
Merge branch 'main' into cooldown-rework
IAvecilla Feb 27, 2026
dc9a9bc
Trigger ci
IAvecilla Feb 27, 2026
422eb3e
Merge branch 'main' into cooldown-rework
IAvecilla Mar 2, 2026
8475551
Fix diagrams and docs
IAvecilla Mar 2, 2026
8f8e2d8
Remove unrelated file
IAvecilla Mar 2, 2026
9501981
Remove old args from TrainArgs
IAvecilla Mar 2, 2026
ab859a1
Update Cargo.lock
IAvecilla Mar 2, 2026
dc3b45a
Trigger ci
IAvecilla Mar 2, 2026
7a0b08e
Use Cargo.lock from main and just update necessary things
IAvecilla Mar 2, 2026
f4c244f
Reset Cargo.lock
IAvecilla Mar 2, 2026
5056b34
Merge branch 'main' into cooldown-rework
IAvecilla Mar 2, 2026
fc0df40
Update Cargo.lock with necessary dependencies
IAvecilla Mar 2, 2026
1bf734d
Fix p2p sanity check
IAvecilla Mar 3, 2026
79da777
Include model.rs as cache key
IAvecilla Mar 3, 2026
f9d99c0
Merge branch 'main' into cooldown-rework
IAvecilla Mar 3, 2026
98ffa4d
signed url first implementation
pefontana Mar 2, 2026
e0e2772
generation number in manifest
pefontana Mar 3, 2026
4d907c9
Add HTTP status checks on signed URL downloads
pefontana Mar 4, 2026
e781454
Warn when run_down_client is None for GCS uploads
pefontana Mar 4, 2026
9a9119b
Replace expect() with proper error for missing RunDownClient
pefontana Mar 4, 2026
d73ad3b
Stream uploads and downloads instead of loading into memory
pefontana Mar 4, 2026
2cd1d94
Use signed-urls subdirectory for cache path
pefontana Mar 4, 2026
5308592
Warn when x-goog-generation header is missing
pefontana Mar 4, 2026
e41cd88
clippy
pefontana Mar 4, 2026
2416057
Add SIGNED_URL_EXPIRY_SECONDS const for signed URL expiry
pefontana Mar 4, 2026
cdde009
Use real GCS generation number from manifest response header
pefontana Mar 4, 2026
607727e
Use ring for rustls dependency
IAvecilla Mar 5, 2026
eddadf3
Merge branch 'cooldown-rework' into run-down-signed-urls
pefontana Mar 5, 2026
8f05146
Merge branch 'main' into cooldown-rework
IAvecilla Mar 5, 2026
82aaa12
Include wallet address in run-down requests
pefontana Mar 5, 2026
8794a90
Include wallet address in signed message for verification
pefontana Mar 5, 2026
11f7617
Remove signed-urls subdirectory from cache path
pefontana Mar 5, 2026
4abde70
Remove GCS credentials from run-manager, now uses signed URLs
pefontana Mar 5, 2026
8675c1c
Merge branch 'cooldown-rework' into run-down-signed-urls
pefontana Mar 5, 2026
1dfec41
Merge branch 'main' into cooldown-rework
IAvecilla Mar 5, 2026
273b604
Fix compilation after merge
IAvecilla Mar 5, 2026
392afd0
Merge branch 'main' into cooldown-rework
IAvecilla Mar 6, 2026
9c02ebe
Fix compilation after merge
IAvecilla Mar 6, 2026
19618bb
Merge branch 'main' into cooldown-rework
IAvecilla Mar 6, 2026
e521064
Fix compilation of centralized client
IAvecilla Mar 6, 2026
acd771d
Fix clippy
IAvecilla Mar 6, 2026
94120e3
Merge branch 'main' into cooldown-rework
IAvecilla Mar 10, 2026
546f05a
Merge branch 'cooldown-rework' into run-down-signed-urls
dsocolobsky Mar 13, 2026
b79f850
Merge validate gcs write permissions to validate function
IAvecilla Mar 16, 2026
bf8b3d4
Check write access to hf repo before joining
IAvecilla Mar 16, 2026
81263fd
Always create valid credentials and refresh on every cooldown step
IAvecilla Mar 16, 2026
b258561
Move rustls as workspace dependency
IAvecilla Mar 16, 2026
8a6e3c8
Export env variables instead of inline usage
IAvecilla Mar 16, 2026
79b1f3c
Fix info comment
IAvecilla Mar 16, 2026
8289d50
Remove duplication of checkpoint types in the enum
IAvecilla Mar 16, 2026
92f1687
Small code changes
IAvecilla Mar 16, 2026
77dc6d1
Always upload before entering pause state
IAvecilla Mar 16, 2026
53171ad
Fix comment
IAvecilla Mar 16, 2026
d9c849f
Rename files to keep consistency
IAvecilla Mar 16, 2026
1224df7
Move all booleans outside step state machine to their respective acti…
IAvecilla Mar 17, 2026
9856722
Add cancel step as part of finish
IAvecilla Mar 17, 2026
3d5423d
Remove unused functions and clean unused dependencies
IAvecilla Mar 17, 2026
b4496cd
Merge branch 'main' into cooldown-rework
IAvecilla Mar 17, 2026
4dd1c14
Rollback to the original checkpoint variants
IAvecilla Mar 17, 2026
1df87fb
Remove old repr for checkpoints no longer needed
IAvecilla Mar 17, 2026
317fc96
Add skip checkpoint upload env variable
IAvecilla Mar 17, 2026
a99fd03
Merge branch 'cooldown-rework' into run-down-signed-urls
dsocolobsky Mar 17, 2026
aad57ec
Merge branch 'cooldown-rework' into run-down-signed-urls
pefontana Mar 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/solana-build-anchor-programs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ jobs:
path: validator-image.tar.gz
# FIXME: this key will make the cache succeed even if some other nix files which affect it change.
# we should *really* build this with nix instead of a manual hashFiles / cache here
key: validator-image-${{ runner.os }}-${{ hashFiles('shared/coordinator/src/coordinator.rs', 'architectures/decentralized/solana-coordinator/**/*.rs', 'architectures/decentralized/solana-coordinator/**/*.toml', 'architectures/decentralized/solana-coordinator/Cargo.lock', 'architectures/decentralized/solana-authorizer/**/*.rs', 'architectures/decentralized/solana-authorizer/**/*.toml', 'architectures/decentralized/solana-authorizer/Cargo.lock', 'docker/test/psyche_solana_validator_entrypoint.sh', 'nix/docker.nix', 'flake.lock') }}
key: validator-image-${{ runner.os }}-${{ hashFiles('shared/coordinator/src/coordinator.rs', 'shared/coordinator/src/model.rs', 'architectures/decentralized/solana-coordinator/**/*.rs', 'architectures/decentralized/solana-coordinator/**/*.toml', 'architectures/decentralized/solana-coordinator/Cargo.lock', 'architectures/decentralized/solana-authorizer/**/*.rs', 'architectures/decentralized/solana-authorizer/**/*.toml', 'architectures/decentralized/solana-authorizer/Cargo.lock', 'docker/test/psyche_solana_validator_entrypoint.sh', 'nix/docker.nix', 'flake.lock') }}
lookup-only: true

# Build validator image if cache fails
Expand Down Expand Up @@ -89,7 +89,7 @@ jobs:
uses: actions/cache/save@v4
with:
path: validator-image.tar.gz
key: validator-image-${{ runner.os }}-${{ hashFiles('shared/coordinator/src/coordinator.rs', 'architectures/decentralized/solana-coordinator/**/*.rs', 'architectures/decentralized/solana-coordinator/**/*.toml', 'architectures/decentralized/solana-coordinator/Cargo.lock', 'architectures/decentralized/solana-authorizer/**/*.rs', 'architectures/decentralized/solana-authorizer/**/*.toml', 'architectures/decentralized/solana-authorizer/Cargo.lock', 'docker/test/psyche_solana_validator_entrypoint.sh', 'nix/docker.nix', 'flake.lock') }}
key: validator-image-${{ runner.os }}-${{ hashFiles('shared/coordinator/src/coordinator.rs', 'shared/coordinator/src/model.rs', 'architectures/decentralized/solana-coordinator/**/*.rs', 'architectures/decentralized/solana-coordinator/**/*.toml', 'architectures/decentralized/solana-coordinator/Cargo.lock', 'architectures/decentralized/solana-authorizer/**/*.rs', 'architectures/decentralized/solana-authorizer/**/*.toml', 'architectures/decentralized/solana-authorizer/Cargo.lock', 'docker/test/psyche_solana_validator_entrypoint.sh', 'nix/docker.nix', 'flake.lock') }}

- name: Build complete
if: steps.cache-validator.outputs.cache-hit != 'true'
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/solana-integration-test-base.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ jobs:
uses: actions/cache/restore@v4
with:
path: validator-image.tar.gz
key: validator-image-${{ runner.os }}-${{ hashFiles('shared/coordinator/src/coordinator.rs', 'architectures/decentralized/solana-coordinator/**/*.rs', 'architectures/decentralized/solana-coordinator/**/*.toml', 'architectures/decentralized/solana-coordinator/Cargo.lock', 'architectures/decentralized/solana-authorizer/**/*.rs', 'architectures/decentralized/solana-authorizer/**/*.toml', 'architectures/decentralized/solana-authorizer/Cargo.lock', 'docker/test/psyche_solana_validator_entrypoint.sh', 'nix/docker.nix', 'flake.lock') }}
key: validator-image-${{ runner.os }}-${{ hashFiles('shared/coordinator/src/coordinator.rs', 'shared/coordinator/src/model.rs', 'architectures/decentralized/solana-coordinator/**/*.rs', 'architectures/decentralized/solana-coordinator/**/*.toml', 'architectures/decentralized/solana-coordinator/Cargo.lock', 'architectures/decentralized/solana-authorizer/**/*.rs', 'architectures/decentralized/solana-authorizer/**/*.toml', 'architectures/decentralized/solana-authorizer/Cargo.lock', 'docker/test/psyche_solana_validator_entrypoint.sh', 'nix/docker.nix', 'flake.lock') }}
fail-on-cache-miss: true

- name: Load Validator Image
Expand Down
Loading
Loading