Skip to content

docs(book): add Node Operator Guide and DisTrO explainer - #653

Closed
salch-cred wants to merge 1 commit into
PsycheFoundation:mainfrom
salch-cred:docs/node-operator-guide
Closed

docs(book): add Node Operator Guide and DisTrO explainer#653
salch-cred wants to merge 1 commit into
PsycheFoundation:mainfrom
salch-cred:docs/node-operator-guide

Conversation

@salch-cred

Copy link
Copy Markdown

What

Adds two new chapters to the psyche-book:

  1. Node Operator Guide (psyche-book/src/explain/operator-guide.md) — a field manual for people operating Psyche training nodes long-term:

    • what the operator actually owns (run-manager, container, keypair, RPC, GPU stack)
    • the epoch lifecycle from the operator's seat (WaitingForMembers -> Warmup -> Rounds -> Cooldown), with per-phase failure modes and how to read epoch boundaries in logs
    • how per-epoch rewards accrue (whole-epoch completion) and what that implies for maintenance windows
    • session management (tmux + systemd unit example), startup checklist, monitoring (container, phases, GPU, disk, RPC), updating (container automatic vs run-manager manual)
    • graceful exit timing, delegation/multi-machine setups, claiming via the Treasurer, and a troubleshooting quick-reference table
  2. DisTrO (psyche-book/src/explain/distro.md) — an explainer for the DisTrO optimizer, grounded in the actual implementation (shared/modeling/src/distro.rs):

    • the synchronization-bandwidth problem and why data-center-style gradient sync fails over the internet
    • the two key ideas: DCT-based compression + top-k, and momentum-residual accumulation so dropped components are recovered in later rounds
    • a step-by-step walkthrough of Distro::generate (decode peers' updates -> apply -> gradient -> residual update -> DCT encode -> top-k -> 1-bit sign quantization -> P2P broadcast), with a mermaid diagram
    • how it fits with the Coordinator rounds/witness quorum (off-chain weights, on-chain state machine)
    • config knobs table (compression_decay, compression_chunk, compression_topk, quantize_1bit) and how they interact with max_round_train_time

Also wires both into SUMMARY.md (DisTrO under "Psyche in depth", Operator Guide as its own top-level section after it).

Why

  • There is currently no operator-oriented documentation: the quickstart covers first boot, the FAQ covers one-liners, but nothing covers running a node over time (epochs, maintenance, monitoring, recovery). Node operators are the run's supply side, and the on-chain epoch/reward semantics (coordinator/mining-pool/treasurer programs) make the timing of joins/exits financially meaningful — that deserves explicit documentation.
  • DisTrO is the core novelty that makes Psyche work over the internet, but it is only documented in blog posts; the repo itself has no explanation of shared/modeling/src/distro.rs, its wire format, or its config knobs. The explainer links the optimizer to the run config and the general workflow chapter.

All content is grounded in the repo itself: the epoch state machine and witness/quorum terminology come from general-workflow.md/glossary.md, the reward rules from rewards.md, the delegation flow from quickstart-compute-provider.md/join-run.md, and the DisTrO mechanics from the Distro/TransformDCT/CompressDCT implementations in shared/modeling/src/distro.rs. Internal links verified to resolve against the book tree.

Happy to restructure (e.g. move the operator guide under enduser/, or change section ordering) if maintainers prefer a different placement.

@salch-cred

Copy link
Copy Markdown
Author

Closing as a duplicate - superseded by two focused PRs: DisTrO explainer in #651 and Node Operator Guide in #652. This PR's head branch now points at the operator-guide-only commit (identical to #652), so its title no longer matches its contents. The two focused PRs merge independently and avoid SUMMARY.md conflicts.

@salch-cred salch-cred closed this Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants