Skip to content

fix: say why a placement fails instead of always blaming memory - #2383

Open
AlexCheema wants to merge 1 commit into
mainfrom
fix/say-why-placement-fails
Open

AlexCheema wants to merge 1 commit into
mainfrom
fix/say-why-placement-fails

Conversation

@AlexCheema

Copy link
Copy Markdown
Contributor

Problem

place_instance raises No cycles found with sufficient memory for three different problems:

  1. Too few nodes are connected to each other for min_nodes (e.g. a model placed across 2 nodes after one of them left).
  2. Not enough free memory: the nodes are connected, but don't have enough free memory between them.
  3. Missing memory reports: some nodes haven't reported their memory yet.

The first is common and sends people looking for a memory problem that isn't there. "Cycles" is also internal vocabulary: this message reaches users through /place_instance, /instance/previews and the dashboard.

Change

Each case gets its own message:

Case Message
no nodes No nodes are available
too few connected Needs 2 connected nodes, but at most 1 are connected to each other
chosen nodes (node_ids on previews) not connected The chosen nodes aren't all connected to each other
not enough memory Not enough memory: <model> needs 15.9 GB, but connected nodes have at most 12.3 GB free between them
memory not reported yet Waiting for the nodes to report their memory

Nothing else in placement changes; the checks run in the same order, and only the text of these errors differs.

#2339 (dashboard: say why a model can't be placed) maps the old memory message to plain language; it now recognises both forms, so the two can land in either order. No other code matches on the text (checked the backend, dashboard and tests).

Tests

test_placement.py: one test per message, asserting the exact text, including the free-memory figure. The existing insufficient-memory test is updated to the new text. All master and API tests pass, and basedpyright and ruff are clean.

🤖 Generated with Claude Code

place_instance raised "No cycles found with sufficient memory" for three
different problems:
- too few nodes connected to each other for min_nodes,
- nodes connected, but without enough free memory between them,
- nodes that haven't reported their memory yet.

The first is common (a 2-node model after one node left) and the message sent
people looking for memory. Each now has its own message, and the memory one
says how much the model needs and how much is free.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant