Repository navigation
fix: say why a placement fails instead of always blaming memory - #2383
Open
AlexCheema wants to merge 1 commit into
Open
AlexCheema wants to merge 1 commit into
AlexCheema wants to merge 1 commit into
Conversation
place_instance raised "No cycles found with sufficient memory" for three different problems: - too few nodes connected to each other for min_nodes, - nodes connected, but without enough free memory between them, - nodes that haven't reported their memory yet. The first is common (a 2-node model after one node left) and the message sent people looking for memory. Each now has its own message, and the memory one says how much the model needs and how much is free. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
place_instanceraisesNo cycles found with sufficient memoryfor three different problems:min_nodes(e.g. a model placed across 2 nodes after one of them left).The first is common and sends people looking for a memory problem that isn't there. "Cycles" is also internal vocabulary: this message reaches users through
/place_instance,/instance/previewsand the dashboard.Change
Each case gets its own message:
No nodes are availableNeeds 2 connected nodes, but at most 1 are connected to each othernode_idson previews) not connectedThe chosen nodes aren't all connected to each otherNot enough memory: <model> needs 15.9 GB, but connected nodes have at most 12.3 GB free between themWaiting for the nodes to report their memoryNothing else in placement changes; the checks run in the same order, and only the text of these errors differs.
#2339 (dashboard: say why a model can't be placed) maps the old memory message to plain language; it now recognises both forms, so the two can land in either order. No other code matches on the text (checked the backend, dashboard and tests).
Tests
test_placement.py: one test per message, asserting the exact text, including the free-memory figure. The existing insufficient-memory test is updated to the new text. All master and API tests pass, and basedpyright and ruff are clean.🤖 Generated with Claude Code