Skip to content

feat(lora): let a running pod fetch and register a LoRA - #315

Open
gioelecerati wants to merge 1 commit into
mainfrom
gio/feat/add-lora-at-runtime
Open

feat(lora): let a running pod fetch and register a LoRA#315
gioelecerati wants to merge 1 commit into
mainfrom
gio/feat/add-lora-at-runtime

Conversation

@gioelecerati

Copy link
Copy Markdown
Collaborator

Why

A pod builds its LoRA catalog once, in register_library() at engine init. Anything trained after that can never be enabled — enable_lora on an id that wasn't in the boot scan raises "not registered". So a LoRA a user trains while their session is open is unreachable, which is exactly when they want it.

/api/loras re-globs the disk per request, so the catalog would even show a file dropped in later. The engine would still refuse it.

What

New add_lora {id, url} command: fetch a .safetensors into loras_dir() and register it into the live catalog.

  • Downloaded on a worker thread. ~170 MB has no business in the dispatch path, let alone the tick. Writes to a .partial and renames on completion, so a dropped connection can't leave a truncated file that the next scan registers and the engine then fails to parse.
  • Queued onto state.pending_register, drained in before_tick beside the existing enable/disable queues, so the catalog mutation serializes with the pipeline like every other LoRA change.
  • Registration runs before the enables in the same drain, deliberately: a client that adds a LoRA and immediately enables it queues both before the next tick, and the other order raises on every such pair. Registration is metadata-only; weights still materialize inside enable_lora.

Guards

The WS is reachable by anyone holding a session, so:

  • the id is sanitized before it becomes a filename and a lora_str_<id> knob name;
  • the URL must be https on an allow-listed host (DEMON_LORA_URL_HOSTS). Without that, add_lora is "make this pod GET an arbitrary URL for me" from inside our network — the classic SSRF shape. Defaults cover the orchestrator (Studio-trained LoRAs) and Tigris (user uploads).

Contract

No failure message on the wire, by design: the fetch outlives the request. The catalog is the completion signal — clients watch for the id to appear, then send enable_lora. Wire types regenerated; the capability-gate and dispatcher-parity invariants that tests/unit/test_wire_contract.py enforces were verified against this branch.

Testing

Not yet exercised against a live pod — this needs a :warm rebake to reach the fleet, which is deliberately not part of this PR.

🤖 Generated with Claude Code

A pod builds its LoRA catalog once, in register_library() at engine init.
Anything trained after that can never be enabled — enable_lora on an id
that wasn't in the boot scan raises "not registered" — so a LoRA a user
trains while their session is open is unreachable, which is exactly when
they want it. /api/loras re-globs the disk per request, so the catalog
would even SHOW a file dropped in later; the engine would still refuse it.

New `add_lora {id, url}`: fetch the file into loras_dir() and register it.

Downloaded on a worker thread — ~170 MB has no business in the dispatch
path, let alone the tick — to a .partial that is renamed on completion, so
a dropped connection can't leave a truncated .safetensors that the next
scan happily registers and the engine then fails to parse. The finished
path is queued onto state.pending_register and drained in before_tick
alongside the existing enable/disable queues, so the catalog mutation
serializes with the pipeline like every other LoRA change.

Registration runs BEFORE the enables in the same drain, deliberately: a
client that adds a LoRA and immediately enables it queues both before the
next tick, and the other order would raise on every such pair.
Registration is metadata-only; weights still materialize inside
enable_lora.

Two guards, because the WS is reachable by anyone holding a session. The
id is sanitized before it becomes a filename and a knob name. The URL must
be https on an allow-listed host (DEMON_LORA_URL_HOSTS) — without that,
add_lora is "make this pod GET an arbitrary URL for me" from inside our
network, which is the whole SSRF shape. Defaults cover the orchestrator
(Studio-trained LoRAs) and Tigris (ones the user uploaded).

There is no failure message on the wire by design: the fetch outlives the
request, and the catalog is the completion signal. Clients watch for the
id to appear, then send enable_lora.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant