fix: add modelReferences to endpoint create/update and document in runpod.toml template#281
fix: add modelReferences to endpoint create/update and document in runpod.toml template#281jebenexer wants to merge 2 commits into
Conversation
|
Promptless prepared a documentation update related to this change. Triggered by PR #281 This documents the new |
jebenexer
force-pushed
the
ben/sls-94-modelname-cli
branch
from
0f09ed8 to
8a4b418
Compare
There was a problem hiding this comment.
Pull request overview
Adds modelReferences support to serverless endpoint creation and introduces a new CLI command to update an existing endpoint’s cached model references, with accompanying documentation in the runpod.toml template.
Changes:
- Extend endpoint API payloads/queries to include
modelReferences, and fix JSON unmarshalling for endpoint IDs. - Read
model_refsfromrunpod.tomlduring project deploy and pass through to endpoint creation. - Add
runpodctl endpoint model ...command for updating model references on an existing endpoint.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| cmd/root.go | Registers the new top-level endpoint command group. |
| cmd/project/tomlBuilder.go | Documents model_refs in the generated runpod.toml template. |
| cmd/project/functions.go | Parses model_refs from TOML and passes them into endpoint creation. |
| cmd/endpoint/update_model.go | Adds CLI subcommand to set endpoint model references. |
| cmd/endpoint/endpoint.go | Introduces the endpoint command group and wires subcommands. |
| api/endpoint.go | Adds ModelReferences to create input, fixes Endpoint.Id JSON tag, extends queries, and adds model update mutation helper. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if refs, ok := endpointConfig.Get("model_refs").([]interface{}); ok { | ||
| for _, r := range refs { | ||
| modelRefs = append(modelRefs, r.(string)) | ||
| } | ||
| } |
| var modelCmd = &cobra.Command{ | ||
| Use: "model <endpoint-id> <model-ref> [model-ref...]", | ||
| Short: "update model references on an endpoint", | ||
| Long: "set the model references (cached models) for a serverless endpoint", | ||
| Args: cobra.MinimumNArgs(2), | ||
| RunE: runModel, | ||
| } | ||
|
|
||
| func runModel(cmd *cobra.Command, args []string) error { | ||
| endpointID := args[0] | ||
| modelRefs := args[1:] | ||
|
|
- Use checked type assertion for model_refs TOML array elements in functions.go to avoid panic on non-string values - Add --clear flag to 'endpoint model' command to allow removing all model references from an endpoint without providing a dummy value
TimPietruskyRunPod
left a comment
TimPietruskyRunPod
left a comment
There was a problem hiding this comment.
the modelReferences plumbing here is solid — CreateEndpointInput, the GetEndpoints query addition, UpdateEndpointModel() via saveEndpoint, and the runpod.toml model_refs key are all the right pieces. requesting changes on the CLI surface only.
don't introduce a new top-level endpoint group — use the existing serverless group.
we already have serverless (create/delete/get/list/update), and serverless create --model-reference already landed in #276. adding endpoint model alongside it gives two command namespaces for the same resource, and model would be the only thing under endpoint — so the surface fragments and --model-reference (create) vs endpoint model (update) becomes inconsistent for the same concept.
endpoint model <id> [refs...] [--clear] is functionally just a serverless update that only touches model refs. fold it into the command that already exists:
- set/replace:
runpodctl serverless update <id> --model-reference a --model-reference b - remove all:
runpodctl serverless update <id> --clear-models
concretely:
- add a repeatable
--model-reference(StringArrayVar) and a--clear-modelsbool tocmd/serverless/update.go, matching the--model-referenceverb already onserverless create - on update, set
modelReferenceson thesaveEndpointinput: the provided refs, or[]when--clear-modelsis passed (same empty-list-to-clear behavior the current--clearalready relies on) - keep all the api /
runpod.tomlplumbing from this PR as-is — only the command placement changes - drop
cmd/endpoint/(and revert the docs to extend the serverless reference page instead of a new endpoint page)
the only thing that would justify a new group is a deliberate rename serverless → endpoint (the API/console domain term is "Endpoint") with serverless kept as a hidden alias — but that's a bigger decision and should be its own PR with a deprecation plan, not introduced as a side effect of adding model refs.
tl;dr: plumbing 👍 — move endpoint model → serverless update --model-reference / --clear-models and drop the new top-level group.
|
to be clear on scope — the read-side additions are great, keep them:
the only change requested is the write-side command placement: drop the new top-level so: read extension to |
Summary
Adds support for
modelReferencesin the endpoint create/update flow.Changes
api/endpoint.go— AddedModelReferences []stringfield toCreateEndpointInput; fixed missingjson:"id"tag onEndpoint.Id; addedmodelReferencestoGetEndpointsGraphQL query; addedUpdateEndpointModel()function that usessaveEndpointmutation to update models on existing endpointscmd/project/functions.go—deployProjectreadsmodel_refsfrom endpoint config and passes it asModelReferencestoCreateEndpointInputcmd/project/tomlBuilder.go— Documents the newmodel_refsconfig key in the[endpoint]section of therunpod.tomltemplatecmd/endpoint/update_model.go(new) — Addsrunpodctl endpoint model <endpoint-id> <model-ref> [model-ref...]CLI command to set model references on an existing endpointcmd/root.go— Registers the new endpoint command groupRelated