Skip to content

Add Anthropic as a supported LLM provider #49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
omkarjoshi0304 wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Add Anthropic as a supported LLM provider #49

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
22ab0b5
Regenerate CRD manifests to pick up updated k8s.io/api doc comment
omkarjoshi0304 Aug 14, 2026
1c57a6b
Add Anthropic as a supported LLM provider
omkarjoshi0304 Aug 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion api/v1beta1/openstacklightspeed_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@ type OpenStackLightspeedCore struct {
LLMEndpoint string `json:"llmEndpoint"`

// +kubebuilder:validation:Required
// +kubebuilder:validation:Enum=azure_openai;openai;watsonx;rhoai_vllm;rhelai_vllm;gemini
// +kubebuilder:validation:Enum=azure_openai;openai;watsonx;rhoai_vllm;rhelai_vllm;gemini;anthropic
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Provider Type"
// Type of the provider serving the LLM
LLMEndpointType string `json:"llmEndpointType"`
Expand Down
13 changes: 7 additions & 6 deletions bundle/manifests/lightspeed.openstack.org_openstacklightspeeds.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@ spec:
- rhoai_vllm
- rhelai_vllm
- gemini
- anthropic
type: string
llmProjectID:
description: Project ID for LLM providers that require it (e.g., WatsonX)
Expand Down Expand Up @@ -206,7 +207,7 @@ spec:
Claims lists the names of resources, defined in spec.resourceClaims,
that are used by this container.

This field depends on the
This is an alpha field and requires enabling the
DynamicResourceAllocation feature gate.

This field is immutable. It can only be set for containers.
Expand Down Expand Up @@ -274,7 +275,7 @@ spec:
Claims lists the names of resources, defined in spec.resourceClaims,
that are used by this container.

This field depends on the
This is an alpha field and requires enabling the
DynamicResourceAllocation feature gate.

This field is immutable. It can only be set for containers.
Expand Down Expand Up @@ -342,7 +343,7 @@ spec:
Claims lists the names of resources, defined in spec.resourceClaims,
that are used by this container.

This field depends on the
This is an alpha field and requires enabling the
DynamicResourceAllocation feature gate.

This field is immutable. It can only be set for containers.
Expand Down Expand Up @@ -409,7 +410,7 @@ spec:
Claims lists the names of resources, defined in spec.resourceClaims,
that are used by this container.

This field depends on the
This is an alpha field and requires enabling the
DynamicResourceAllocation feature gate.

This field is immutable. It can only be set for containers.
Expand Down Expand Up @@ -476,7 +477,7 @@ spec:
Claims lists the names of resources, defined in spec.resourceClaims,
that are used by this container.

This field depends on the
This is an alpha field and requires enabling the
DynamicResourceAllocation feature gate.

This field is immutable. It can only be set for containers.
Expand Down Expand Up @@ -543,7 +544,7 @@ spec:
Claims lists the names of resources, defined in spec.resourceClaims,
that are used by this container.

This field depends on the
This is an alpha field and requires enabling the
DynamicResourceAllocation feature gate.

This field is immutable. It can only be set for containers.
Expand Down
13 changes: 7 additions & 6 deletions config/crd/bases/lightspeed.openstack.org_openstacklightspeeds.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@ spec:
- rhoai_vllm
- rhelai_vllm
- gemini
- anthropic
type: string
llmProjectID:
description: Project ID for LLM providers that require it (e.g., WatsonX)
Expand Down Expand Up @@ -206,7 +207,7 @@ spec:
Claims lists the names of resources, defined in spec.resourceClaims,
that are used by this container.

This field depends on the
This is an alpha field and requires enabling the
DynamicResourceAllocation feature gate.

This field is immutable. It can only be set for containers.
Expand Down Expand Up @@ -274,7 +275,7 @@ spec:
Claims lists the names of resources, defined in spec.resourceClaims,
that are used by this container.

This field depends on the
This is an alpha field and requires enabling the
DynamicResourceAllocation feature gate.

This field is immutable. It can only be set for containers.
Expand Down Expand Up @@ -342,7 +343,7 @@ spec:
Claims lists the names of resources, defined in spec.resourceClaims,
that are used by this container.

This field depends on the
This is an alpha field and requires enabling the
DynamicResourceAllocation feature gate.

This field is immutable. It can only be set for containers.
Expand Down Expand Up @@ -409,7 +410,7 @@ spec:
Claims lists the names of resources, defined in spec.resourceClaims,
that are used by this container.

This field depends on the
This is an alpha field and requires enabling the
DynamicResourceAllocation feature gate.

This field is immutable. It can only be set for containers.
Expand Down Expand Up @@ -476,7 +477,7 @@ spec:
Claims lists the names of resources, defined in spec.resourceClaims,
that are used by this container.

This field depends on the
This is an alpha field and requires enabling the

@umago umago Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

most nit: These changes could be on a separated commit since it's unrelated

DynamicResourceAllocation feature gate.

This field is immutable. It can only be set for containers.
Expand Down Expand Up @@ -543,7 +544,7 @@ spec:
Claims lists the names of resources, defined in spec.resourceClaims,
that are used by this container.

This field depends on the
This is an alpha field and requires enabling the
DynamicResourceAllocation feature gate.

This field is immutable. It can only be set for containers.
Expand Down
4 changes: 4 additions & 0 deletions internal/controller/constants.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,7 @@ const (
RHELAIVLLMProviderName = "rhelai_vllm"
RHOAIVLLMProviderName = "rhoai_vllm"
GeminiProviderName = "gemini"
AnthropicProviderName = "anthropic"
AzureOpenAIProviderName = "azure_openai"
OpenAIProviderName = "openai"
WatsonXProviderName = "watsonx"
Expand All @@ -188,6 +189,9 @@ const (
// EnvVarSuffixAPIKey is the environment variable suffix for API key credentials
EnvVarSuffixAPIKey = "_API_KEY"

// APIKeyConfigField is the Llama Stack provider config field name for API key credentials
APIKeyConfigField = "api_key"

// VectorDBVolumeName is the name of the volume used by init containers to
// store discovered values from vector DB images.
VectorDBVolumeName = "vector-db-discovered-values"
Expand Down
22 changes: 15 additions & 7 deletions internal/controller/llama_stack_config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ func buildLlamaStackInferenceProviders(_ *common_helper.Helper, _ context.Contex

// Map provider types to Llama Stack provider types
switch provider.Type {
case OpenAIProviderName, GeminiProviderName, RHOAIVLLMProviderName, RHELAIVLLMProviderName:
case OpenAIProviderName, GeminiProviderName, AnthropicProviderName, RHOAIVLLMProviderName, RHELAIVLLMProviderName:
config := map[string]interface{}{}
// Determine the appropriate Llama Stack provider type:
// - OpenAI uses remote::openai
Expand All @@ -132,18 +132,26 @@ func buildLlamaStackInferenceProviders(_ *common_helper.Helper, _ context.Contex
switch provider.Type {
case OpenAIProviderName:
providerConfig["provider_type"] = "remote::openai"
apiKeyField = "api_key"
apiKeyField = APIKeyConfigField
case GeminiProviderName:
providerConfig["provider_type"] = "remote::gemini"
apiKeyField = "api_key"
apiKeyField = APIKeyConfigField
case AnthropicProviderName:
providerConfig["provider_type"] = "remote::anthropic"
apiKeyField = APIKeyConfigField
default:
providerConfig["provider_type"] = "remote::vllm"
apiKeyField = "api_token"
}
// Llama Stack will substitute ${env.VAR_NAME} with the actual env var value
config[apiKeyField] = fmt.Sprintf("${env.%s%s}", envVarName, EnvVarSuffixAPIKey)

// Add custom URL if specified
// Add custom URL if specified. Note: remote::gemini and remote::anthropic
// have fixed, well-known API endpoints and their Llama Stack config schemas
// don't define a base_url field at all, so setting llmEndpoint for those
// providers has no effect on the generated config - Llama Stack silently
// ignores it on its end. That's upstream (llama-stack/OGX) behavior, not
// something this operator works around.

@umago umago Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, so for those providers OGX will totally ignore the custom URL ? But not for openai for example ?

That's such a weird behavior.

(Nothing to do on our side, since it's OGX/Llama-Stack)

@omkarjoshi0304 omkarjoshi0304 Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remote::gemini/remote::anthropic configs simply have no base_url field so it's silently dropped by Pydantic. Nothing we can do on our side it's an Llama-Stack schema thing. and yes I won't ignore for openai provider

if provider.URL != "" {
config["base_url"] = provider.URL
}
Expand All @@ -156,7 +164,7 @@ func buildLlamaStackInferenceProviders(_ *common_helper.Helper, _ context.Contex

// Azure supports both API key and client credentials authentication
// Always include api_key (required by LiteLLM's Pydantic validation)
config["api_key"] = fmt.Sprintf("${env.%s_API_KEY}", envVarName)
config[APIKeyConfigField] = fmt.Sprintf("${env.%s_API_KEY}", envVarName)

// Also include client credentials fields (will be empty if not using client credentials)
config["client_id"] = fmt.Sprintf("${env.%s_CLIENT_ID:=}", envVarName)
Expand All @@ -180,7 +188,7 @@ func buildLlamaStackInferenceProviders(_ *common_helper.Helper, _ context.Contex

config := map[string]interface{}{}
config["base_url"] = provider.URL
config["api_key"] = fmt.Sprintf("${env.%s_API_KEY}", envVarName)
config[APIKeyConfigField] = fmt.Sprintf("${env.%s_API_KEY}", envVarName)

if provider.WatsonProjectID != "" {
config["project_id"] = provider.WatsonProjectID
Expand All @@ -190,7 +198,7 @@ func buildLlamaStackInferenceProviders(_ *common_helper.Helper, _ context.Contex

default:
supportedProviders := []string{
OpenAIProviderName, GeminiProviderName, RHOAIVLLMProviderName, RHELAIVLLMProviderName,
OpenAIProviderName, GeminiProviderName, AnthropicProviderName, RHOAIVLLMProviderName, RHELAIVLLMProviderName,
AzureOpenAIProviderName, WatsonXProviderName,
}
return nil, fmt.Errorf(
Expand Down
10 changes: 10 additions & 0 deletions internal/controller/llama_stack_config_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,10 @@ func getOpenStackLightspeedProvidersInstance(provider string) *apiv1beta1.OpenSt
instance.Spec.LLMEndpointType = GeminiProviderName
instance.Spec.ModelName = "gemini-2.0-flash"
return instance
case AnthropicProviderName:
instance.Spec.LLMEndpointType = AnthropicProviderName
instance.Spec.ModelName = "claude-sonnet-4-5"
return instance
case RHOAIVLLMProviderName:
instance.Spec.LLMEndpointType = RHOAIVLLMProviderName
instance.Spec.LLMEndpoint = "https://vllm.example.com/v1"
Expand Down Expand Up @@ -100,6 +104,11 @@ var _ = Describe("Llama Stack config", func() {
Expect(config["api_key"]).To(Equal("${env.OPENSTACK_LIGHTSPEED_PROVIDER_API_KEY}"))
Expect(config).NotTo(HaveKey("base_url"))
}),
Entry("for anthropic", AnthropicProviderName, "remote::anthropic",
func(config map[string]interface{}, _ *apiv1beta1.OpenStackLightspeed) {
Expect(config["api_key"]).To(Equal("${env.OPENSTACK_LIGHTSPEED_PROVIDER_API_KEY}"))
Expect(config).NotTo(HaveKey("base_url"))
}),
Entry("for rhoai_vllm", RHOAIVLLMProviderName, "remote::vllm",
func(config map[string]interface{}, instance *apiv1beta1.OpenStackLightspeed) {
Expect(config["api_token"]).To(Equal("${env.OPENSTACK_LIGHTSPEED_PROVIDER_API_KEY}"))
Expand Down Expand Up @@ -147,6 +156,7 @@ var _ = Describe("Llama Stack config", func() {
},
Entry("for openai", OpenAIProviderName),
Entry("for gemini", GeminiProviderName),
Entry("for anthropic", AnthropicProviderName),
Entry("for rhoai_vllm", RHOAIVLLMProviderName),
Entry("for rhelai_vllm", RHELAIVLLMProviderName),
Entry("for azure_openai", AzureOpenAIProviderName),
Expand Down
Loading