diff --git a/docs/audit-logs-api.md b/docs/audit-logs-api.md index f9a5d1fe9..3e92da6d5 100644 --- a/docs/audit-logs-api.md +++ b/docs/audit-logs-api.md @@ -5,7 +5,7 @@ description: "Retrieve organization activity logs for security monitoring, compl # Logfire audit logs API -The Audit Logs API lets you retrieve activity logs for your organization. This feature is available on the [Enterprise plan](./enterprise.md) only. +The Audit Logs API lets you retrieve activity logs for your organization. This feature is available for Enterprise Cloud organizations and self-hosted deployments. Each log entry records user actions: logins, project updates, token changes, and more. Use it for security monitoring, compliance reporting, and usage auditing. @@ -20,7 +20,7 @@ Each log entry records user actions: logins, project updates, token changes, and ## Authentication -Requests are authenticated with a Bearer token scoped to `organizations:auditlog`. See [API Keys docs](./reference/advanced//use-api-keys.md) for instructions on how to generate one. +Requests are authenticated with a Bearer token scoped to `organization:auditlog`. See [API Keys docs](./reference/advanced/use-api-keys.md) for instructions on how to generate one. **Type:** Bearer token diff --git a/docs/comparisons/arize-phoenix.md b/docs/comparisons/arize-phoenix.md index 60337cb1d..f85e20952 100644 --- a/docs/comparisons/arize-phoenix.md +++ b/docs/comparisons/arize-phoenix.md @@ -9,7 +9,7 @@ Arize Phoenix is an ML observability platform focused on model monitoring, drift | **Primary Focus** | AI observability for agents and apps | ML model monitoring | | **Strength** | AI + application tracing | Drift detection, model performance | | **Non-AI Tracing** | Full support | Limited | -| **Language Support** | Python, JS/TS, Rust (SDKs) + any OTel | Python-focused | +| **Language Support** | Python, JS/TS, Rust (SDKs) + any OTel | Python and JS SDKs + OTLP | | **Evals** | Integrated web-UI - Code-based via `pydantic-evals` | Integrated web-UI - Code-based via external library | | **Pricing** | Per-span ($2/million)* | Usage-based | | **Setup** | 3 lines of code | OTel-based (several lines of code) | @@ -73,7 +73,7 @@ logfire.instrument_openai() Three lines, and you're observing AI calls with full application context. -**Arize Phoenix** requires more configuration, especially for non-AI instrumentation. +**Arize Phoenix** is OpenTelemetry-based, but Logfire provides more opinionated shortcuts for application and AI instrumentation in one SDK. ### Query Interface — Essential for Agentic Coding @@ -86,7 +86,7 @@ Three lines, and you're observing AI calls with full application context. When you're iterating on AI applications with coding agents, the agent needs to understand production behavior. With SQL, it can ask any question. With proprietary interfaces, it's constrained to anticipated queries. -**Arize Phoenix** has its own query interface optimized for ML metrics but less flexible for ad-hoc analysis. +**Arize Phoenix** has query tools optimized for ML and trace workflows, but Logfire exposes a PostgreSQL-compatible SQL interface for ad-hoc analysis. ## Complementary Use diff --git a/docs/comparisons/datadog.md b/docs/comparisons/datadog.md index 4e1ee1687..42203d810 100644 --- a/docs/comparisons/datadog.md +++ b/docs/comparisons/datadog.md @@ -6,11 +6,11 @@ Datadog is a comprehensive enterprise monitoring platform. Logfire is an AI-nati | Feature | Logfire | Datadog | |-------------------------|--------------------------------|---------| -| **Architecture** | OpenTelemetry-native | Proprietary agents | +| **Architecture** | OpenTelemetry-native | Datadog agents plus OTel ingestion | | **Pricing Model** | Per-span ($2/million)* | Per-host + ingestion + custom metrics | | **Host Fees** | None | $15-40/host/month | -| **AI/LLM Support** | First-class, one function call | Add-on, separate product | -| **Query Language** | SQL (Postgres-compatible) | Proprietary | +| **AI/LLM Support** | First-class, one function call | LLM Observability product with SDK/OTel options | +| **Query Language** | SQL (Postgres-compatible) | Product-specific query tools | | **Setup Complexity** | 3 lines of code | Agent deployment per host | | **Autoscaling Impact** | Linear cost increase | High-water-mark billing spikes | | **Dashboards & alerts** | Included, SQL-based | Included, proprietary query language | @@ -55,7 +55,7 @@ Datadog is a comprehensive enterprise monitoring platform. Logfire is an AI-nati ### AI/LLM Support -**Datadog** added LLM observability as a separate product. It works, but AI isn't central to the platform's design. +**Datadog** has an LLM Observability product, including SDK and OpenTelemetry GenAI ingestion options. It is part of a broad enterprise monitoring platform rather than the central design point of the product. **Logfire** was built for the AI era. [One function call](https://pydantic.dev/docs/logfire/integrations/?utm_source=datadog_comparison_docs) (`logfire.instrument_openai()`) gives you: @@ -67,13 +67,13 @@ Datadog is a comprehensive enterprise monitoring platform. Logfire is an AI-nati ### OpenTelemetry -**Datadog** uses proprietary agents. While they support OTel export, it's not the native path. +**Datadog** supports OpenTelemetry ingestion, including GenAI semantic-convention traces for LLM Observability, while much of the broader Datadog experience still uses Datadog agents and product-specific setup. **Logfire** is OpenTelemetry-native with [first-class integrations for most technologies](https://pydantic.dev/docs/logfire/integrations/?utm_source=datadog_comparison_docs). Any OTel instrumentation works automatically. Your instrumentation is portable: if you ever want to switch, your code doesn't change. ### Query Language — Essential for Agentic Coding -**Datadog** uses a proprietary query language for dashboards and analysis. This creates limitations: +**Datadog** uses product-specific query tools for dashboards and analysis. Compared with a PostgreSQL-compatible SQL interface, this creates tradeoffs: - Learning curve for humans and AI alike - Coding agents are constrained to anticipated queries @@ -86,7 +86,7 @@ Datadog is a comprehensive enterprise monitoring platform. Logfire is an AI-nati - **Agentic workflows** — When coding agents debug your AI application, they can write arbitrary queries - **Familiar syntax** — No new query language to learn -When you're iterating on AI applications with coding agents, the agent needs to understand production behavior. With SQL, it can ask any question. With proprietary DSLs, it's constrained to what someone anticipated. +When you're iterating on AI applications with coding agents, the agent needs to understand production behavior. With SQL, it can ask any question. With product-specific query interfaces, it may be constrained by the available interface for that product area. ## Migration Path diff --git a/docs/comparisons/langfuse.md b/docs/comparisons/langfuse.md index cb0faec50..35535ed11 100644 --- a/docs/comparisons/langfuse.md +++ b/docs/comparisons/langfuse.md @@ -14,7 +14,7 @@ Both Logfire and Langfuse help you observe AI/LLM applications, but they take fu | **Python Support** | First-class (Pydantic team) | Good | | **Non-AI Tracing** | Full support | Limited | | **LLM Features** | Token tracking, costs, panels | Token tracking, costs, evals, prompt mgmt | -| **OpenTelemetry** | Native | Export support | +| **OpenTelemetry** | Native | Native SDK / OTLP ingestion | *Logfire Cloud pricing (Team or Growth plans). Enterprise pricing available [on request](https://calendar.app.google/k9pkeuNMmzJAJ4Mx5). @@ -45,7 +45,7 @@ This matters because AI applications don't exist in isolation. They call APIs, q ### Query Language — Essential for Agentic Coding -**Langfuse** uses a custom UI and API for querying data. +**Langfuse** uses a custom UI and API for querying data, with OpenTelemetry-based ingestion support for traces. **Logfire** uses SQL with PostgreSQL-compatible syntax. This is a significant advantage for AI-assisted development: diff --git a/docs/comparisons/sentry.md b/docs/comparisons/sentry.md index b5c2b30f7..70ec2943c 100644 --- a/docs/comparisons/sentry.md +++ b/docs/comparisons/sentry.md @@ -8,7 +8,7 @@ Sentry is a mature error monitoring platform. Logfire is an AI-native observabil |---------------------|----------------------------------------|------------------------------------------| | **Primary Focus** | Full observability (logs, traces, AI) | Error monitoring | | **App Tracing** | Core capability | Available, not a core focus | -| **AI/LLM Support** | First-class, automatic instrumentation | Generic function tracing only | +| **AI/LLM Support** | First-class, automatic instrumentation | AI Performance/LLM monitoring features | | **Logging** | Structured logs with full context | Error-focused | | **Live View** | Real-time "pending spans" | ❌ | | **Query Interface** | SQL (Postgres-compatible) | Custom UI | @@ -48,14 +48,9 @@ Sentry is a mature error monitoring platform. Logfire is an AI-native observabil ### AI/LLM Support -**Sentry** treats AI calls like any other function. You'll see that an error occurred, but you won't see: +**Sentry** now has AI Performance and LLM monitoring features for supported SDKs and integrations. Those features are useful when your workflow is already centered on Sentry's error and performance tooling. -- What prompt was sent -- What the model responded -- Token usage and costs -- Tool calls and their results - -**Logfire** was built for AI applications. One function call gives you complete LLM visibility: +**Logfire** was built for AI applications and full-stack telemetry from the start. One function call gives you LLM visibility alongside the rest of your OpenTelemetry data: ```python skip="true" skip-reason="incomplete" import logfire @@ -71,7 +66,7 @@ logfire.instrument_openai() # That's it ### SQL-Powered Analytics — Essential for Agentic Coding -**Sentry** uses a custom UI for querying and filtering. +**Sentry** uses custom query and exploration UIs for querying and filtering. **Logfire** uses SQL with PostgreSQL-compatible syntax. This is a significant advantage for AI-assisted development: diff --git a/docs/enterprise.md b/docs/enterprise.md index f5a1e4416..e80305cc9 100644 --- a/docs/enterprise.md +++ b/docs/enterprise.md @@ -14,9 +14,9 @@ In addition to the [Team and Growth plans](https://pydantic.dev/pricing), Pydant ## Enterprise Single Sign-On (SSO) -Logfire Enterprise supports SSO through [Dex](https://github.com/dexidp/dex), an open-source OIDC gateway. The same Dex configuration model works across Enterprise Cloud, Enterprise Dedicated, and Enterprise Self-Hosted deployments. +Logfire Enterprise SSO is built on [Dex](https://github.com/dexidp/dex), an open-source OIDC gateway. -Dex works with common identity providers including Okta, Azure AD, Auth0, Google Workspace, LDAP/AD, and generic OIDC or SAML providers. +Enterprise Cloud and Enterprise Dedicated support managed OIDC identity providers such as Okta, Microsoft Azure Entra ID, and Keycloak. Self-hosted deployments configure Dex directly through Helm values, so they can use any connector supported by Dex, including OIDC, SAML, LDAP, GitHub, and Google. ## Enterprise Cloud diff --git a/docs/evaluate/datasets/evaluations.md b/docs/evaluate/datasets/evaluations.md index a9fb1069b..4cef894e4 100644 --- a/docs/evaluate/datasets/evaluations.md +++ b/docs/evaluate/datasets/evaluations.md @@ -123,7 +123,7 @@ print(f'Fetched {len(dataset.cases)} cases') print(f'First case input type: {type(dataset.cases[0].inputs).__name__}') ``` -If you have custom evaluator types stored with your cases, pass them via `custom_evaluator_types` so they can be deserialized: +If you have custom evaluator types stored with your cases or dataset, pass them via `custom_evaluator_types` so they can be deserialized: ```python skip="true" skip-reason="external-connection" dataset = client.get_dataset( @@ -131,6 +131,7 @@ dataset = client.get_dataset( input_type=QuestionInput, output_type=AnswerOutput, custom_evaluator_types=[MyCustomEvaluator], + custom_report_evaluator_types=[MyCustomReportEvaluator], ) ``` diff --git a/docs/evaluate/datasets/ui.md b/docs/evaluate/datasets/ui.md index 2c2b26f12..efcd743b6 100644 --- a/docs/evaluate/datasets/ui.md +++ b/docs/evaluate/datasets/ui.md @@ -68,7 +68,7 @@ Once created, you can edit the dataset to add a description and define schemas. From the dataset detail page, click **Edit** to modify the dataset's configuration. The edit form has two sections: - **General**: Name and description. -- **Schemas**: Define JSON schemas for inputs, expected outputs, and metadata. Use the **Generate schema** toggle to have Pydantic AI create schemas from a natural language description of your data shape. +- **Schemas**: Define JSON schemas for inputs, expected outputs, and metadata. Use the **Generate schema** action to have Pydantic AI create schemas from a natural language description of your data shape. ## Managing Cases @@ -134,11 +134,13 @@ This preserves a link back to the source trace, so you always know where a test From the dataset detail page, click **Export** to download the dataset in one of two formats: - **JSON**: Raw JSON representation of all cases. -- **pydantic-evals**: A YAML format compatible with `pydantic_evals.Dataset.from_file()`. +- **Python (pydantic-evals)**: JSON in the pydantic-evals-compatible `{name, cases, evaluators, report_evaluators}` shape, suitable for loading with `pydantic_evals.Dataset.from_dict()`. ## What's Next? Once you have cases in a dataset, you can: +- Add dataset-level or report-level evaluators from the dataset detail page's **Evaluators** tab. + - Run evaluations against it — see [Running Evaluations](evaluations.md). - View and compare experiment results — see [Evals: Datasets & Experiments](../../guides/web-ui/evals.md#viewing-experiments). diff --git a/docs/gateway-migration.md b/docs/gateway-migration.md index 8f309cb27..de17aa8ed 100644 --- a/docs/gateway-migration.md +++ b/docs/gateway-migration.md @@ -3,19 +3,19 @@ title: "Migrating from Pydantic AI Gateway" description: "How to migrate from the legacy gateway.pydantic.dev to the AI Gateway on Pydantic Logfire." --- -# Pydantic AI Gateway is Moving to Pydantic Logfire +# Pydantic AI Gateway Has Moved to Pydantic Logfire -We're consolidating the AI Gateway into Logfire. This means [gateway.pydantic.dev](https://gateway.pydantic.dev/) is being deprecated, and the gateway is now managed through your Logfire account. +The AI Gateway has moved into Logfire. The legacy [gateway.pydantic.dev](https://gateway.pydantic.dev/) platform has reached end of life, and the gateway is now managed through your Logfire account. ## Shutdown Timeline | Date | Event | |------|-------| -| **15 March 2026** | Self-service refunds available in the legacy gateway platform | +| **15 March 2026** | Self-service refunds became available in the legacy gateway platform | | **13 April 2026 at 3pm UTC** | Legacy gateway fully shut down (end of life) | -| **By end of April 2026** | Automatic refunds processed for any remaining balances | +| **By end of April 2026** | Automatic refunds were processed for any remaining balances | -**Please migrate before 13 April 2026.** If you need help, email us at [engineering@pydantic.dev](mailto:engineering@pydantic.dev). +If you still need help after the shutdown, email us at [engineering@pydantic.dev](mailto:engineering@pydantic.dev). ## Why We Made This Change @@ -32,9 +32,7 @@ Moving the gateway into Logfire unlocks a number of improvements: ### What happens to my current balance? -From **15 March 2026**, you can request a refund of your remaining balance via the button in the [legacy gateway platform](https://gateway.pydantic.dev). The refund will be issued to the original payment method you used. - -If you do not request a refund manually, any outstanding credits will be refunded automatically before the end of April 2026. +Self-service refunds became available on **15 March 2026** in the [legacy gateway platform](https://gateway.pydantic.dev). Any outstanding credits that were not requested manually were scheduled for automatic refund before the end of April 2026. ### Do I need to create a new account? diff --git a/docs/guides/web-ui/alerts.md b/docs/guides/web-ui/alerts.md index c768409ac..8c391b9d4 100644 --- a/docs/guides/web-ui/alerts.md +++ b/docs/guides/web-ui/alerts.md @@ -10,8 +10,8 @@ With **Logfire**, use Alerts to notify you when certain conditions are met. Let's see in practice how to create an alert. -1. Go to the **Alerts** tab in the left sidebar. -2. Click the **Create alert** button. +1. Go to **Notify** → **Alerts** in the left sidebar. +2. Click the **New Alert** button. Then you'll see the following form: @@ -34,10 +34,13 @@ WHERE 1. The `SELECT ... FROM records` statement is the base query that will be executed. The **records** table contains the spans and logs data. `trace_id` links to the trace in the live view when viewing the alert run results in the web UI. 2. The `attributes` field is a JSON field that contains additional information about the record. In this case, we're using the `http.route` attribute to filter the records by route. -The **Time window** field allows you to specify the time range over which the query will be executed. +Use the **Notifications** section to choose: -The **Webhook URL** field is where you can specify a URL to which the alert will send a POST request when triggered. -For now, **Logfire** alerts only send the requests in [Slack format]. +- **Include rows from**: the time window of data included every time the query runs. +- **Run the query**: how often Logfire executes the query. +- **Notify me when**: which result condition sends a notification. + +Select one or more notification channels for delivery. If you have not created a channel yet, go to **Notify** → **Delivery** → **Channels** and click **New channel**. For Slack, create a Slack incoming webhook and choose the Slack channel type. ??? tip "Get a Slack webhook URL" To get a Slack webhook URL, follow the instructions in the [Slack documentation](https://api.slack.com/messaging/webhooks). @@ -84,14 +87,12 @@ Otherwise, you'll see the number of matches highlighted in orange. ![Alerts list with error](../../images/guide/browser-alerts-error.png) -In this case, you'll also receive a notification in the Webhook URL you've set up. +In this case, you'll also receive notifications in the channels you've selected. ## Edit an alert -You can configure an alert by clicking on the **Configuration** button on the right side of the alert. +You can configure an alert by opening it from the alerts list and clicking **Edit Alert**. ![Edit alert](../../images/guide/browser-alerts-edit.png) You can update the alert, or delete it by clicking the **Delete** button. If instead of deleting the alert, you want to disable it, you can click on the **Active** switch. - -[Slack format]: https://api.slack.com/reference/surfaces/formatting diff --git a/docs/guides/web-ui/evals.md b/docs/guides/web-ui/evals.md index 2279340c7..3fee17fd2 100644 --- a/docs/guides/web-ui/evals.md +++ b/docs/guides/web-ui/evals.md @@ -34,9 +34,10 @@ Click a dataset name to open its detail page. The page has tabs for: - **Experiments** --- all evaluation runs against this dataset - **Cases** --- test cases (editable for hosted datasets) -- **Schema** --- input, output, and metadata schemas +- **Evaluators** --- dataset-level and report-level evaluators for hosted datasets +- **Schemas** --- input, output, and metadata schemas -The header shows the dataset name, experiment count, case count, and aggregate pass rate. Use the **Export** button to download cases, the **Edit** button to modify the dataset, or the **`<> SDK`** button to view code snippets for working with this dataset programmatically. +The header shows the dataset name, experiment count, case count, and aggregate pass rate. Use the **Export** button to download cases as JSON, the **Edit** button to modify the dataset, or the **Push dataset from code** snippet to view code for working with this dataset programmatically. If the dataset has no experiments yet, the empty state walks you through the setup: define your schema, add test cases, then run your first experiment from code. @@ -51,6 +52,8 @@ Click any experiment row to see detailed results including: - **Performance metrics** --- duration, token usage, and custom scores - **Evaluation scores** --- detailed scoring from all evaluators +Hosted datasets also include an **Evaluators** tab where you can manage dataset-level and report-level evaluators. To add evaluators to a specific case, edit that case from the **Cases** tab. + ## Comparing Experiments To compare multiple runs side by side: diff --git a/docs/guides/web-ui/live-evals.md b/docs/guides/web-ui/live-evals.md index 73bb6dd74..4775a9f54 100644 --- a/docs/guides/web-ui/live-evals.md +++ b/docs/guides/web-ui/live-evals.md @@ -20,7 +20,7 @@ Each row shows: - **Events** — total number of evaluation events in the window - **Last activity** — when the most recent event arrived -Use the time-window tabs in the page header (**1h**, **6h**, **24h**, **7d**, **30d**) to adjust the range. Narrowing the window never empties the page: a target that was active anywhere in the last 30 days still appears as a silent row so you can see that evaluators are configured even if traffic is quiet right now. +Use the time-window tabs in the page header (**5m**, **1h**, **6h**, **24h**, **7d**, **30d**) to adjust the range. Narrowing the window never empties the page: a target that was active anywhere in the last 30 days still appears as a silent row so you can see that evaluators are configured even if traffic is quiet right now. The column headers are sortable — click **Target**, **Type**, **Events**, or **Last activity** to re-order. The sort state is persisted in the URL so you can share a specific view. diff --git a/docs/guides/web-ui/live.md b/docs/guides/web-ui/live.md index 85249330d..78b9fcb7b 100644 --- a/docs/guides/web-ui/live.md +++ b/docs/guides/web-ui/live.md @@ -10,15 +10,15 @@ The live view is useful for watching what's going on within your application in ## SQL search pane -To search the live view, click `Search your spans` (keyboard shortcut `/`), this opens the search pane: +To search the live view, click `Search & filter with SQL...` (keyboard shortcut `/`), this opens the search pane: ![Search box](../../images/guide/live-view-search.png) ### SQL Search For confident SQL users, write your queries directly here. For devs who want a bit of help, -try the new [Pydantic AI](https://pydantic.dev/docs/ai/overview/) feature which generates a SQL query based on your prompt. -You can also review the fields available and populate your SQL automatically using the `Reference` list, see more on this below. +try **Generate SQL with AI**, which generates a SQL query based on your prompt. +You can also review the fields available and populate your SQL automatically using the `Filter by...` list, see more on this below. **WHERE clause** As the greyed out `SELECT * FROM RECORDS WHERE` implies, you're searching inside the `WHERE` clause of a SQL query. @@ -29,7 +29,7 @@ Note: you can run more complex queries on the [explore screen](explore.md) The records table fields are documented in the [SQL reference](../../reference/sql.md). -You can search for any of them in the `Reference` list: +You can search for any of them in the `Filter by...` list: ![Search box reference](../../images/guide/live-view-reference.png) @@ -37,19 +37,19 @@ If you're not sure where to start, scroll down to the `Start here` for beginner- ![Search box start here](../../images/guide/live-view-start-here.png) -### Ask in Language -> Get SQL +### Generate SQL with AI Write your question in your native language, and the model will convert that question to a SQL query. ![Search box natural language](../../images/guide/live-view-natural-language.png) -This is useful if you're not confident with SQL and/or can't quite remember how to format more complicated clauses. You have the option to create a completely new query with `Get new SQL`, or (if you have some SQL already) modify the existing query with `Modify existing SQL`. +This is useful if you're not confident with SQL and/or can't quite remember how to format more complicated clauses. Open `Generate SQL with AI`, describe what you want to search for, and use `Include existing SQL code` if you want the AI to modify or build on the SQL already in the editor. Under the hood this feature uses an LLM running with [Pydantic AI](https://github.com/pydantic/pydantic-ai). -### Reference +### Filter by -Reference: A list of pre-populated query clauses. Clicking any of the clauses will populate the SQL editor, and (where applicable) you can choose a value from the autopopulated dropdown. +`Filter by...`: A list of pre-populated query clauses. Clicking any of the clauses will populate the SQL editor, and (where applicable) you can choose a value from the autopopulated dropdown. This list gives you a powerful way to rapidly generate the query you need, while simultaneously learning more about all the ways you can search your data. Clicking multiple clauses will add them diff --git a/docs/guides/web-ui/organizations-and-projects.md b/docs/guides/web-ui/organizations-and-projects.md index 24fc046c5..bd6239b4c 100644 --- a/docs/guides/web-ui/organizations-and-projects.md +++ b/docs/guides/web-ui/organizations-and-projects.md @@ -17,7 +17,7 @@ Depending on the user's organization role, they may have implicit access to the ## Which organization type should I use? While you _can_ use your personal org for production use-cases (e.g. if you -are working alone or in a small team), we strongly encourage using a a normal organization if you are working +are working alone or in a small team), we strongly encourage using a normal organization if you are working at a larger company and want to create a more "official" Logfire org for that company. This also means you don't have to share your personal org's projects (which you may wish to keep private) with any colleagues. @@ -62,5 +62,5 @@ Three project roles are available: ### Custom roles and externally managed teams -Organizations using the [entreprise plan](../../enterprise.md) have the ability to create custom roles (on top of the existing ones), as well +Organizations using the [enterprise plan](../../enterprise.md) have the ability to create custom roles (on top of the existing ones), as well as using external identity and access management providers such as [Microsoft Entra ID](https://www.microsoft.com/en-us/security/business/identity-access/microsoft-entra-id). diff --git a/docs/guides/web-ui/prompt-playground.md b/docs/guides/web-ui/prompt-playground.md index 0be055ee3..cc869267f 100644 --- a/docs/guides/web-ui/prompt-playground.md +++ b/docs/guides/web-ui/prompt-playground.md @@ -11,7 +11,7 @@ on the system prompt, and see if you can get better results by tweaking it. Taki title: 'Logfire instrumentation of an agent run' /// -Clicking **Open in Playground** will redirect you to the Agent Playground, prefilled with the agent run data: +Clicking **Open in Playground** will redirect you to the Playground, prefilled with the agent run data: ![Prompt playground](../../images/guide/prompt-playground/prompt_playground.png) @@ -21,6 +21,6 @@ You can also configure which tools are available as well as the [model settings] ## Compared with Prompt Management -The Prompt Playground is for exploratory iteration on a captured run. You tweak the system prompt, user messages, tool calls, or settings on that specific run and immediately re-execute it. +The Playground is for exploratory iteration on a captured run. You tweak the system prompt, user messages, tool calls, or settings on that specific run and immediately re-execute it. If you want a persistent prompt that your application imports from Logfire, use [Prompt Management](../../reference/advanced/prompt-management/index.md) instead. Prompt Management stores prompt templates and versions, plus testing artifacts such as saved scenarios and run history. diff --git a/docs/guides/web-ui/public-traces.md b/docs/guides/web-ui/public-traces.md index 2fc32492e..8c7628f8c 100644 --- a/docs/guides/web-ui/public-traces.md +++ b/docs/guides/web-ui/public-traces.md @@ -22,16 +22,17 @@ When you create a public trace link, anyone with access to the link can view the ![Private button](../../images/public-traces/private-button.png) -4. Configure the link expiration +4. Enter an optional title and choose when the link expires: **3 days**, **1 week**, **1 month**, or **Never** 5. Click **Create** to generate the shareable link ## Managing Public Traces -All public trace links are managed in the **Shared Traces** section under Project Settings. From here you can: +All public trace links are managed in **Settings** → **Project** → **Public traces**. From here you can: - View all active public links with their trace/span IDs - See creation dates and expiration times - Copy existing public links +- Rename public links - Delete public links when no longer needed ## Security Considerations diff --git a/docs/guides/web-ui/saved-searches.md b/docs/guides/web-ui/saved-searches.md index a1fe7ff3c..85e7b85e7 100644 --- a/docs/guides/web-ui/saved-searches.md +++ b/docs/guides/web-ui/saved-searches.md @@ -22,7 +22,7 @@ You'll see a panel where you can enter a query using either: ## 2. Writing and Generating Queries - **Direct SQL**: Type your SQL directly in the editor. Autocomplete and schema hints are available. -- **Generate SQL with AI**: Click the "Generate SQL with AI" option, then describe what you want (e.g., "all exceptions in the last hour"). The AI will convert your description into a SQL query. +- **Generate SQL with AI**: Click the **Generate SQL with AI** option, then describe what you want (e.g., "all exceptions in the last hour"). The AI will convert your description into a SQL query. - You can also enable the **Include existing SQL code** option. This allows the AI to modify or build upon your current SQL query, rather than starting from scratch. ![Natural language to SQL](../../images/guide/browser-saved-searches-natural-language.png) @@ -89,4 +89,4 @@ flowchart TD **Next Steps:** - [Explore more about querying with SQL](explore.md) -- [Learn about dashboards and alerts](dashboards.md), (alerts.md) +- [Learn about dashboards](dashboards.md) and [alerts](alerts.md) diff --git a/docs/how-to-guides/alternative-backends.md b/docs/how-to-guides/alternative-backends.md index 1e0c83a28..0f8633065 100644 --- a/docs/how-to-guides/alternative-backends.md +++ b/docs/how-to-guides/alternative-backends.md @@ -7,11 +7,11 @@ description: Learn how to connect Logfire to any backend that supports OpenTelem **Logfire** uses the OpenTelemetry standard. This means that you can configure the SDK to export to any backend that supports OpenTelemetry. The easiest way is to set the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable to a URL that points to your backend. -This will be used as a base, and the SDK will append `/v1/traces` and `/v1/metrics` to the URL to send traces and metrics, respectively. +This will be used as a base, and the SDK will append `/v1/traces`, `/v1/metrics`, and `/v1/logs` to the URL to send traces, metrics, and logs, respectively. Alternatively, you can use the `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`, `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` and `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` environment variables to specify the URLs for traces, metrics and logs separately. -These URLs should include the full path, including `/v1/traces` and `/v1/metrics`. +These URLs should include the full path, including `/v1/traces`, `/v1/metrics`, or `/v1/logs`. !!! note The data will be encoded using **Protobuf** (not JSON) and sent over **HTTP** (not gRPC). @@ -66,6 +66,6 @@ And this is how a more "complex" trace would look like: ![Jaeger complete trace](../images/guide/jaeger-complete-trace-view.png) ## Other environment variables -If `OTEL_TRACES_EXPORTER` and/or `OTEL_METRICS_EXPORTER` are set to any non-empty value other than `otlp`, then **Logfire** will ignore the corresponding `OTEL_EXPORTER_OTLP_*` variables. This is because **Logfire** doesn't support other exporters, so we assume that the environment variables are intended to be used by something else. Normally you don't need to worry about this, and you don't need to set these variables at all unless you want to prevent **Logfire** from setting up these exporters. +If `OTEL_TRACES_EXPORTER`, `OTEL_METRICS_EXPORTER`, and/or `OTEL_LOGS_EXPORTER` are set to any non-empty value other than `otlp`, then **Logfire** will ignore the corresponding `OTEL_EXPORTER_OTLP_*` variables. This is because **Logfire** doesn't support other exporters, so we assume that the environment variables are intended to be used by something else. Normally you don't need to worry about this, and you don't need to set these variables at all unless you want to prevent **Logfire** from setting up these exporters. See the [OpenTelemetry documentation](https://opentelemetry-python.readthedocs.io/en/latest/exporter/otlp/otlp.html) for information about the other headers you can set, such as `OTEL_EXPORTER_OTLP_HEADERS`. diff --git a/docs/how-to-guides/client-side-feature-flags.md b/docs/how-to-guides/client-side-feature-flags.md index 0197ccfb9..169614ec3 100644 --- a/docs/how-to-guides/client-side-feature-flags.md +++ b/docs/how-to-guides/client-side-feature-flags.md @@ -6,7 +6,7 @@ This guide shows how to set up a **JavaScript/TypeScript web application** using ## Prerequisites -1. **Create your variables** in the Logfire UI (Settings > Variables) and mark them as **external** — see [External Variables and OFREP](../reference/advanced/managed-variables/external.md) +1. **Create your variables** in the Logfire UI (**Runtime > Managed Variables**) and mark them as **External** — see [External Variables and OFREP](../reference/advanced/managed-variables/external.md) 2. **Create an API key** with the `project:read_external_variables` scope — this restricted scope is safe to use in client-side code since it only exposes variables you've explicitly marked as external ## Installation @@ -40,16 +40,18 @@ import { OFREPWebProvider } from '@openfeature/ofrep-web-provider' import { OpenFeature } from '@openfeature/web-sdk' const LOGFIRE_API_KEY = 'your-api-key' // project:read_external_variables scope -const LOGFIRE_API_HOST = 'logfire-api.pydantic.dev' // or your self-hosted API host +const LOGFIRE_API_HOST = 'api-us.pydantic.dev' // use api-eu.pydantic.dev for EU projects const provider = new OFREPWebProvider({ - baseUrl: `https://${LOGFIRE_API_HOST}/v1/ofrep/v1`, + // The OFREP provider appends /ofrep/v1/... to this base URL. + baseUrl: `https://${LOGFIRE_API_HOST}/v1`, fetchImplementation: (input, init) => fetch(input, { ...init, headers: { ...Object.fromEntries(new Headers(init?.headers).entries()), Authorization: `Bearer ${LOGFIRE_API_KEY}`, + 'Content-Type': 'application/json', }, }), }) @@ -159,15 +161,16 @@ import { OFREPWebProvider } from '@openfeature/ofrep-web-provider' import { OpenFeature } from '@openfeature/web-sdk' // Initialize once at app startup -function initFeatureFlags(apiKey: string, apiHost: string) { +function initFeatureFlags(apiKey: string, apiHost = 'api-us.pydantic.dev') { const provider = new OFREPWebProvider({ - baseUrl: `https://${apiHost}/v1/ofrep/v1`, + baseUrl: `https://${apiHost}/v1`, fetchImplementation: (input, init) => fetch(input, { ...init, headers: { ...Object.fromEntries(new Headers(init?.headers).entries()), Authorization: `Bearer ${apiKey}`, + 'Content-Type': 'application/json', }, }), }) @@ -206,11 +209,11 @@ OpenFeature provides SDKs and OFREP providers for many languages. You can use th See the [OpenFeature ecosystem page](https://openfeature.dev/ecosystem) for a full list. -The OFREP endpoint format is the same regardless of client: +The OFREP endpoint format is the same regardless of client. Use `https://api-eu.pydantic.dev` instead for EU-region projects: ``` -POST https:///v1/ofrep/v1/evaluate/flags/{flag_key} -POST https:///v1/ofrep/v1/evaluate/flags +POST https://api-us.pydantic.dev/v1/ofrep/v1/evaluate/flags/{flag_key} +POST https://api-us.pydantic.dev/v1/ofrep/v1/evaluate/flags ``` Both endpoints accept a JSON body with a `context` object containing `targetingKey` and any additional targeting attributes: diff --git a/docs/how-to-guides/convert-to-organization.md b/docs/how-to-guides/convert-to-organization.md index 2530cf7bc..1c11979db 100644 --- a/docs/how-to-guides/convert-to-organization.md +++ b/docs/how-to-guides/convert-to-organization.md @@ -5,8 +5,8 @@ description: Convert your Logfire Personal account to an Organization account. B # How to Convert a Personal Account to an Organization Logfire allows you to convert your personal account into an organization, making it easier to collaborate with a team and manage projects at scale. -Converting to an organization requires selecting a [paid plan](https://pydantic.dev/pricing). You will have _5 days_ to chose a plan once you -converted your account, before the organization gets locked. +Converting to an organization requires selecting a [paid plan](https://pydantic.dev/pricing). You will have _5 days_ to choose a plan once you +convert your account, before the organization gets locked. --- diff --git a/docs/how-to-guides/detect-service-is-down.md b/docs/how-to-guides/detect-service-is-down.md index 6fd7ffb22..1777f7b9e 100644 --- a/docs/how-to-guides/detect-service-is-down.md +++ b/docs/how-to-guides/detect-service-is-down.md @@ -36,7 +36,7 @@ for a certain amount of time. ## Create the Alert -Go to [your alerts tab](https://logfire.pydantic.dev/-/redirect/latest-project/alerts/) and click on "New Alert". +Go to **Notify** → **Alerts** in your project and click **New Alert**. Then add the following query to the alert: ```sql @@ -54,8 +54,8 @@ WHERE This query will return `backend is down` if the `/health` endpoint on the `'backend'` service is not called. -On the "Alert Parameters", we want to be notified as soon as possible, so we should execute the query `"every minute"`, -include rows from `"the last minute"`, and notify us if `"the query's results change"`. +On the "Alert Parameters", we want to be notified as soon as possible, so we should set **Run the query** to `every minute`, +**Include rows from** to `the last minute`, and **Notify me when** to `the query's results change`. Then you need to set up a channel to send this notification, which can be a Slack channel or a webhook. See more about it on the [alerts documentation](../guides/web-ui/alerts.md). diff --git a/docs/how-to-guides/infrastructure-as-code.md b/docs/how-to-guides/infrastructure-as-code.md index 6e9ce3873..4ee3035b0 100644 --- a/docs/how-to-guides/infrastructure-as-code.md +++ b/docs/how-to-guides/infrastructure-as-code.md @@ -31,7 +31,7 @@ Resources include: - Terraform CLI `1.8+` (per provider docs) - A Logfire API key, passed as `api_key` or `LOGFIRE_API_KEY` -- A Logfire base URL, passed as `base_url` or `LOGFIRE_BASE_URL` +- A Logfire base URL, passed as `base_url` or `LOGFIRE_BASE_URL`, when the provider cannot infer the hosted region from the API key or when you use self-hosted Logfire See [API keys](../reference/advanced/use-api-keys.md) for key creation and scopes. @@ -47,9 +47,9 @@ terraform { } provider "logfire" { - # You can also set LOGFIRE_BASE_URL and LOGFIRE_API_KEY. - base_url = "https://logfire-us.pydantic.dev" - api_key = var.logfire_api_key + # You can also set LOGFIRE_API_KEY. Hosted Logfire base URLs are inferred + # from regional API keys; set base_url explicitly for self-hosted Logfire. + api_key = var.logfire_api_key } resource "logfire_project" "production" { diff --git a/docs/how-to-guides/otel-collector/otel-collector-overview.md b/docs/how-to-guides/otel-collector/otel-collector-overview.md index feca86ccb..ad46a60e9 100644 --- a/docs/how-to-guides/otel-collector/otel-collector-overview.md +++ b/docs/how-to-guides/otel-collector/otel-collector-overview.md @@ -124,7 +124,7 @@ Then you can point the collector at a tool like Jaeger, `otel-tui`, or Grafana T ## Collecting system logs -This example shows how you can use the OpenTelemetry collector to collect systems logs (logs on stdoutt/stderr) from Kubernetes and send them to Logfire. +This example shows how you can use the OpenTelemetry collector to collect system logs (logs on stdout/stderr) from Kubernetes and send them to Logfire. This may be useful as part of a migration to Logfire if you aren't able to immediately edit all of the applications to install the Logfire SDK, although the data you receive won't be as rich as it would be from tracing with the Logfire SDK. This relatively simple example is enough in many cases to replace existing systems like ElasticSearch, Loki or Splunk. @@ -383,7 +383,7 @@ spec: name: otel-collector-config ``` -Apply this configuration via `kubectl apply -f otel-collector.yaml`. +Apply this configuration via `kubectl apply -f collector.yaml`. You should now see logs from the `plain-app` and `json-app` in your Logfire dashboard! @@ -443,7 +443,7 @@ metadata: subjects: - kind: ServiceAccount name: otel-collector - namespace: otel-collector + namespace: default roleRef: kind: ClusterRole name: otel-collector @@ -473,42 +473,43 @@ data: http: endpoint: "0.0.0.0:4318" processors: - # by default the connection IP is used to match data with k8s object - # when using, for example, a daemonset to send logs to a gateway - # you can use `pod_association` to configure which fields to use for matching. - pod_association: - - sources: - - from: resource_attribute - name: k8s.pod.uid - # If you're using a namespaced RBAC, you'll need to set this filter - # filter: - # namespace: default - extract: - metadata: - # the cluster's UID won't be set with the namespaced configuration - - k8s.cluster.uid - - k8s.pod.name - - k8s.pod.uid - - k8s.deployment.name - - k8s.namespace.name - - k8s.node.name - - k8s.pod.start_time - - k8s.replicaset.name - - k8s.replicaset.uid - - k8s.daemonset.name - - k8s.daemonset.uid - - k8s.job.name - - k8s.job.uid - - k8s.cronjob.name - - k8s.statefulset.name - - k8s.statefulset.uid - - container.image.name - - container.image.tag - - container.id - - k8s.container.name - - container.image.name - - container.image.tag - - container.id + k8sattributes: + # by default the connection IP is used to match data with k8s object + # when using, for example, a daemonset to send logs to a gateway + # you can use `pod_association` to configure which fields to use for matching. + pod_association: + - sources: + - from: resource_attribute + name: k8s.pod.uid + # If you're using a namespaced RBAC, you'll need to set this filter + # filter: + # namespace: default + extract: + metadata: + # the cluster's UID won't be set with the namespaced configuration + - k8s.cluster.uid + - k8s.pod.name + - k8s.pod.uid + - k8s.deployment.name + - k8s.namespace.name + - k8s.node.name + - k8s.pod.start_time + - k8s.replicaset.name + - k8s.replicaset.uid + - k8s.daemonset.name + - k8s.daemonset.uid + - k8s.job.name + - k8s.job.uid + - k8s.cronjob.name + - k8s.statefulset.name + - k8s.statefulset.uid + - container.image.name + - container.image.tag + - container.id + - k8s.container.name + - container.image.name + - container.image.tag + - container.id exporters: debug: otlphttp: diff --git a/docs/how-to-guides/otel-collector/otel-collector-scrubbing.md b/docs/how-to-guides/otel-collector/otel-collector-scrubbing.md index 18b2b4061..5df1b54a3 100644 --- a/docs/how-to-guides/otel-collector/otel-collector-scrubbing.md +++ b/docs/how-to-guides/otel-collector/otel-collector-scrubbing.md @@ -108,12 +108,13 @@ receivers: processors: # First, do simple key-based scrubbing/removal. attributes: - - key: session_id - action: update - value: "[Scrubbed due to session_id]" - - key: user_token - action: update - value: "[Scrubbed due to user_token]" + actions: + - key: session_id + action: update + value: "[Scrubbed due to session_id]" + - key: user_token + action: update + value: "[Scrubbed due to user_token]" # Next, find and mask any PII values we missed. redaction: @@ -152,4 +153,4 @@ service: Now you should have a clearer sense of what's possible using the OpenTelemetry Collector processors for data scrubbing. - Remember, for this scrubbing to work, ensure all telemetry data is only routed through the OTel Collector by setting `logfire.configure(send_to_logfire=False)` +Remember, for this scrubbing to work, ensure all telemetry data is only routed through the OTel Collector by setting `logfire.configure(send_to_logfire=False)` diff --git a/docs/how-to-guides/setup-slack-alerts.md b/docs/how-to-guides/setup-slack-alerts.md index fe5c3d3d8..4aa493fbc 100644 --- a/docs/how-to-guides/setup-slack-alerts.md +++ b/docs/how-to-guides/setup-slack-alerts.md @@ -37,8 +37,8 @@ There are a few ways to create an alert. You can: We'll create an alert that will let us know if any HTTP request takes longer than a second to execute. * Login to **Logfire** and [navigate to your project](https://logfire-us.pydantic.dev/-/redirect/latest-project) -* Click on **Alerts** in the top navigation bar -* Select the **New Alert** button in the top right +* Go to **Notify** → **Alerts** in the left sidebar +* Select the **New Alert** button * Let's give this Alert a name of **Slow Requests** * For the query, we'll group results by the http path and duration. We want to include the **max** duration in a given time frame. We also want to filter out any traces that aren't http requests, and order by the max duration, so we can see which routes are the slowest. This query looks like: ```sql @@ -76,9 +76,9 @@ Let's set up a channel, then test that alerts can be sent with the URL: * Select **New channel** to open the New Channel dialog * Put in a name such as **Logfire Alerts**. This does need to be the name of your Slack channel -* Select **Slack** as the format +* Select **Slack** as the type * Paste in your Webhook URL from the Slack [Apps Management Dashboard] (https://api.slack.com/apps) -* Click on **Send a test alert** and check that you can see the alert in Slack. +* Click **Send test** and check that you can see the alert in Slack. * Click **Create Channel** to create the channel and close the dialog * Click the checkbox next to your new channel to select it diff --git a/docs/how-to-guides/sso-setup.md b/docs/how-to-guides/sso-setup.md index ff5aa9761..a4e364e0c 100644 --- a/docs/how-to-guides/sso-setup.md +++ b/docs/how-to-guides/sso-setup.md @@ -1,13 +1,13 @@ --- title: "SSO Setup (Enterprise Cloud)" -description: "Step-by-step guide to configure Single Sign-On (SSO) for Logfire Enterprise Cloud. Supports Okta, Microsoft Azure Entra ID, Keycloak, and any OIDC-compatible provider." +description: "Step-by-step guide to configure Single Sign-On (SSO) for Logfire Enterprise Cloud. Supports Okta, Microsoft Azure Entra ID, and Keycloak." --- # SSO Setup -Logfire Enterprise Cloud supports Single Sign-On (SSO) via [OIDC-compatible](https://openid.net/developers/how-connect-works/) identity providers, including Okta, Microsoft Azure Entra ID, and Keycloak. Under the hood, Logfire uses [Dex](https://github.com/dexidp/dex), an open-source OIDC gateway. +Logfire Enterprise Cloud supports Single Sign-On (SSO) via managed [OIDC-compatible](https://openid.net/developers/how-connect-works/) identity providers. The currently supported provider types are Okta, Microsoft Azure Entra ID, and Keycloak. Under the hood, Logfire uses [Dex](https://github.com/dexidp/dex), an open-source OIDC gateway. -This guide uses **Microsoft Azure Entra ID** as an example, but the general steps — registering an OIDC app, obtaining a Client ID, Client Secret, and Issuer URL, then connecting it in Logfire — apply to any supported provider. +This guide uses **Microsoft Azure Entra ID** as an example, but the general steps — registering an OIDC app, obtaining a Client ID, Client Secret, and Issuer URL, then adding it in Logfire — also apply to Okta and Keycloak. !!! note "Enterprise Cloud Required" SSO is available exclusively on the **Enterprise Cloud** plan. Ensure your organization has Enterprise Cloud enabled before proceeding. [Contact sales](mailto:sales@pydantic.dev) if you need to upgrade. @@ -29,7 +29,7 @@ This guide uses **Microsoft Azure Entra ID** as an example, but the general step 1. Log in to Logfire and switch to your **Enterprise Cloud organization**. 2. Go to **Settings** in the left-hand menu. -3. Scroll down to the **Identity Providers** section. +3. Scroll down to the **Organization identity providers** section. 4. Note the **Redirect URI** shown — you will need this when configuring the Azure app. --- @@ -67,8 +67,8 @@ From your app registration, gather the following: ## Step 5: Configure the OIDC Provider in Logfire -1. Return to **Logfire** → **Organization Settings** → **Identity Providers**. -2. Click **Add OIDC Provider** and select **Azure** (Microsoft Entra ID). +1. Return to **Logfire** → **Organization Settings** → **Organization identity providers**. +2. Click **Add OIDC provider** and select **Azure** (Microsoft Entra ID). 3. Fill in the fields: - **Client ID**: your Azure Client ID - **Client Secret**: your Azure Client Secret @@ -78,11 +78,9 @@ From your app registration, gather the following: --- -## Step 6: Connect Entra ID +## Step 6: Enable the Identity Provider -After submitting, click the **Connect** button next to the Entra ID provider. - -A request will be sent to your Azure admin for approval. The Azure admin should approve this in the Entra ID admin center. Once approved, the identity provider status will update to **Linked**. +After submitting, confirm that the provider appears in the **Organization identity providers** list with status **Enabled**. If your Azure tenant requires admin consent, complete that consent in the Entra ID admin center before testing user login. --- @@ -118,7 +116,7 @@ During the transition, existing login methods (e.g., Google, GitHub) remain acti Once your team has successfully migrated to Entra ID SSO: -- You can **disconnect** individual login methods from **Organization Settings** → **Identity Providers**. +- You can **disable** or delete individual organization identity providers from **Organization Settings** → **Organization identity providers**. - Advise team members to use the SSO login URL going forward. If other providers are still enabled, users may inadvertently log in with their personal accounts instead. ### Linking Accounts for Existing Users @@ -144,7 +142,7 @@ Users who joined the organization before SSO was configured need to connect thei | 3 | Generate a Client Secret in Azure | | 4 | Collect Client ID, Client Secret, and Tenant ID | | 5 | Add Azure OIDC provider in Logfire with Issuer URL `https://login.microsoftonline.com/{tenant-id}/v2.0` | -| 6 | Connect Entra ID and approve the request in Azure | +| 6 | Confirm the provider is enabled and complete Azure admin consent if your tenant requires it | | 7 | Test SSO login via your region's URL: `https://logfire-us.pydantic.dev/login/{org-name}` or `https://logfire-eu.pydantic.dev/login/{org-name}` | | 8 | Share the invite link with your team (redirects to SSO login if unauthenticated) | diff --git a/docs/index.md b/docs/index.md index c27530cf7..fed2f7a49 100644 --- a/docs/index.md +++ b/docs/index.md @@ -121,7 +121,7 @@ logfire auth ``` !!! info "" - Running this command stores a Write Token used by the SDK to send data to a file in the current directory, at `.logfire/logfire_credentials.json` + The SDK reads `LOGFIRE_TOKEN` directly from the environment. If you configure a project with `logfire projects use` instead, the CLI writes credentials to `.logfire/logfire_credentials.json` in the current directory. 3. Write some basic logs in your Python app diff --git a/docs/integrations/index.md b/docs/integrations/index.md index 38f01b88a..19c9dd1a4 100644 --- a/docs/integrations/index.md +++ b/docs/integrations/index.md @@ -29,12 +29,12 @@ If a package you are using is not listed in this documentation, please let us kn **Logfire** has documented integrations with many technologies, including: -- _LLM Clients and AI Frameworks_: Pydantic AI, OpenAI, Anthropic, LangChain, LlamaIndex, Mirascope, LiteLLM, Magentic +- _LLM Clients and AI Frameworks_: Pydantic AI, OpenAI, Anthropic, Google Gen AI, LangChain, LlamaIndex, Mirascope, LiteLLM, DSPy, MCP, Claude Agent SDK, Magentic - _Web Frameworks_: FastAPI, Django, Flask, Starlette, AIOHTTP, ASGI, WSGI - _Database Clients_: Psycopg, SQLAlchemy, Asyncpg, PyMongo, MySQL, SQLite3, Redis, BigQuery - _HTTP Clients_: HTTPX, Requests, AIOHTTP - _Task Queues and Schedulers_: Airflow, FastStream, Celery -- _Logging Libraries_: Standard Library Logging, Loguru, Structlog +- _Logging Libraries_: Standard Library Logging, Loguru, Structlog, `print()` - _Testing_: Pytest - and more, such as Stripe, AWS Lambda, and system metrics. @@ -52,19 +52,24 @@ The below table lists these integrations and any corresponding `logfire.instrume | [Asyncpg](databases/asyncpg.md) | Database | [`logfire.instrument_asyncpg()`][logfire.Logfire.instrument_asyncpg] | | [BigQuery](databases/bigquery.md) | Database | N/A (built in, no config needed) | | [Celery](event-streams/celery.md) | Task Queue | [`logfire.instrument_celery()`][logfire.Logfire.instrument_celery] | +| [Claude Agent SDK](llms/claude-agent-sdk.md) | AI Framework | [`logfire.instrument_claude_agent_sdk()`][logfire.Logfire.instrument_claude_agent_sdk] | | [Django](web-frameworks/django.md) | Web Framework | [`logfire.instrument_django()`][logfire.Logfire.instrument_django] | +| [DSPy](llms/dspy.md) | AI Framework | [`logfire.instrument_dspy()`][logfire.Logfire.instrument_dspy] | | [FastAPI](web-frameworks/fastapi.md) | Web Framework | [`logfire.instrument_fastapi()`][logfire.Logfire.instrument_fastapi] | | [FastStream](event-streams/faststream.md) | Task Queue | N/A (built in, config needed) | | [Flask](web-frameworks/flask.md) | Web Framework | [`logfire.instrument_flask()`][logfire.Logfire.instrument_flask] | +| [Google Gen AI](llms/google-genai.md) | AI | [`logfire.instrument_google_genai()`][logfire.Logfire.instrument_google_genai] | | [HTTPX](http-clients/httpx.md) | HTTP Client | [`logfire.instrument_httpx()`][logfire.Logfire.instrument_httpx] | | [LangChain](llms/langchain.md) | AI Framework | N/A (built-in OpenTelemetry support) | | [LlamaIndex](llms/llamaindex.md) | AI Framework | N/A (requires LlamaIndex OpenTelemetry package) | -| [LiteLLM](llms/litellm.md) | AI Gateway | N/A (requires LiteLLM callback setup) | +| [LiteLLM](llms/litellm.md) | AI Gateway | [`logfire.instrument_litellm()`][logfire.Logfire.instrument_litellm] | | [Loguru](loguru.md) | Logging | See documentation | | [Magentic](llms/magentic.md) | AI Framework | N/A (built-in Logfire support) | | [Mirascope](llms/mirascope.md) | AI Framework | N/A (use mirascope `@with_logfire` decorator) | +| [MCP](llms/mcp.md) | AI Framework | [`logfire.instrument_mcp()`][logfire.Logfire.instrument_mcp] | | [MySQL](databases/mysql.md) | Database | [`logfire.instrument_mysql()`][logfire.Logfire.instrument_mysql] | -| [OpenAI](llms/openai.md) | AI | [`logfire.instrument_openai()`][logfire.Logfire.instrument_openai] | +| [OpenAI](llms/openai.md) | AI | [`logfire.instrument_openai()`][logfire.Logfire.instrument_openai], [`logfire.instrument_openai_agents()`][logfire.Logfire.instrument_openai_agents] | +| [Print](print.md) | Logging | [`logfire.instrument_print()`][logfire.Logfire.instrument_print] | | [Psycopg](databases/psycopg.md) | Database | [`logfire.instrument_psycopg()`][logfire.Logfire.instrument_psycopg] | | [Pytest](pytest.md) | Testing | N/A (built-in plugin, use `pytest --logfire`) | | [PyMongo](databases/pymongo.md) | Database | [`logfire.instrument_pymongo()`][logfire.Logfire.instrument_pymongo] | diff --git a/docs/reference/advanced/managed-variables/external.md b/docs/reference/advanced/managed-variables/external.md index b60550943..485e02f41 100644 --- a/docs/reference/advanced/managed-variables/external.md +++ b/docs/reference/advanced/managed-variables/external.md @@ -8,7 +8,7 @@ There are two fundamentally different ways to read managed variables, and the ch ### Pull-Based (Logfire SDK) -When you use `logfire.variable.get()` (or the `.get()` method on a variable created with `logfire.var()`), the SDK **pulls the full variable configuration** from the server and evaluates it locally: +When you call `.get()` on a variable created with `logfire.var()`, the SDK **pulls the full variable configuration** from the server and evaluates it locally: 1. The SDK fetches all variable definitions, versions, labels, and rollout rules in a single request 2. A background thread polls for updates (or listens via SSE) @@ -26,7 +26,7 @@ The OFREP endpoints take a different approach — every evaluation is a **server 2. The server evaluates the variable using the context and returns only the resolved value 3. The client never sees the full configuration, other versions, or rollout rules -This is less efficient (one network request per evaluation or batch), but the full configuration is **never exposed** to the client. A client would have to brute-force individual keys to discover what variables exist. +This is less efficient (one network request per evaluation or batch), but the full configuration is **never exposed** to the client. Single-flag evaluation returns only the requested variable's resolved value, while bulk evaluation returns resolved values for the variables the token is allowed to access. **Requires:** `project:read_variables` or `project:read_external_variables` scope. @@ -46,7 +46,7 @@ By default, variables are **internal** — they are only accessible with an API | `project:write_variables` | Create, update, and delete variables and variable types | !!! warning "API key scope and SDK access" - An API key with only the `project:read_external_variables` scope **cannot be used with `logfire.variable.get()`** or any of the pull-based SDK variable methods. The SDK's pull-based approach requires `project:read_variables` because it fetches the full configuration. The `project:read_external_variables` scope only grants access to the OFREP evaluation endpoints. + An API key with only the `project:read_external_variables` scope **cannot be used with `.get()` on variables created with `logfire.var()`** or any of the pull-based SDK variable methods. The SDK's pull-based approach requires `project:read_variables` because it fetches the full configuration. The `project:read_external_variables` scope only grants access to the OFREP evaluation endpoints. ### Setting a Variable as External @@ -58,7 +58,7 @@ You can set a variable as external in the Logfire UI when creating a variable (v import httpx httpx.post( - 'https://logfire-api.pydantic.dev/v1/variables/bulk/', + 'https://api-us.pydantic.dev/v1/variables/bulk/', # or https://api-eu.pydantic.dev for EU projects headers={'Authorization': 'Bearer YOUR_API_KEY'}, json=[ { @@ -82,13 +82,15 @@ httpx.post( Logfire exposes managed variables via the OpenFeature Remote Evaluation Protocol (OFREP). These endpoints evaluate variables as feature flags using a targeting context. -**Endpoints (API base URL + paths):** +**Endpoints (regional API base URL + paths):** ```text -POST /v1/ofrep/v1/evaluate/flags/{key} -POST /v1/ofrep/v1/evaluate/flags +POST https://api-us.pydantic.dev/v1/ofrep/v1/evaluate/flags/{key} +POST https://api-us.pydantic.dev/v1/ofrep/v1/evaluate/flags ``` +Use `https://api-eu.pydantic.dev` instead for EU-region projects. The same `/v1/ofrep/v1/...` paths are also available on your regional Logfire app host. + **Request body (single or bulk):** ```json @@ -103,7 +105,7 @@ POST /v1/ofrep/v1/evaluate/flags - `targetingKey` is required and is used for deterministic rollout selection. - Any additional fields in `context` become attributes for conditional rules. -- The OFREP response maps labels to the `variant` field for compatibility with OpenFeature clients. +- The OFREP response maps the selected label or target to the `variant` field for compatibility with OpenFeature clients. For example, the variant can be a label such as `production`, the automatic `latest` target, or `null` when no remote value is resolved. **Caching (bulk endpoint):** @@ -114,9 +116,9 @@ These endpoints require an API key with the `project:read_variables` or `project ### Response Behavior -When the server resolves a variable successfully, the OFREP response includes the resolved `value`, a `variant` (the label name), and `reason: "TARGETING_MATCH"`. +When the server resolves a variable successfully, the OFREP response includes the resolved `value`, a `variant`, and a `reason` such as `"STATIC"`, `"SPLIT"`, or `"TARGETING_MATCH"`. -When no value can be resolved — either because the variable has no versions, or because no label matches the evaluation context (e.g., rollout weights sum to less than 1.0 and no latest version fallback exists) — the server returns: +When no remote value can be resolved — for example, because the variable has no versions yet or routing selects **Code default** — the server returns: ```json { diff --git a/docs/reference/advanced/managed-variables/index.md b/docs/reference/advanced/managed-variables/index.md index b5c8174db..124953976 100644 --- a/docs/reference/advanced/managed-variables/index.md +++ b/docs/reference/advanced/managed-variables/index.md @@ -39,11 +39,11 @@ And two labels pointing to those versions: To roll out v3 to everyone, just move the `production` label from v2 to v3. To roll back, move it back to v2. No new versions need to be created — the label is just a pointer. -!!! tip "Code default fallback" - If no labels are configured in the rollout, or if rollout weights sum to less than 1.0, the remaining traffic uses the **code default** (the `default` value passed to `logfire.var()`). To direct remaining traffic to the latest version instead, create a label that references `latest` and include it in your rollout. +!!! tip "Latest and Code default" + In the Logfire UI, every variable has an automatic `latest` entry that points to the most recently created version. New variables start by routing traffic to `latest`, but `latest` has no value until you create the first version. The **Targeting** tab also shows **Code default**, which is the `default` value passed to `logfire.var()` in your application. !!! note "Code default as safety net" - The `default` value you pass to `logfire.var()` serves as an always-available fallback hard-coded into your source code. If no versions have been created yet, or if the remote configuration is unreachable due to a networking issue, or if a remote value fails validation against your type, the SDK returns the code default instead of raising an error. This means your application always has a working value — the remote configuration improves it, but never breaks it. + The `default` value you pass to `logfire.var()` serves as an always-available fallback hard-coded into your source code. If no remote value can be resolved, if the remote configuration is unreachable due to a networking issue, or if a remote value fails validation against your type, the SDK returns the code default instead of raising an error. This means your application always has a working value — the remote configuration improves it, but never breaks it. ## Structured Configuration diff --git a/docs/reference/advanced/managed-variables/ui.md b/docs/reference/advanced/managed-variables/ui.md index 241f2c531..555d77f51 100644 --- a/docs/reference/advanced/managed-variables/ui.md +++ b/docs/reference/advanced/managed-variables/ui.md @@ -1,11 +1,11 @@ # Managing Variables in the Logfire UI -The Logfire web UI provides a complete interface for managing your variables without any code changes. You can find it under **Settings > Variables** in your project. The page includes two tabs: +The Logfire web UI provides a complete interface for managing your variables without any code changes. You can find it under **Runtime > Managed Variables** in your project. The page includes two tabs: - **Variables**: browse, create, and manage your managed variables - **Types**: define reusable JSON schemas for custom variable types -Clicking a variable opens its **detail page**, which has four tabs: **Values**, **Targeting**, **History**, and **Settings**. +Clicking a variable opens its **detail page**, which has **Values**, **Targeting**, **Optimize**, and **History** tabs. Users with write access also see **Settings**. ![Variables list](../images/variables-list.png) @@ -31,17 +31,19 @@ For **Custom Types**, the schema is derived from the type and shown read-only; e The **Values tab** is the primary interface for viewing and editing your variable's content. It combines label management with value editing in a single view. -The left sidebar shows all labels (both active and inactive), while the right panel displays the value for the selected label. Each label in the sidebar shows its name and what it points to (e.g., a version number, another label, or "latest"). +The left sidebar shows the automatic `latest` entry plus all labels (both active and inactive), while the right panel displays the value for the selected entry. Each label in the sidebar shows its name and what it points to (e.g., a version number, another label, `latest`, or **Code default**). -- Select a label in the sidebar to view its current value +- Select `latest` or a label in the sidebar to view its current value - Click the **copy** button to copy the displayed value to your clipboard - Click the **compare** button to diff the selected label's value against another label — useful for reviewing differences between production and staging prompts, for example -- Click **Edit** to modify the value, then **Save new version** to create a new version and assign it to the selected label -- Click **Add label** to create new labels pointing to a specific version, another label, the latest version, or the code default +- Click **Edit** to modify the value, then **Save new version** to create a new version. When `latest` is selected, the new version becomes the latest version. When a label is selected, the new version is assigned to that label. +- Click **Add label** to create new labels pointing to a specific version, another label, `latest`, or **Code default** ![Variable detail values](../images/variable-detail-values.png) -**Labels** are mutable pointers to specific versions. They work like Docker tags or git branch names — you can move them to point at any version at any time. +`latest` is always present and points to the most recently created version. A new variable has no versions yet, so `latest` has no value until you create the first version. + +**Labels** are mutable pointers to specific versions or targets. They work like Docker tags or git branch names — you can move them to point at any version, another label, `latest`, or **Code default** at any time. Common label patterns: @@ -49,8 +51,8 @@ Common label patterns: - **`control`** / **`treatment`**: A/B testing labels - **`stable`** / **`experimental`**: Risk-based labels -!!! note "No labels = code default" - If a variable has no labels configured in its routing, your application serves the code default (the `default` value passed to `logfire.var()`). To serve the latest version instead, create a label that references `latest` and include it in your routing. +!!! note "Code default" + **Code default** is the `default` value passed to `logfire.var()` in your application. Use it when you want some traffic to ignore remote versions and keep using the value from code. ## Browsing Version History @@ -70,13 +72,12 @@ The **History tab** lets you browse all saved versions: ## Configuring Label Routing -The **Targeting tab > Default** section controls what percentage of requests receive each labeled version. The weights are entered as percentages (0–100%) and must sum to 100% or less: +The **Targeting tab > Default** section controls what percentage of requests receive each label, `latest`, or **Code default**. The editable weights are entered as percentages (0-100%) and must sum to 100% or less. **Code default** is the remaining percentage after label and `latest` weights are allocated. - Set `production` to `90` and `canary` to `10` for a 10% canary deployment - Set `control` to `50` and `treatment` to `50` for a 50/50 A/B test -- To send some traffic to the latest version, create a label that references `latest` and include it in the routing — for example, a `latest` label referencing latest at `10` and `control` at `50` sends 10% of traffic to the latest version, 50% to the control label, and the remaining 40% falls back to the code default -- If weights sum to less than 100%, the remaining percentage uses the **code default** -- If no labels are in the routing (empty), all traffic uses the code default +- Set `latest` to `10` and `control` to `50` to send 10% of traffic to the most recently created version, 50% to the control label, and the remaining 40% to **Code default** +- New variables start with `latest` at `100`, so all traffic uses the newest version once one exists ## Targeting with Conditional Rules @@ -100,4 +101,4 @@ For example, to give enterprise customers the production experience: ![Variable detail targeting](../images/variable-detail-routing.png) !!! important "Variable names must match" - The variable name in the UI must exactly match the `name` parameter in your `logfire.var()` call. If they don't match, your application will use the code default instead of the remote configuration. + The variable name in the UI must exactly match the `name` parameter in your `logfire.var()` call. If they don't match, your application will use **Code default** instead of the remote configuration. diff --git a/docs/reference/advanced/use-api-keys.md b/docs/reference/advanced/use-api-keys.md index 812ec7c2b..6a8190545 100644 --- a/docs/reference/advanced/use-api-keys.md +++ b/docs/reference/advanced/use-api-keys.md @@ -6,8 +6,7 @@ description: "Guide on how to create API keys and use them to call Logfire publi **Logfire** provides public APIs that allow you to programmatically manage your organizations, projects, and other resources. To access these APIs, you'll need to create an **API key**. !!! info "Public APIs" - API keys are for accessing the Logfire platform APIs, _not_ for sending telemetry data (traces, logs, metrics). - To send data to Logfire, use [write tokens](../../how-to-guides/create-write-tokens.md). + API keys are primarily for accessing the Logfire platform APIs. Project-scoped API keys can also be granted OTLP scopes for telemetry ingestion or Query API access. Existing [write tokens](../../how-to-guides/create-write-tokens.md) remain supported for sending telemetry. !!! tip "What you can do" **Available to all plans:** @@ -19,6 +18,9 @@ description: "Guide on how to create API keys and use them to call Logfire publi - **Dashboards**: Create, list, update, and delete dashboards - **Notification channels**: Create, list, update, and delete notification destinations (Slack, webhooks, etc.) - **Variables**: Create, read, and update project variables + - **External variables**: Evaluate variables marked as external through OFREP + - **OTLP**: Send telemetry data and query OTLP data with project-scoped keys + - **Gateway proxy**: Proxy AI model requests through a specific project gateway **Enterprise / Self-hosted only:**{ .enterprise } @@ -66,21 +68,16 @@ Organization and project admins can choose whether to create a personal or non-p ## API Key Scopes When creating an API key, set the scope to define which actions the key can perform. -Available scopes depend on whether you're creating an organization or project API key: - -| Scope | Organization API Key | Project API Key | -| ------------------------------------ | -------------------- | --------------- | -| Organization management | ✓ | — | -| Notification channels | ✓ | — | -| Audit logs | ✓ | — | -| SCIM provisioning | ✓ | — | -| Billing usage | ✓ | — | -| Project settings | ✓ | ✓ | -| Write tokens management | ✓ | ✓ | -| Read tokens management | ✓ | ✓ | -| Alerts management | ✓ | ✓ | -| Dashboards management | ✓ | ✓ | -| Variables management | ✓ | ✓ | +Available scopes depend on whether you're creating an organization or project API key. In the UI, organization keys can optionally apply to all projects or to one selected project. Project keys are always tied to the project where you create them. Some scopes, including OTLP and gateway proxy scopes, require a single project and cannot be granted to personal API keys: + +| Scope prefix / scope | Organization API Key | Project API Key | +| ---------------------------------------------------- | -------------------- | --------------- | +| `organization:*` organization management scopes | ✓ | — | +| `project:*` project settings and resource scopes | ✓ | ✓ | +| `project:read_variables` / `project:write_variables` | ✓ | ✓ | +| `project:read_external_variables` | ✓ | ✓ | +| `project:read_otlp` / `project:write_otlp` | ✓ | ✓ | +| `project:gateway_proxy` | ✓ | ✓ | !!! info Select only the scopes your application needs to follow the principle of least privilege. diff --git a/docs/reference/api/datasets.md b/docs/reference/api/datasets.md index 3375b24f1..89d75469f 100644 --- a/docs/reference/api/datasets.md +++ b/docs/reference/api/datasets.md @@ -16,10 +16,10 @@ For usage examples, see the [SDK Guide](../../evaluate/datasets/sdk.md). | Method | Description | |--------|-------------| | `list_datasets()` | List all datasets in the project. | -| `get_dataset(id_or_name, input_type, output_type, metadata_type, *, include_cases)` | Get a dataset, optionally as a typed [`pydantic_evals.Dataset`][pydantic_evals.Dataset]. Pass `include_cases=False` to get metadata only. | +| `get_dataset(id_or_name, input_type, output_type, metadata_type, *, include_cases, custom_evaluator_types, custom_report_evaluator_types)` | Get a dataset, optionally as a typed [`pydantic_evals.Dataset`][pydantic_evals.Dataset]. Pass `include_cases=False` to get metadata only. Custom evaluator type arguments are used when deserializing hosted evaluator specs. | | `push_dataset(dataset, *, name, description, on_case_conflict)` | Publish a local [`pydantic_evals.Dataset`][pydantic_evals.Dataset] to hosted storage. Creates or updates the hosted dataset, infers schemas from typed `Dataset[...]` generics when available, uploads all cases, and returns metadata only. | -| `create_dataset(name, *, input_type, output_type, metadata_type, description)` | Create a new dataset. Types are converted to JSON schemas automatically. | -| `update_dataset(id_or_name, *, name, input_type, output_type, metadata_type, description)` | Update a dataset's metadata or schemas. | +| `create_dataset(name, *, input_type, output_type, metadata_type, description, evaluators, report_evaluators)` | Create a new dataset. Types are converted to JSON schemas automatically, and evaluator instances are serialized to hosted evaluator specs. | +| `update_dataset(id_or_name, *, name, input_type, output_type, metadata_type, description, evaluators, report_evaluators)` | Update a dataset's metadata, schemas, or dataset/report evaluators. Pass `None` for evaluator arguments to clear hosted evaluators. | | `delete_dataset(id_or_name)` | Delete a dataset and all its cases. | | `list_cases(dataset_id_or_name)` | List all cases in a dataset. | | `get_case(dataset_id_or_name, case_id)` | Get a specific case. | @@ -27,7 +27,7 @@ For usage examples, see the [SDK Guide](../../evaluate/datasets/sdk.md). | `update_case(dataset_id_or_name, case_id, *, name, inputs, expected_output, metadata, evaluators)` | Update an existing case. | | `delete_case(dataset_id_or_name, case_id)` | Delete a case. | -`push_dataset(...)` rejects dataset-level `evaluators` and `report_evaluators` for now, because hosted datasets do not store them yet. Case-level evaluators are still uploaded with their cases. +`push_dataset(...)` uploads case-level evaluators with their cases, and uploads dataset-level `evaluators` and `report_evaluators` on the hosted dataset itself. Each push overwrites the hosted dataset/report evaluator values to match the local [`pydantic_evals.Dataset`][pydantic_evals.Dataset]. An async version, `AsyncLogfireAPIClient`, provides the same methods as async coroutines.