Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -537,7 +537,7 @@ jobs:
run: npx prettier --check .

- name: Lint with ESLint
run: npx eslint src --ext .ts,.tsx --max-warnings 0
run: npm run lint -- --max-warnings 0

- name: Type check with TypeScript
run: npx tsc --noEmit
Expand Down
4 changes: 2 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ CHECK_CMD := scripts/run-parallel \
"py test" "uv run --locked pytest -m 'not integration'" \
"ts lint" "$(NPM_OBSERVATORY) run lint" \
"ts format" "$(NPM_OBSERVATORY) run format -- --check ." \
"ts typecheck" "$(NPM_OBSERVATORY) exec tsc -- -p $(OBSERVATORY_DIR)/tsconfig.json --noEmit"
"ts typecheck" "$(NPM_OBSERVATORY) run typecheck"
CORE_REGRESSION_TEST_PATHS ?= tests
CORE_REGRESSION_PYTEST_ARGS ?= --tb=short
QUICKSTART_SMOKE_PYTEST_ARGS ?= --tb=short
Expand Down Expand Up @@ -143,7 +143,7 @@ format-ts:
$(NPM_OBSERVATORY) run format -- --check .

typecheck-ts:
$(NPM_OBSERVATORY) exec tsc -- -p $(OBSERVATORY_DIR)/tsconfig.json --noEmit
$(NPM_OBSERVATORY) run typecheck

check:
@$(CHECK_CMD)
Expand Down
6 changes: 6 additions & 0 deletions conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,12 @@ def reset_test_env(monkeypatch):
"""Reset environment variables before each test."""
monkeypatch.setenv("PHLO_ENV", "test")
monkeypatch.setenv("PHLO_LOG_LEVEL", "DEBUG")
# Provider mutation unit tests need isolated governance settings, not a
# connection to the production-default PostgreSQL settings store.
monkeypatch.setenv("PHLO_OBSERVATORY_SETTINGS_BACKEND", "memory")
settings = sys.modules.get("phlo.plugins.observatory_settings")
if settings is not None:
settings._reset_memory_service()
# Disable DLT telemetry
monkeypatch.setenv("DLT__RUNTIME__DLTHUB_TELEMETRY", "False")
monkeypatch.setenv("DLT_TELEMETRY_DISABLED", "1")
Expand Down
38 changes: 37 additions & 1 deletion docs/reference/generated/http-api.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"title": "Phlo API",
"version": "0.1.0"
},
"endpoint_count": 205,
"endpoint_count": 209,
"endpoints": [
{
"method": "GET",
Expand Down Expand Up @@ -1100,6 +1100,15 @@
"v1 assets"
]
},
{
"method": "POST",
"operation_id": "v1_asset_audit_proposal_test_api_v1_assets__asset_id__audits__proposal_id__test_post",
"path": "/api/v1/assets/{asset_id}/audits/{proposal_id}/test",
"summary": "V1 Asset Audit Proposal Test",
"tags": [
"v1 assets"
]
},
{
"method": "POST",
"operation_id": "v1_asset_backfill_api_v1_assets__asset_id__backfill_post",
Expand Down Expand Up @@ -1172,6 +1181,15 @@
"v1 assets"
]
},
{
"method": "POST",
"operation_id": "v1_asset_exact_row_count_api_v1_assets__asset_id__row_count_post",
"path": "/api/v1/assets/{asset_id}/row-count",
"summary": "V1 Asset Exact Row Count",
"tags": [
"v1 assets"
]
},
{
"method": "GET",
"operation_id": "v1_asset_runs_api_v1_assets__asset_id__runs_get",
Expand Down Expand Up @@ -1793,6 +1811,15 @@
"v1 staging"
]
},
{
"method": "POST",
"operation_id": "v1_table_rollback_api_v1_tables__table_name__rollback_post",
"path": "/api/v1/tables/{table_name}/rollback",
"summary": "V1 Table Rollback",
"tags": [
"v1 assets"
]
},
{
"method": "GET",
"operation_id": "v1_table_schema_history_api_v1_tables__table_name__schema_history_get",
Expand Down Expand Up @@ -1820,6 +1847,15 @@
"v1 usage"
]
},
{
"method": "GET",
"operation_id": "v1_wap_runs_api_v1_wap_runs_get",
"path": "/api/v1/wap/runs",
"summary": "V1 Wap Runs",
"tags": [
"v1 WAP"
]
},
{
"method": "GET",
"operation_id": "health_health_get",
Expand Down
26 changes: 25 additions & 1 deletion docs/reference/generated/settings.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"field_count": 177,
"field_count": 180,
"model_count": 28,
"models": [
{
Expand Down Expand Up @@ -353,6 +353,14 @@
{
"class": "AlertingSettings",
"fields": [
{
"alias": "phlo_alert_consumer_recipients",
"annotation": "dict[str, list[str]]",
"default": "<factory:dict>",
"description": null,
"name": "phlo_alert_consumer_recipients",
"required": false
},
{
"alias": "phlo_alert_email_recipients",
"annotation": "list[str]",
Expand Down Expand Up @@ -393,6 +401,14 @@
"name": "phlo_alert_email_smtp_user",
"required": false
},
{
"alias": "phlo_alert_owner_recipients",
"annotation": "dict[str, list[str]]",
"default": "<factory:dict>",
"description": null,
"name": "phlo_alert_owner_recipients",
"required": false
},
{
"alias": "phlo_alert_pagerduty_key",
"annotation": "str | None",
Expand All @@ -401,6 +417,14 @@
"name": "phlo_alert_pagerduty_key",
"required": false
},
{
"alias": "phlo_alert_qa_email_recipients",
"annotation": "list[str]",
"default": "<factory:list>",
"description": null,
"name": "phlo_alert_qa_email_recipients",
"required": false
},
{
"alias": "phlo_alert_slack_channel",
"annotation": "str | None",
Expand Down
6 changes: 5 additions & 1 deletion docs/reference/http-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ The API reports version `0.1.0` in OpenAPI. Phase 1 introduces `/api/v1` alongsi

`/health` is public. The security manifest classifies every other route as protected. `/api/v1` always requires identity and authorization; legacy authorization can be optional outside regulated and production modes. See [Authentication and access](auth-and-access.md) for configuration.

## Endpoints (205)
## Endpoints (209)

| Method | Path | Summary | Tags |
| --- | --- | --- | --- |
Expand Down Expand Up @@ -136,6 +136,7 @@ The API reports version `0.1.0` in OpenAPI. Phase 1 introduces `/api/v1` alongsi
| `POST` | `/api/v1/assets/{asset_id}/audits` | V1 Asset Audit Proposal | `v1 assets` |
| `GET` | `/api/v1/assets/{asset_id}/audits/{proposal_id}` | V1 Asset Audit Proposal Detail | `v1 assets` |
| `POST` | `/api/v1/assets/{asset_id}/audits/{proposal_id}/pull-request` | V1 Asset Audit Proposal Pull Request | `v1 assets` |
| `POST` | `/api/v1/assets/{asset_id}/audits/{proposal_id}/test` | V1 Asset Audit Proposal Test | `v1 assets` |
| `POST` | `/api/v1/assets/{asset_id}/backfill` | V1 Asset Backfill | `v1 assets` |
| `GET` | `/api/v1/assets/{asset_id}/checks` | V1 Asset Checks | `v1 assets` |
| `GET` | `/api/v1/assets/{asset_id}/incident-policy` | Get Asset Incident Policy | `v1 incidents` |
Expand All @@ -144,6 +145,7 @@ The API reports version `0.1.0` in OpenAPI. Phase 1 introduces `/api/v1` alongsi
| `POST` | `/api/v1/assets/{asset_id}/materialize` | V1 Asset Materialize | `v1 assets` |
| `GET` | `/api/v1/assets/{asset_id}/preview` | V1 Asset Preview | `v1 assets` |
| `GET` | `/api/v1/assets/{asset_id}/query-usage` | V1 Asset Query Usage | `v1 assets` |
| `POST` | `/api/v1/assets/{asset_id}/row-count` | V1 Asset Exact Row Count | `v1 assets` |
| `GET` | `/api/v1/assets/{asset_id}/runs` | V1 Asset Runs | `v1 assets` |
| `GET` | `/api/v1/assets/{asset_id}/usage` | V1 Asset Usage | `v1 assets` |
| `GET` | `/api/v1/branches` | V1 Branches | `v1 branches` |
Expand Down Expand Up @@ -213,7 +215,9 @@ The API reports version `0.1.0` in OpenAPI. Phase 1 introduces `/api/v1` alongsi
| `GET` | `/api/v1/staging/promotions/candidate` | V1 Staging Candidate | `v1 staging` |
| `POST` | `/api/v1/staging/promotions/candidate/checks` | V1 Staging Checks | `v1 staging` |
| `POST` | `/api/v1/staging/resync` | V1 Staging Resync | `v1 staging` |
| `POST` | `/api/v1/tables/{table_name}/rollback` | V1 Table Rollback | `v1 assets` |
| `GET` | `/api/v1/tables/{table_name}/schema-history` | V1 Table Schema History | `v1 assets` |
| `GET` | `/api/v1/tables/{table_name}/snapshots` | V1 Table Snapshots | `v1 assets` |
| `POST` | `/api/v1/trino/query-completed` | V1 Trino Query Completed | `v1 usage` |
| `GET` | `/api/v1/wap/runs` | V1 Wap Runs | `v1 WAP` |
| `GET` | `/health` | Health | none |
10 changes: 6 additions & 4 deletions docs/reference/observatory-extensions.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

An Observatory extension is trusted Python packaging plus browser code. Install it in the `phlo-api` environment, expose it through the `phlo.plugins.observatory` entry-point group, and restart the API so package metadata can see it.

The API still discovers extensions and serves their manifests, assets, and settings. The current Observatory UI does not load extension browser modules or render contributed routes, navigation, slots, or settings panels. The old browser loader was removed with the old frontend. The contracts below describe the retained API, not a working browser integration.

## Manifest contract

The entry point must load an `ObservatoryExtensionPlugin` instance or a no-argument class. Its `manifest` property returns an `ObservatoryExtensionManifest` or a dictionary accepted by that Pydantic model. Its `asset_root` property returns an `importlib.resources.abc.Traversable` directory.
Expand All @@ -19,13 +21,13 @@ The entry point must load an `ObservatoryExtensionPlugin` instance or a no-argum
| `ui.slots[]` | `slot_id`, `module`, `export`, default export `registerSlot` | Named UI slot contribution |
| `ui.settings[]` | `module`, `export`, default export `registerSettings` | Settings panel contribution |

See the exact [Pydantic models](../../src/phlo/plugins/observatory.py) and the matching [TypeScript API types](../../packages/phlo-observatory/src/phlo_observatory/src/observatory/api/extensions.ts).
See the exact [Pydantic models](../../src/phlo/plugins/observatory.py).

## Assets and module URLs

The API advertises `/api/observatory/extensions/<plugin-metadata-name>/assets` as `assets_base_path`. Asset requests must be non-empty relative POSIX paths. Absolute paths and any `..` segment return HTTP 400; missing files return HTTP 404. The server joins the validated path to `asset_root`, extracts packaged resources when necessary, copies the selected file to a response-lifetime temporary directory, and removes that directory after the response.

Use root-relative module names such as `/example.js`. The browser prefixes non-HTTP module values with the API browser URL and the advertised asset base. An explicit `http://` or `https://` module value remains unchanged. However, the current registry only invokes modules included in Observatory's build-time `import.meta.glob`; an arbitrary remote or API-served URL resolves to an empty module and contributes nothing. This is a current implementation constraint, not a remote-module API. See the [asset endpoint](../../packages/phlo-api/src/phlo_api/observatory_api/extensions.py) and [browser registry](../../packages/phlo-observatory/src/phlo_observatory/src/extensions/registry.tsx).
Use root-relative module names such as `/example.js` in manifests. The API serves assets but does not execute browser modules. No browser registry exists in the current Observatory UI. See the [asset endpoint](../../packages/phlo-api/src/phlo_api/observatory_api/extensions.py).

## Discovery and compatibility

Expand All @@ -41,9 +43,9 @@ The default backend is the durable PostgreSQL capability supplied by `phlo-postg

## Trust and content security policy boundary

Installing an extension grants code execution in two places. Its Python entry point runs inside `phlo-api`, and registered browser modules run with Observatory's origin and user session. Manifests and assets are not a sandbox or an authorisation boundary. Install only packages you trust, pin and review their distributions, and put extension mutations through authenticated, authorised API endpoints.
Installing an extension grants Python code execution inside `phlo-api`. A browser loader would also grant modules access to Observatory's origin and user session, but the current UI has no loader. Manifests and assets are not a sandbox or an authorisation boundary. Install only packages you trust, pin and review their distributions, and put extension mutations through authenticated, authorised API endpoints.

The extension loader does not define or enforce a Content Security Policy (CSP). No extension-specific CSP allowlist exists in the current API or Observatory server. A deployment-level CSP can therefore block extension scripts, while weakening CSP to admit a remote module also expands the trust boundary. Prefer packaged, same-deployment assets and test the effective ingress/server CSP before promotion.
No extension-specific Content Security Policy (CSP) allowlist exists in the current API or Observatory server. Serving an asset does not authorise its execution. Any future browser integration needs a reviewed module-loading and CSP policy.

## Minimal package

Expand Down
4 changes: 2 additions & 2 deletions docs/reference/packages.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,11 +92,11 @@ Provides Nessie catalog integration and catalog CLI commands. Install with `pip

## phlo-observatory

Provides Observatory UI integration, extension loading, run views, and service metadata. Install with `pip install phlo-observatory`. Settings class: `ObservatorySettings`. Service: `observatory`.
Provides the API-backed Observatory UI, run views, Python extension contracts, and service metadata. The old frontend and browser-extension loader have been removed. Install with `pip install phlo-observatory`. Settings class: `ObservatorySettings`. Service: `observatory`.

## Durable run-report support boundary

The Observatory surface provides an authenticated durable per-run report API and UI projection at alpha maturity. The support registry records this capability under `phlo-observatory` and `phlo-api`. Authentication and route authorisation remain configuration and deployment concerns described in [Auth and access](auth-and-access.md).
Phlo API provides an authenticated durable per-run report at alpha maturity. The dedicated report projection in Observatory is pending; pipeline run views are not that durable report. The support registry records this capability under `phlo-observatory` and `phlo-api`. Authentication and route authorisation remain configuration and deployment concerns described in [Auth and access](auth-and-access.md).

## phlo-pandera

Expand Down
5 changes: 4 additions & 1 deletion docs/reference/settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

> Generated by `scripts/generate_reference_docs.py`. Do not edit this page directly.

This inventory contains 177 fields from 28 Pydantic settings models. 0 models could not be introspected.
This inventory contains 180 fields from 28 Pydantic settings models. 0 models could not be introspected.

Defaults come from model metadata. The generator does not instantiate settings or read environment values. Passwords and keys shown here are public development defaults from source code, not deployed secret values; replace them outside local development.

Expand Down Expand Up @@ -84,12 +84,15 @@ Source: `packages/phlo-alerting/src/phlo_alerting/settings.py`

| Field | Alias | Type | Required | Default | Description |
| --- | --- | --- | --- | --- | --- |
| `phlo_alert_consumer_recipients` | `phlo_alert_consumer_recipients` | `dict[str, list[str]]` | false | `<factory:dict>` | none |
| `phlo_alert_email_recipients` | `phlo_alert_email_recipients` | `list[str]` | false | `<factory:list>` | Email recipients for alerts |
| `phlo_alert_email_smtp_host` | `phlo_alert_email_smtp_host` | `str \| None` | false | `none` | SMTP server hostname |
| `phlo_alert_email_smtp_password` | `phlo_alert_email_smtp_password` | `str \| None` | false | `none` | SMTP password |
| `phlo_alert_email_smtp_port` | `phlo_alert_email_smtp_port` | `<class 'int'>` | false | `587` | SMTP server port |
| `phlo_alert_email_smtp_user` | `phlo_alert_email_smtp_user` | `str \| None` | false | `none` | SMTP username |
| `phlo_alert_owner_recipients` | `phlo_alert_owner_recipients` | `dict[str, list[str]]` | false | `<factory:dict>` | none |
| `phlo_alert_pagerduty_key` | `phlo_alert_pagerduty_key` | `str \| None` | false | `none` | PagerDuty Events API v2 integration key |
| `phlo_alert_qa_email_recipients` | `phlo_alert_qa_email_recipients` | `list[str]` | false | `<factory:list>` | none |
| `phlo_alert_slack_channel` | `phlo_alert_slack_channel` | `str \| None` | false | `none` | Default Slack channel for alerts |
| `phlo_alert_slack_webhook` | `phlo_alert_slack_webhook` | `str \| None` | false | `none` | Slack incoming webhook URL |

Expand Down
Loading
Loading