feat(observatory): replace legacy UI in Phlo

[iamgp]

Scope

Replace the legacy frontend with the real-API Observatory inside phlohouse/phlo, preserving the original Observatory design. This is the sole bundled UI for native development, wheels, Docker, and CI. The temporary phlohouse/phlo-observatory repository remains a design reference, not a release destination.

The authenticated proxy forwards the user's token to Phlo API, which verifies it independently. Legacy backend routes and Python extension contracts remain. No secrets, synthetic datasets, installed dependencies, or generated builds are committed. Trino now declares the existing locked SQLGlot parser dependency for mutation classification.

The parity follow-up adds:

• Scoped asset filters, exact executed-SQL prefill, declared column lineage, physical schema history, and revision-bound MFA snapshot rollback.
• Latest, time-range backfill, and dbt root-only full refresh with actual plans, ref/job revisions, downstream selection, confirmation, and idempotency.
• Source-only audit proposals, bounded real Pandera dry-runs, failure policies, signature evidence, date filters, and CSV export. Activation requires a reviewed merge and code reload.
• Pipeline facets, history, timeline, and guarded controls. Exact job selectors preserve membership when production and staging share asset keys.
• Multi-asset incident creation, owner/severity filters, specialized investigations, resolved runs, authorized durable notification/pause effects, and actual query execution-ID pinning. Queries remain read-only Trino.
• Durable operational settings, freshness policies, health observations, and operator-bound Dagster consumers. Unknown evidence remains unknown.
• Protected-main write guards, substantive merge reasons, atomic distinct managed-human MFA review, and real verified signed promotion tags when enabled. Automation retains candidates and reports review-required when it cannot prove independent review.

Deployment requirements and limits

This follow-up includes additive migration 003_incident_query_parity.sql for incident fields, durable query evidence, and effect delivery. API startup applies the incident schema migrations. Migration verification used disposable PostgreSQL only; no shared or production database was changed.

Operators must bind environments and Compose projects, enable intended STOPPED sensors, configure independent pause permissions, and configure signing keys and trust if signing is enabled. A settings save does not grant those permissions. Missing evidence or denied effects fail visibly.

Rows, duration, and compute cost estimates remain unavailable. DLT full refresh has no safe replacement contract and stays blocked. Plans revalidate before launch, not atomically with later code or ref changes. Native backfill acceptance is verified; daemon completion is not claimed. Maintenance target MB controls small-file selection, not rewritten output size. Tagged snapshots always remain protected. Sealed audit chains are never pruned; automatic authenticated chain expiry is not implemented.

Notification delivery uses durable leases and at-least-once recovery. Pause recovery requires the initiating actor's authenticated request. No production environment/ref-scoped DuckDB query provider, browser-extension loader, or dedicated durable run-report projection is added. Single-replica action restrictions remain.

Dagster 1.13.7 collapses duplicate asset-check keys across repositories before its nested pipeline filter. Without proof that check definitions belong uniquely to the selected repository, direct checks fail closed and overview quality remains unknown. Legacy check runs without repository-origin evidence also remain unknown. Repeated unbatched definition reads exposed an upstream PostgreSQL connection leak; this PR reduces query fan-out but does not patch Dagster or change its pool configuration.

Verification

• pytest -q -m 'not integration': 5,412 passed, 4 skipped, 231 deselected on the final integrated changes.
• Fresh pytest-only release harness: 49 passed, 1 platform-specific skip. The shared settings-reset fixture does not import Phlo into jobs that do not install it.
• Disposable PostgreSQL incident, resolution, and usage integration: 8 passed. Real Dagster duplicate-location execution, dbt DuckDB full refresh, and Git/SSH signing were exercised separately.
• Frontend: 33 tests passed; strict types, zero-warning ESLint, Prettier, and production build passed. Existing third-party bundler warnings remain.
• Python source types, repository-wide package-aware lint, complexity and formatting, applicable pre-commit hooks, lock/version drift, and generated references passed.
• Live disposable IoT and Authentik/MFA checks exercised scoped reads, real failing audit dry-runs, Trino queries and durable incident pinning. Desktop and 390px Chromium captures were inspected. Unconfirmed controls stay disabled, and plan changes invalidate confirmation. No control bypass or production write was used.
• Final duplicate-check guard: direct prod/staging checks return actionable 503. Six repeated overviews return 200 with check_definition_scope_unverified and unknown quality. Both layouts show unavailable audit quality, no false All clear, and no horizontal overflow. Scoped previews retain main/dev isolation.
• Source-discovery follow-up: 910 API tests passed. Live prod/staging return all four explicitly tagged DLT ingestion assets, preserving materialization flags and pagination. Desktop and 390px views list the sources; Materialize/Backfill remain enabled. DLT, Airbyte, and Sling regression cases pass. Final-head GitHub checks: 33 passed, 4 skipped, no failures.
• Launch follow-up: accepted launches close the dialog and select the returned run ID. Active selected runs refresh automatically every five seconds, without overlapping page loads, and stop at terminal states or navigation. A real disposable staging run updated from STARTED to SUCCESS without a Refresh click; polling remained stopped for 11 seconds afterward. Rejected launches stayed in the dialog. Desktop and 390px captures were inspected; frontend tests, lint, types, build, and hooks passed.

Landing route

The prerequisites in the stack landing plan, including API #1049 and OAuth #1050, have squash-merged. This PR is now rebased onto origin/main and targets main.

The restack used the original Wave 9 boundary, e15b71c. Obtain passing fresh CI and resolved review conversations before using the squash merge queue. Leave #950, #961, and #970 untouched. Do not merge the temporary repository's PRs.

Keep #1035 open for the remaining product/API gaps. Legacy backend retirement belongs to #1013; multi-replica actions belong to #989. This PR is not a deployment or compliance approval.

UI review follow-up

• Use Tablecn's command-filter interaction with cmdk and Base UI. Keep the existing sample table styling, validate typed server predicates, and preserve applied filters and rows on failure. Compact field rows align metadata and highlight only the active option.
• Use TanStack Query and Virtual for run logs, with cursor, run, environment, and empty-page guards, stable row measurements, explicit retry, and filtered download.
• Preserve query drafts and table previews by environment and principal. Improve asset sorting, exact snapshot-bound row counts, materialization planning, responsive metadata, and keyboard-accessible scroll regions.
• Add a read-only WAP section with environment-correlated Dagster launches, Nessie presence, and manifest-verified lifecycle reports. Missing evidence remains unknown. Temporary references receive no new mutation controls.
• Regenerate the HTTP contract documentation with the WAP endpoint. API and frontend changes must ship together.

Follow-up verification: 97 frontend tests, 971 API tests, and 19 targeted Dagster regressions passed. The full pytest -m "not integration" pre-push suite also passed. Frontend lint, strict types, production build, all commit hooks, and make typecheck-python passed. Live filter checks covered matching rows, editing, reset, zero, null, failure preservation, pending-control locking, keyboard selection, both themes, and 390px layout.

The optional raw pre-push ty-check hook failed with 748 diagnostics because it scans test files as well as sources. This is not a green gate; the source-only Makefile type check passed. Populated WAP lifecycle states remain fixture-tested because the live demo has no WAP launches. Narrow Chromium checks are not physical touch-device verification. No shared deployment or production database write was performed.

Restack verification

The affected backend suite passed with 1,330 tests, one skip, and 19 integration tests deselected. The frontend passed 97 tests, zero-warning lint, strict types, Prettier, and build. Its source remains byte-identical to the reviewed UI head.

Fresh CI caught an undeclared Pandera dependency used by audit dry-runs and expected Dagster warnings in package-isolated inventory tests. The correction declares the runtime dependency, asserts those exact warnings, and uses Dagster's test instance helper to reap code servers. The repository's warnings-as-errors policy remains unchanged.

After the correction, separate pytest -q -m 'not integration' runs passed all 976 API tests and 474 Dagster tests. The focused dependency and inventory regression run passed 10 tests. Ruff, C901, formatting, source types, module headers, lockfile, generated references, and whitespace checks passed. Fresh PR and merge-queue CI remain required before merge.

[coldtea-pr-lens]

◈ PR Lens

Note

This drawing shows 6a69b40, and the branch has new commits since. Tick Redraw to draw the latest one

• Redraw

🟢 +4 new · 🟠 ~1 changed · 🔴 -0 removed · 2 flows · 14 files · commit 6a69b40

Architecture

5 components touched across 6 lanes.

Play the interactive walkthrough

Inside the changed components — 2 views

Component view — Observatory replacement UI

Modules forming the TanStack Start replacement frontend within the Observatory container.

Component view — Runtime UI selector

Startup and switching architecture allowing selection between legacy and replacement UIs.

Data flow

Asset inspection and audit proposal · Staging candidate verification and promotion

Follow each request, response and payload

The other flows — 1 sequence

Staging candidate verification and promotion

View

• Architecture lens
• Data flow lens
• Expand every detail

Tip

Untick Architecture lens or Data flow lens under View to hide a diagram, or tick Expand every detail to open every section. The comment redraws in a few seconds

🪧 More tips

• Run npx skills add coldteadotai/pr-lens, then tell your coding agent: "Diagram the change you just made with PR Lens and attach it to the pull request."
• Run npx @coldtea/pr-lens-cli analyze --base origin/main on a branch, then npx @coldtea/pr-lens-cli render .pr-lens/graph.json. Same lenses, your own model key, before the pull request exists
• Click the link under each diagram to open it on a canvas you can zoom, pan and step through
• The diagrams are links. Click one to open it on the canvas, then press W or click play to walk through the change
• Open a diagram on the canvas, then press W or click play to walk through the change one step at a time
• The CLI's render reads .github/pr-lens.yml and applies your renames, exclusions and lane pins at draw time
• Set github.comment.collapsed: true in .github/pr-lens.yml to fold the comment behind one View architecture and data flow row. Drawing still runs as before
• Set github.draw: on-demand in .github/pr-lens.yml and PR Lens stops drawing pull requests on its own. Comment @pr-lens draw on a pull request when you want that one drawn
• Add .github/workflows/pr-lens.yml with coldteadotai/pr-lens/packages/action@v0 and your model provider's key as its api-key to run PR Lens from your own CI. Any /chat/completions endpoint works
• Push a commit and the drawing stays, with a note that it is out of date. Tick Redraw in the note to draw the new head
• Switch GitHub to dark mode and the diagrams follow. The moving dots are this pull request's data in motion

---

Thanks for using PR Lens! It's built by Coldtea, free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Bluesky
• Mastodon
• Reddit
• LinkedIn