diff --git a/.gitignore b/.gitignore index de8528aed..20abd2ef1 100644 --- a/.gitignore +++ b/.gitignore @@ -168,3 +168,6 @@ clouds.yaml secure.yaml **/clouds.yaml **/secure.yaml + +# Optional local agent context and private research routes +/AGENTS_Private.md diff --git a/AGENTS.md b/AGENTS.md index 8c7a481a8..dbfec6ab2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -72,6 +72,12 @@ The controlling design rule is: ## 2. Grounding and reading +If `AGENTS_Private.md` exists at this repository's root, read it for supplementary +local context and research routes. It is gitignored and optional; if absent, +continue with the public guidance. Use relevant accessible research as background +without treating it as governing authority or copying private material into +public artefacts. + Every agent must read this file before acting. Then use progressive disclosure: 1. Inspect the current repo/worktree and the live authority or evidence surface diff --git a/IMPLEMENTATION_SUMMARY.md b/IMPLEMENTATION_SUMMARY.md deleted file mode 100644 index 3f919405e..000000000 --- a/IMPLEMENTATION_SUMMARY.md +++ /dev/null @@ -1,120 +0,0 @@ -# Chat Exchange Delete Feature - Implementation Summary - -## Overview -Implemented a delete button (✕) UI feature for chat exchanges with confirmation dialogs, significance warnings, and automatic context reload. - -## Changes Made - -### 1. **chatTab.js** - Core Functionality - -#### New Function: `deleteExchange(turnId, isUserMessage)` -- **Location**: Line 166 -- **Purpose**: Handles deletion logic with safety checks -- **Features**: - - Finds message containers by `data-turn-id` attribute - - Calculates total message length (significance threshold: 200+ chars) - - Shows confirmation dialog with optional warning for significant content - - Removes message containers from DOM - - Removes from `transcriptTurns` array - - Reloads previous context if last exchange deleted - - Updates history banners and counts - - Handles both user and Von message types - -#### Modified Function: `appendMessage()` -Added delete buttons to both Von and user message headers: - -**Von Message Delete Button** (Line ~792): -- Shows when turnId is present -- Passes `false` to `deleteExchange()` (indicates Von response) -- Styled with hover effects (gray to red transition) -- Positioned right side of header (margin-left: auto) - -**User Message Delete Button** (Line ~814): -- Shows when turnId is present and not an Error message -- Passes `true` to `deleteExchange()` (indicates user message) -- Same styling as Von delete button -- Positioned right side of header - -**Message Container Tracking** (Lines ~745 and ~873): -- Added `className = 'message-container'` to both Von and user message containers -- Enables DOM querying for message count and deletion - -### 2. **styles.css** - Visual Styling - -#### New Classes (After line 1451): - -```css -/* Delete exchange button styling */ -.btn-delete-exchange { - transition: color 0.15s ease-in-out; - opacity: 0.7; -} - -.btn-delete-exchange:hover { - opacity: 1; -} - -.btn-delete-exchange:focus { - outline: 2px solid rgba(217, 83, 79, 0.5); - outline-offset: -2px; -} -``` - -- Smooth opacity transition on hover -- Red focus outline for accessibility -- Integrates with existing `.btn-mini` base class - -## User Experience - -### Interaction Flow -1. User hovers over Von response or user message header → Delete button (✕) appears -2. User clicks delete button -3. **Confirmation dialog** appears: - - Simple message: "Delete this Von response?" or "Delete this user message?" - - **For significant content** (>200 chars): Adds warning with character count -4. User confirms deletion -5. **Result**: - - Exchange removed from chat display - - Removed from transcript - - History counts updated - - If last exchange: previous context reloaded automatically -6. User can cancel at any time with the Cancel button in dialog - -### Visual Design -- **Delete button**: Light gray (✕) -- **On hover**: Turns red (#d9534f) for danger visual cue -- **Size**: Small (font-size 1.2em) to avoid clutter -- **Position**: Top-right corner of message header (margin-left: auto) -- **Accessibility**: Focus outline on keyboard navigation - -## Safety Features - -1. **Confirmation Dialog**: All deletes require explicit confirmation -2. **Significance Warning**: Messages with >200 characters trigger a warning with character count -3. **Content Tracking**: Exchanges maintain proper transcript state -4. **History Management**: Automatic reload of previous context if only exchange deleted -5. **Count Updates**: History banners and counts refresh after deletion - -## Testing -- ✅ Frontend tests pass (10/10 tests) -- ✅ No syntax errors in chatTab.js -- ✅ No CSS compilation errors in styles.css -- Ready for manual UI testing in browser - -## Code Quality -- Follows existing code patterns in chatTab.js -- Consistent with project's Vanilla JS architecture -- Proper event handling with stopPropagation() -- No external dependencies added -- Inline styles for quick prototyping (can move to CSS later) - -## Related JIRA Issue -- **Issue Created**: JVNAUTOSCI-806 - Implement Coding_Agent identity concept (assigned to Michael Witbrock) - -## Potential Future Improvements -1. Add undo/redo functionality -2. Add keyboard shortcut (e.g., Delete key) for faster deletion -3. Store deleted exchanges in a temporary "trash" for 24 hours -4. Add analytics tracking for deleted exchanges -5. Add "delete all" functionality with bulk confirmation -6. Persist delete history for audit trail diff --git a/ci/known_test_failures.txt b/ci/known_test_failures.txt index 673093eb6..8ce4f8ef3 100644 --- a/ci/known_test_failures.txt +++ b/ci/known_test_failures.txt @@ -24,7 +24,7 @@ # being red on day one. It is a ratchet in the useful direction: the # accompanying test asserts it never grows. Removing an entry is the only # permitted edit. A guardrail that certifies its own growth is the failure mode -# in docs/engineering/reliability_ratchet_articles_and_cases.md, so raising this +# in the historical Reliability Ratchet case record (Von commit 5b2f8b976069), so raising this # baseline is deliberately not an option. # # Splitting this into "broken" and "needs a database" is worth doing; it would diff --git a/docs/AINotes.md b/docs/AINotes.md deleted file mode 100644 index 86dc0b64c..000000000 --- a/docs/AINotes.md +++ /dev/null @@ -1,160 +0,0 @@ -# AI Notes — Frozen Operational Snapshot - -> **Document status: Frozen tactical snapshot from 23 April 2026; not current -> task, branch, tool, or repository status.** Start with [`AGENTS.md`](../AGENTS.md), -> the [design and engineering document index](design_index.md), live `git` -> state, and live Jira. Revalidate any operational advice below before using it. - -- **Kind:** Tactical notes snapshot -- **Lifecycle:** Frozen -- **Authority:** Historical only -- **State as of:** 2026-04-23 - ---- - -## Focus recorded on 23 April 2026 -- Fix/verify: interaction-session UX issues (see current branch name in git). -- Run focused tests after UI changes: `npm run test:frontend`. - -## Follow-ups recorded on 23 April 2026 -- Add a short regression note here when a bug is fixed (1–2 bullets), then move details to Jira/PR. -- Keep Phase/initiative tracking in Jira rather than in this file. - -## Quick operational notes - -### MCP access (Vontology/VonRAG) -- MCP resources list may only show MongoDB; Vontology/VonRAG expose tools, not resources. -- Validate access with tool calls like `mcp__vontology__get_context` and `mcp__vonrag__rag_list_collections`. - -### Safe `.env` lookup -If you need values from `.env`, prefer the allowlisted helper script instead of reading the file directly: - -- Script: `utilities/get_safe_env.py` -- Example: `pdm run python utilities/get_safe_env.py VON_DB_NAME` - -### Test database hygiene -When running tests that require `VON_DB_NAME=test_von_db`, ensure the environment is reset afterwards. - -### Server safety -Do not auto-start the server as part of automation unless explicitly requested. - -### Git merge workflow -When merging feature branches: **commit → push → checkout main → pull → merge → push main → delete branch**. -Push the feature branch before merging so the remote tracking ref exists; otherwise `git branch -d` warns "not fully merged". - -### Subtask branch base guardrail -- For Jira subtasks, branch from the parent issue branch, not directly from `main`. -- Example: for `JVNAUTOSCI-1087` (subtask of `JVNAUTOSCI-1084`), create/switch with `git checkout -b JVNAUTOSCI-1087-... JVNAUTOSCI-1084/main`. -- Verify ancestry before coding: `git merge-base --is-ancestor JVNAUTOSCI-1084/main HEAD` (exit code must be `0`). -- If the base is wrong and there are no local commits yet: `git rebase JVNAUTOSCI-1084/main`. - -## High-signal references -- Agent rules: `AGENTS.md` -- User-facing overview: `USER_GUIDE.md` -- Engineering notes: `docs/engineering/` -- Archive (historical details): `docs/AINotes_archive.md` - -## Decisions and changes recorded in this snapshot -- Presenter “spoken vs screen” behaviour is documented in Jira (JVNAUTOSCI-894); avoid duplicating the write-up here. -- 2026-04-20: Atlassian MCP transport verification/migration follow-through - - Repo workspace config now ships an Atlassian MCP entry in `.vscode/mcp.json` on `https://mcp.atlassian.com/v1/mcp`. - - Added `scripts/powershell/verify_atlassian_mcp_transport.ps1` to inventory live config surfaces (`~/.codex/config.toml`, workspace `.vscode/mcp.json`, VS Code user `mcp.json`) and optionally run `codex mcp list` / a read-only smoke test. - - `scripts/powershell/setup_atlassian_codex_mcp.ps1` now preserves the rest of the user Codex config and upserts the Atlassian block instead of overwriting the whole file. - - Current Codex expects Atlassian as a native streamable-HTTP config entry (`url = "https://mcp.atlassian.com/v1/mcp"`), not a legacy `npx mcp-remote ...` wrapper, if you want `codex mcp login atlassian` to work. -- 2026-02-08: Atlassian MCP reliability fix - - Root cause: mixed MCP endpoints and overly narrow Copilot model sampling. - - Canonical Atlassian MCP endpoint is now `https://mcp.atlassian.com/v1/mcp` (not `/v1/sse`). - - Keep these aligned when debugging Jira MCP: - - `~/.codex/config.toml` (`[mcp_servers.atlassian].url`) - - workspace `.vscode/mcp.json` (`servers.atlassian.url`) - - VS Code user MCP config (`%APPDATA%\\Code\\User\\mcp.json`, and Insiders equivalent if present) - - Use `scripts/powershell/verify_atlassian_mcp_transport.ps1 -ShowCodexList` before assuming the live machine matches the repo docs. - - If Atlassian tools only appear on some models, widen `chat.mcp.serverSampling` for Atlassian in both workspace and user settings. -- 2026-02-08: Von internal Jira proxy health check - - Internal Jira path (`jira_proxy_mcp.py` -> `mcp_server.py`) uses `ATLASSIAN_EMAIL` + `ATLASSIAN_API_TOKEN` (Basic auth), not Atlassian MCP OAuth. - - Initial symptom was `jira_get_myself` `401`; resolved by rotating `ATLASSIAN_API_TOKEN`, updating `.env`, and restarting Von. - - Current verified state (via `InternalMCPGateway.invoke()`): - - `jira_get_myself`: returns user profile. - - `jira_get_issue` (`JVNAUTOSCI-1086`): returns issue payload (`status=Done`). - - If this regresses: rotate token -> update `.env` -> restart Von -> re-run `jira_get_myself`. -- 2026-04-23: repo-shell Jira/Vontology MCP invocation - - `jira_proxy_mcp.py` now applies repo-root `.env` overrides itself before spawning the Jira MCP subprocess, and `jira_get_auth_config` reports the same effective env view. - - For shell-side verification from the repo root, use: - - `pdm run python utilities/invoke_internal_mcp_tool.py jira_get_auth_config` - - `pdm run python utilities/invoke_internal_mcp_tool.py jira_get_myself` - - This avoids depending on whether the parent PowerShell session happened to inherit `.env`. -- 2026-02-08: WS2 workflow safety envelope (JVNAUTOSCI-1087) - - `ActionRegistry.execute()` now stamps canonical action outcome keys in context (`last_action_failed`, `last_step_ok`, `last_action_error`, etc.) on both success and failure. - - `WorkflowExecutor` and `DurableWorkflowExecutor` now honour explicit `on_failure` transitions instead of always failing fast on action errors. - - Regression anchors: - - `tests/backend/test_e2e_vontology_workflow.py::TestBranchingWorkflowWithMCP::test_on_failure_routes_to_recovery_tool` - - `tests/backend/test_durable_workflow_executor_safety.py` - - `tests/backend/test_mcp_action_handler.py::TestActionOutcomeEnvelope` -- 2026-02-08: WS3 executable discovery/routing hardening (JVNAUTOSCI-1088) - - Discovery now annotates each workflow candidate with: - - `is_executable` - - `executability_reason` (`executable_now` / `graph_incomplete` / `non_executable_design_artifact`) - - confidence + routing eligibility fields - - Discovery payload now separates: - - `candidates` (full ranked set, with reasons) - - `matches` / `routing_matches` (selector-eligible set) - - Orchestrator selector filtering defaults: - - drop non-executable candidates - - drop policy-unsafe candidates (not in current workflow registry) - - Explicit overrides: - - `VON_WORKFLOW_SELECTOR_ALLOW_NON_EXECUTABLE=1` - - `VON_WORKFLOW_SELECTOR_ALLOW_POLICY_UNSAFE=1` - - Regression anchors: - - `tests/backend/test_workflow_discovery_service.py` - - `tests/backend/test_orchestrator_workflow_selector_routing.py` -- 2026-02-08: WS4 MCP introspection reliability (JVNAUTOSCI-1089) - - Root cause found: `chat_introspect` handler returned fields not declared in its MCP `output_schema`, causing `InternalMCPGateway.invoke("chat_introspect", ...)` to fail with `SchemaValidationError`. - - Fix pattern: - - keep `chat_introspect` output schema aligned with handler payload keys - - allow forward-compatible diagnostics fields for introspection snapshots - - expose workflow/introspection surface matrix in tool responses (`workflow_list_definitions`) and docs (`docs/engineering/workflow_mcp_capability_matrix.md`) - - provide actionable `unknown_tool` diagnostics on `mcp_stdio_server` for `workflow_*` and chat introspection tools (classification: `internal_mcp_only_tool`) - - keep workflow/introspection surface tool names centralised in `workflow_surface_capabilities.py` and validated by tests to prevent drift - - add `workflow_mcp_health_check` for read-only gateway-path checks of core workflow MCP tools - - Regression anchors (gateway path, not direct handler only): - - `tests/backend/test_internal_mcp_chat_prompt_tools.py::test_chat_introspect_gateway_invoke_success_path` - - `tests/backend/test_internal_mcp_chat_prompt_tools.py::test_chat_introspect_gateway_invoke_handler_error_path` - - `tests/backend/test_internal_mcp_workflow_tools.py::test_workflow_list_definitions_gateway_invoke_success_path` - - `tests/backend/test_internal_mcp_workflow_tools.py::test_workflow_list_instances_gateway_invoke_error_path` - - `tests/backend/test_internal_mcp_workflow_tools.py::test_workflow_mcp_health_check_gateway_invoke_success_path` - - `tests/backend/test_mcp_stdio_unknown_tool_diagnostics.py` -- 2026-02-08: WS5 durable schedule scalability/correctness (JVNAUTOSCI-1091) - - Replaced broad due-schedule concept scans with a query-first path over `text_relations`: - - primary due filter uses `predicate=#V#next_run_scheduled_for` + relation context `next_run_epoch_ms` - - enabled filter uses `predicate=#V#is_schedule_enabled` + relation context `enabled_bool` - - Write path now persists those context fields when schedules are created/updated (`_set_next_run_timestamp`, `_set_enabled_state`), with UTC-normalised timestamps. - - Kept compatibility fallback for legacy relations that lack context metadata. - - Added scheduler poll telemetry snapshot (`WorkflowScheduler.get_poll_metrics`) including: - - due-set size - - triggered count - - due-lookup latency - - total poll latency - - Regression anchors: - - `tests/backend/test_durable_workflow_system.py::TestScheduleManagement::test_find_due_schedules_avoids_concept_list_scan` - - `tests/backend/test_durable_workflow_system.py::TestScheduleManagement::test_find_due_schedules_legacy_relation_fallback` - - `tests/backend/test_durable_workflow_system.py::TestWorkflowScheduler::test_process_due_schedules_updates_poll_metrics` -- 2026-02-08: WS8 E2E/docs/rollout controls (JVNAUTOSCI-1093) - - Added metadata validation rollout control: - - env var `VON_WORKFLOW_METADATA_VALIDATION_MODE` with modes `enforce` (default), `warn`, `off` - - shared telemetry fields on validation events: `mode` + `enforced` - - If workflow hardening causes unexpected failures in production-like runs: - 1. set mode to `warn`, restart, inspect `workflow_metadata_validation_events` - 2. if still unstable, set mode to `off` as temporary fallback - 3. return to `enforce` after remediation - - Regression anchors: - - `tests/backend/test_orchestrator_write_tool_guard_gateway_e2e.py` - - `tests/backend/test_internal_mcp_workflow_tools.py::test_workflow_schedule_gateway_tools_integrate_with_scheduler` - - `tests/backend/test_workflow_metadata_validation.py` - - `tests/backend/test_durable_workflow_executor_safety.py` - -- 2026-01-06 diary - - Swift blob store: improved OpenStack cloud config error message; fixed `openstacksdk` object-store method signature compatibility; added regression tests (JVNAUTOSCI-878). - - Orchestrator: injected client speech-synthesis snapshot for "what voice?" questions; added regression test. - - Write-tool guardrail: moved write permissioning into a workflow-driven policy (`write_tool_policy`) instead of hard-coded orchestrator logic; added regression test (JVNAUTOSCI-956). -- 2026-01-09 diary - - Chat prompts: treat legacy `#V#von_llm_prompt` as a behaviour prompt so user-specific prompts load in chat and MCP introspection; updated tests; closed JVNAUTOSCI-974. diff --git a/docs/AINotes_archive.md b/docs/AINotes_archive.md deleted file mode 100644 index 4ad501b0c..000000000 --- a/docs/AINotes_archive.md +++ /dev/null @@ -1,181 +0,0 @@ -# AI Notes Archive - -Archived from `docs/AINotes.md` on 2026-01-02. - ---- - -# AI Notes - -## Agent Reminder (Config Lookup) - -If you need values from `.env`, prefer the allowlisted helper script instead of reading the file directly: - -- Script: `utilities/get_safe_env.py` -- Example: `pdm run python utilities/get_safe_env.py VON_DB_NAME` - -It only prints allowlisted keys and redacts secret-like values (tokens, passwords, etc.). - -## Current Status - 1. Integration testing of full Phase 1 flow - 2. Proceed with Phase 2 (UI components and org selector) - 3. Or address any issues/feedback from Phase 1 review - ---- - -## What Was Done -**JVNAUTOSCI-789: UI Integration for Organisation and Role Selection** - -Implemented Phase 2 infrastructure for user-facing organisation switching and persistent membership model: - -#### Deliverable 1: Organisation Membership Service ✅ -- File: `src/backend/services/organisation_membership_service.py` (410 lines) -- Purpose: Vontology-based membership management with memberOf predicates -- Key Functions: - - `create_organisation_membership()` → Creates memberOf relationship + hasRole text relation - - `get_user_memberships()` → Retrieves user's organisations and roles - - `get_organisation_members()` → Lists org members with optional role filtering - - `update_user_role()` → Updates user's role in organisation - - `remove_organisation_membership()` → Removes user from organisation -- Data Model: Uses memberOf edge relationships + hasRole text relations with org context -- Access Control: Integrates with access_control system for permission checking -- Testing: 20 comprehensive unit tests (100% pass rate) covering all functions and error paths - -#### Deliverable 2: Backend API Endpoints ✅ -- Modified: `src/backend/server/routes/von_routes.py` -- Endpoint 1: `POST /api/session/set_organisation` - - Switches user's organisation context in Flask session - - ### What Was Done - - `role_in_org: Optional[str]` → User's role in org (from resolver) - - `namespace: Optional[str]` → Derived composite namespace -- Backward Compatible: All fields optional, existing code unaffected -- Updated: `src/backend/services/concept_service.py::start_interaction_session()` - - Now derives composite namespace using namespace_service - - Reads org_id and role_in_org from Flask session - - Falls back gracefully to user-only namespace if no org context - -#### Deliverable 4: Access Control Extensions ✅ -- Modified: `src/backend/security/access_control.py` -- Enhanced `build_visibility_filter()` function: - - Now reads `organisation_concept_id` from Flask session - - Applies org-specific visibility filters (filters concepts by user AND org) - - Log message: "Including org-specific concepts for org={user_org_id}" - - Maintains backward compatibility with user-only sessions - -#### Deliverable 5: RAG Metadata Integration ✅ -- Modified: `src/backend/services/chat_history_service.py` - - Added `get_session_context()` helper - - `add_message_to_history()` now includes org metadata in RAG documents - - Metadata: user_id, session_id, role, timestamp, type, organisation_concept_id, role_in_org -- Modified: `src/backend/services/rag_sync_service.py` - - `collect_indexed_sessions()` fetches org fields from interaction_sessions - - `sync_one_session()` preserves org metadata when syncing to RAG -- Modified: `src/backend/services/rag_backends/llamaindex_backend.py` - - `query()` method now supports organisation_concept_id filtering - - MetadataFilters include both user_id AND organisation_concept_id - - Implements org-scoped access control at RAG retrieval level -- Modified: `src/backend/integrations/internal_mcp/catalogue.py` - - `_search_knowledge_base()` now passes permissions_context with org_id to RAG query -- Modified: `src/backend/mcp_server/mcp_stdio_server.py` - - search_knowledge_base handler now includes org context -- Modified: `src/backend/mcp_server/mcp_server.py` - - search_knowledge_base endpoint now includes session-based org context to queries - -### Test Results -- **Namespace Service Tests**: 35/35 PASSED ✅ - - Derivation (9 tests) - - Parsing (8 tests) - - Helpers (9 tests) - - Edge cases (9 tests) -- **All Backend Tests**: 62 passed, 3 skipped ✅ - - No regressions - - Chat history segmentation fixed (sys.path issue) - - RAG service tests all passing - -### Git Commits -1. `a8445b0`: Phase 1 Composite Namespace Model + Stub Resolver + Tests (597 insertions) -2. `f15ea96`: Phase 1 RAG Metadata Integration for Org-Scoped Content (130 insertions) -3. `9723a97`: Fix test_chat_history_segmentation.py import path (8 insertions) - -### Design Decisions -1. **Namespace Format**: `#V#{user_id}@{org_id}` (simple, parseable, consistent with #V# naming) -2. **Role Storage**: In metadata/session, not in namespace (allows role changes without namespace change) -3. **Graceful Degradation**: Sessions without org context fall back to user-only namespace -4. **Query Filtering**: AND logic (both user AND org must match for org-scoped queries) -5. **Backward Compatibility**: All new fields optional, no breaking changes -6. **Test Pattern**: sys.path injection in test file (handles pytest context issues) - -### Phase 1 Scope (From JVNAUTOSCI-787) -- ✅ Composite namespace derivation with user@org format -- ✅ Namespace parsing and validation utilities -- ✅ Stub role resolver with hardcoded mappings -- ✅ Session model updates for org and role tracking -- ✅ Access control extensions for org-specific visibility -- ✅ RAG metadata inclusion (org_id and role_in_org) -- ✅ RAG query filtering by organisation -- ✅ Comprehensive test coverage (35 namespace tests) -- ✅ All backend tests passing (62 passed, 3 skipped) - -### Known Limitations & Phase 3 Tasks -- Role mappings currently hardcoded (Phase 3: Database-backed RBAC) -- No UI for organisation selection (Phase 2: UI components) -- No database model for organisation memberships (Phase 2: Vontology integration) -- RAG filtering basic AND logic (Future: More complex permission rules) - -### What's Next -**Phase 2**: UI and Vontology Integration -- Create UI components for organisation selector -- Implement `/api/session/set_organisation` endpoint -- Create Vontology-based organisation model -- Document organisation membership workflows - -**Phase 3**: Production-Ready RBAC -- Replace hardcoded role mappings with database -- Implement granular permission model -- Add role management UI -- Add audit logging for role changes - -## Previous Status (Before Phase 1) -- **Date**: 2025-12-04 -- **Recent Activity**: - - Added quote and space validation to concept name generation (JVNAUTOSCI-760) - - Created reusable `validate_concept_name_for_id()` helper function in utils_vontology.py - - Updated linkification regex to remove spaces from allowed character set (consistent with validation) - - Concept IDs now disallow: quotes (reserved for text boundaries), spaces (use underscores/hyphens) - -## Todo -- [ ] Integration testing of full Phase 1 flow (namespace → access control → RAG queries) -- [ ] Phase 2 implementation: UI components for org selector -- [ ] Phase 2 implementation: Organisation membership Vontology model -- [ ] Phase 3 implementation: Database-backed RBAC system - -## Technical Debt & Refactoring Notes - -### Type Safety Enforcement (CRITICAL - Identified December 2025) - -**Problem:** Multiple instances of `# type: ignore` comments appeared during JVNAUTOSCI-800 cleanup, particularly in scenarios like wrapper pattern usage (`_RestrictedGateway`). These ignores are a symptom of insufficient type enforcement in the codebase. - -**Root Cause:** Some architectural patterns (wrapper classes, duck typing) and legacy code don't leverage proper typing: -- Wrapper classes that delegate to underlying implementations without using Protocol or abstract base classes -- Functions accepting loosely-typed arguments (Any, dict) when more specific types would be appropriate -- Missing TypedDict or dataclass definitions for structured data -- Inconsistent use of Optional/Union vs None coercion - -**Recommendation:** Establish a **type enforcement strategy** for the refactoring backlog: -1. **Enable strict type checking** in Pyright configuration (`pyrightConfig: {"typeCheckingMode": "strict"}` in pyrightconfig.json) -2. **Use Protocol/ABC** for wrapper patterns instead of relying on duck typing and type ignores -3. **Define explicit types** for complex dictionaries (TypedDict or dataclasses) -4. **Replace Any types** with specific union types or protocols where feasible -5. **Document intentional type mismatches** with inline comments (not just `# type: ignore`) - -**Example Fix Pattern:** -- **Before:** `gateway: Any` → cast to InternalMCPGateway → needs `# type: ignore[arg-type]` -- **After:** Define `ManagedGateway` Protocol, implement wrapper against Protocol → type safe, no ignores needed - -**Affected Files Requiring Review:** -- `src/backend/mcp_server/mcp_stdio_server.py` (line 1238: _RestrictedGateway wrapper) -- `src/backend/services/rag_service.py` (dict-based operations) -- `src/backend/integrations/internal_mcp/catalogue.py` (Tool definitions with flexible schemas) -- `src/backend/services/chat_auxiliary_prompt_service.py` (context dict handling) - -**Priority:** Medium - not blocking, but should be addressed in next major refactoring cycle to improve IDE support and prevent subtle type-related bugs. diff --git a/docs/Conversation Turn Workdlow.md b/docs/Conversation Turn Workdlow.md deleted file mode 100644 index 0e6e970c5..000000000 --- a/docs/Conversation Turn Workdlow.md +++ /dev/null @@ -1,236 +0,0 @@ -# Conversation Turn Workflow - -## Purpose -Provide a clear, end-to-end model of what happens from a user submitting a prompt to Von returning a response. This document describes the current behaviour and sequencing, with explicit decision points and outputs so it can serve as a baseline for future workflow design. - -## Actors and Components -- **Frontend chat UI**: collects the prompt, renders the assistant message, manages LLM debug display, and post-processes rendered HTML. See [src/frontend/web/von_interface/static/js/chatTab.js](src/frontend/web/von_interface/static/js/chatTab.js). -- **Backend generate handler**: `/von/generate` orchestrates prompt loading, tool use, presenter handling, and response assembly. See [src/backend/server/routes/von_routes.py](src/backend/server/routes/von_routes.py). -- **Internal MCP orchestrator**: LLM + tool loop and workflow selector. See [src/backend/integrations/internal_mcp/orchestrator.py](src/backend/integrations/internal_mcp/orchestrator.py). -- **Workflow tracing utilities**: execution traces and workflow definitions. See [src/backend/workflows/definitions.py](src/backend/workflows/definitions.py) and [src/backend/workflows/__init__.py](src/backend/workflows/__init__.py). - -## Inputs and Outputs -### Inputs -- User prompt text and client context (user id, org id, language, gmail profile). -- Presenter mode flag (always true from the chat UI). -- Authenticated session/namespace and prompt fragments from ontology. - -### Outputs -- `response` (screen text shown to the user). -- `presenter_channels` (screen + spoken + format metadata). -- `llm_debug` (context stats, tool stats, aux traces). -- `rag_trace` and tool progress metadata (when enabled). - -## Workflow Model (Current State) -### Stage 1: Frontend submit -1. The user enters text and submits the prompt. -2. The UI sends `/von/generate` with `presenter_mode: true`. See [src/frontend/web/von_interface/static/js/chatTab.js](src/frontend/web/von_interface/static/js/chatTab.js#L5640-L5750). -3. A tool-use progress polling loop starts while the request is in flight. See [src/frontend/web/von_interface/static/js/chatTab.js](src/frontend/web/von_interface/static/js/chatTab.js#L120-L250). - -### Stage 2: Backend intake and context assembly -1. `/von/generate` derives the effective user/namespace and loads prompt fragments from the ontology (behaviour, screen, narration). See [src/backend/server/routes/von_routes.py](src/backend/server/routes/von_routes.py#L2050-L2245). -2. The handler builds the enhanced context from history + system prompts, and trims it for size. Tool results can also be truncated for storage. See [src/backend/server/routes/von_routes.py](src/backend/server/routes/von_routes.py#L603-L760) and [src/backend/server/routes/von_routes.py](src/backend/server/routes/von_routes.py#L2800-L2920). - -### Stage 3: Orchestrator run (LLM + tool loop) -1. If the internal MCP orchestrator is available, it is invoked with prompt, context, model, namespace, and auxiliary system prompt. See [src/backend/server/routes/von_routes.py](src/backend/server/routes/von_routes.py#L2910-L3050). -2. The orchestrator may call a workflow selector for authenticated presenter turns, used mainly to decide narration routing. See [src/backend/integrations/internal_mcp/orchestrator.py](src/backend/integrations/internal_mcp/orchestrator.py#L2905-L2965). -3. The orchestrator executes the tool loop: LLM response → strict tool-call JSON parsing → tool invocation → tool result message → repeat until no further tool calls or caps reached. See [src/backend/integrations/internal_mcp/orchestrator.py](src/backend/integrations/internal_mcp/orchestrator.py#L2774-L2925). -4. During post-tool backfill, the orchestrator now runs a **minimal-imposition auto-proceed gate**. If the model says it is continuing (for example “Proceeding now”) and does not request a user decision, the missing-tool-call subworkflow is invoked to continue execution in the same turn. This is controlled by `auto_proceed_minimal_imposition_enabled` (default: enabled). - -### Stage 3a: Ontology discovery context (current reality) -There is **no dedicated ontology discovery pre-flight** today. The orchestrator does not inject a curated list of existing predicate/type concepts before tool planning. This means the LLM must already “know” or guess correct `#V#...` predicate IDs, which increases invalid predicate attempts. - -Existing building blocks that are **not yet wired into tool planning**: -- **Annotations pipeline** (span extraction + candidate concept lookup). The backend can extract spans and candidate concepts via `extract_annotations` + `concept_search_service`, but chat only triggers this when the UI toggle is enabled, and results are not fed into tool selection. See [src/backend/services/annotation_extraction_service.py](src/backend/services/annotation_extraction_service.py) and [src/backend/server/routes/annotations_routes.py](src/backend/server/routes/annotations_routes.py). -- **Salient predicates** (per-instance/type). A cached endpoint exists for `#V#salient_binary_predicate_for_type`, but it is currently used for UI tooling, not tool planning. See [src/backend/server/routes/vontology_routes.py](src/backend/server/routes/vontology_routes.py#L3233-L3760). -- **RAG concept description search**. `search_concept_descriptions` is available via internal MCP, but the default chat flow does not invoke it (except the RAG counts fastpath). See [src/backend/integrations/internal_mcp/catalogue.py](src/backend/integrations/internal_mcp/catalogue.py#L3317-L3580). - -### Stage 4: Presenter protocol handling -1. The model response is parsed for `` and `` tags. See [src/backend/server/routes/von_routes.py](src/backend/server/routes/von_routes.py#L3035-L3120). - - Tags inside fenced code blocks are ignored, so “format examples” do not count as real presenter output. -2. **Screen backfill** runs when tools were used and the screen channel is missing or unsuitable. Heuristics include missing tags, empty screen, tool-dump-like output, or screen identical to spoken. See [src/backend/server/routes/von_routes.py](src/backend/server/routes/von_routes.py#L3050-L3185). - - Backfill order: response text (if safe) → LLM synthesis using tool summaries → deterministic tool summary. See [src/backend/server/routes/von_routes.py](src/backend/server/routes/von_routes.py#L3185-L3335). - - If **no tool messages exist**, screen backfill still runs, but uses a non-tool synthesis prompt (user + model response only). -3. **Spoken backfill / narration** runs if spoken is missing. It uses either a narration workflow (when tracing is enabled) or a local LLM narration prompt. See [src/backend/server/routes/von_routes.py](src/backend/server/routes/von_routes.py#L3360-L3535). - -### Stage 5: Response assembly and persistence -1. The final response is the screen channel (except for debug-only tool-result fallback formats). See [src/backend/server/routes/von_routes.py](src/backend/server/routes/von_routes.py#L3535-L3615). -2. Debug metadata is assembled (context stats, tool stats, presenter health). See [src/backend/server/routes/von_routes.py](src/backend/server/routes/von_routes.py#L3615-L3845). -3. On authenticated turns, user/tool/assistant messages are persisted to session history; unauthenticated flows use in-memory context. See [src/backend/server/routes/von_routes.py](src/backend/server/routes/von_routes.py#L2760-L2890). - -### Stage 6: Frontend render and post-processing -1. The UI renders the assistant message using `presenter_channels.screen`, with optional spoken text stored for TTS. See [src/frontend/web/von_interface/static/js/chatTab.js](src/frontend/web/von_interface/static/js/chatTab.js#L5675-L5745). -2. If markdown rendering is enabled, the rendered HTML is post-processed for quick-reply “buttonify” transforms. See [src/frontend/web/von_interface/static/js/chatTab.js](src/frontend/web/von_interface/static/js/chatTab.js#L1730-L1825). -3. If annotation is enabled, the response is sent for annotation and suggestions are rendered. See [src/frontend/web/von_interface/static/js/chatTab.js](src/frontend/web/von_interface/static/js/chatTab.js#L5695-L5745). - -## Decision Points and Branches -- **Direct tool-call debug mode**: optional path allowing user-provided tool JSON (read-only by default). See [src/backend/server/routes/von_routes.py](src/backend/server/routes/von_routes.py#L2737-L2835). -- **Orchestrator availability**: no orchestrator → single LLM call, no tool loop. See [src/backend/server/routes/von_routes.py](src/backend/server/routes/von_routes.py#L2910-L2995). -- **Tool-call parsing failures**: parsing errors are captured as tool-invocation errors and surfaced in the response. See [src/backend/server/routes/von_routes.py](src/backend/server/routes/von_routes.py#L2990-L3050). -- **Screen/spoken backfill**: only triggers when presenter channels are missing or unsuitable. See [src/backend/server/routes/von_routes.py](src/backend/server/routes/von_routes.py#L3050-L3535). - -## Buttonify (Quick-Reply) Model -When markdown rendering is active, the UI transforms certain patterns into prompt-insert buttons. - -### Trigger points -- Render pipeline: `renderChatMarkdownIntoContainer()` applies transforms after HTML injection. See [src/frontend/web/von_interface/static/js/chatTab.js](src/frontend/web/von_interface/static/js/chatTab.js#L1730-L1785). -- Render toggle: `setVonMessageRenderMode()` re-applies transforms when switching to rendered view. See [src/frontend/web/von_interface/static/js/chatTab.js](src/frontend/web/von_interface/static/js/chatTab.js#L1785-L1825). - -### Transform types -1. Blockquote quoted instruction → button. See [src/frontend/web/von_interface/static/js/chatTab.js](src/frontend/web/von_interface/static/js/chatTab.js#L1472-L1565). -2. Inline strong quoted instruction → button (gated by allowed patterns). See [src/frontend/web/von_interface/static/js/chatTab.js](src/frontend/web/von_interface/static/js/chatTab.js#L1572-L1635). -3. Reply options list → buttons. See [src/frontend/web/von_interface/static/js/chatTab.js](src/frontend/web/von_interface/static/js/chatTab.js#L1323-L1470). -4. List item quoted instruction → button. See [src/frontend/web/von_interface/static/js/chatTab.js](src/frontend/web/von_interface/static/js/chatTab.js#L1619-L1720). - -### Behaviour -- Click inserts text into the prompt. -- Shift-click inserts without auto-send. -- Auto-send uses `submitChatPromptImmediately()`. See [src/frontend/web/von_interface/static/js/chatTab.js](src/frontend/web/von_interface/static/js/chatTab.js#L740-L820). - -### Optional model-driven buttonify -When `VON_BUTTONIFY_MODEL_ENABLE=1`, the backend runs the `buttonify` stage / `#V#chat_buttonify_workflow` to emit structured quick replies in `llm_debug.buttonify.options`. -The live backend path is structured-output-only: there is no heuristic preflight or prose-scraping fallback. If the workflow/LLM path cannot emit valid JSON options, buttonify no-ops with source `none`; successful structured extraction records source `llm`. - -### Canonical response-transformation telemetry contract -All response transformations now emit a canonical, per-turn telemetry payload in `llm_debug.response_transformations` (schema `response_transformations_v1`). - -Each transformation event includes mandatory fields: -- `transform_name` -- `transform_version` -- `status` -- `input_summary` -- `output_summary` -- `options_emitted_count` -- `source_path` -- `latency_ms` -- `model_id` -- `suppression_reason` -- `error_class` -- `timestamp_utc` - -Current required adopters: -- `buttonify` -- `screen_backfill` -- `spoken_backfill` - -The contract is emitted for success, fallback, no-op, skipped, and failure paths. - -## Observability and Trace Artefacts -- **LLM debug payload**: response, tool invocations, tool stats, context stats, presenter metadata. See [src/backend/server/routes/von_routes.py](src/backend/server/routes/von_routes.py#L3615-L3845). -- **Response transformation telemetry**: canonical transformation events at `llm_debug.response_transformations`. -- **Turn execution timing breakdown**: `llm_debug.turn_execution_diagnostics.timing_breakdown` includes per-stage elapsed time, per-stage LLM time, and stage/model LLM duration rows for before/after latency comparisons. -- **Workflow traces**: stored as execution traces and surfaced via `aux_llm_calls` when enabled. See [src/backend/server/routes/von_routes.py](src/backend/server/routes/von_routes.py#L3400-L3465). -- **Tool-use progress**: emits progress updates for the UI “Thinking…” indicator. See [src/backend/server/routes/von_routes.py](src/backend/server/routes/von_routes.py#L603-L705). -- **Model registry snapshot**: summary metadata (source + sample models) attached to `aux_llm_calls` and workflow traces. See [src/backend/integrations/internal_mcp/orchestrator.py](src/backend/integrations/internal_mcp/orchestrator.py#L3668-L3730). - -### Lightweight telemetry introspection by turn -`GET /von/history/debug?session_id=&history_index=&view=transformations` now returns only the response-transformation telemetry payload for that turn (`response_transformations` + `transformations_count`), which is useful for sampled quality audits. - -## Observed Failure Modes (from recent traces) -1. **Screen backfill over-asserts tool outcomes** - - Response-based screen backfill can restate unverified actions when only partial tool evidence exists. -2. **Excess caution blocks trivial writes** - - The model may request manual confirmation for prerequisite checks instead of executing them immediately. -3. **Auto-proceed gate disabled** - - If `auto_proceed_minimal_imposition_enabled` is disabled, “Proceeding now” style responses can intentionally stop after narration and wait for another user turn. -4. **Workflow selector verdicts are non-binding** - - Selector metadata does not enforce subsequent stage execution. -5. **Predicate discovery gap** - - The model may emit invalid predicates (e.g., `has_author`) when no existing predicate ID is surfaced in context. -6. **Annotation results are not used for tool planning** - - Annotation suggestions exist but are not part of the orchestrator’s decision context. -7. **Presenter tags in fenced code blocks** - - If the model prints `/` in a code block, it should be treated as explanatory text, not as valid presenter output. -8. **No-tool presenter response degenerates to spoken-only** - - Without backfill, a presenter response that only provides `` would collapse into a screen-only line, losing structured output. - -## Known Gaps (Foundations for a Formal Workflow) -- **Screen grounding check**: validate screen claims against tool outputs before final response emission. -- **Complexity analysis**: explicit pre-tool assessment with a stored rationale. -- **Tool policy selection**: explicit budget and policy step that gates tool execution. -- **Ontology discovery pre-flight**: deterministic, read-only candidate discovery for types/predicates before tool selection. -- **Multi-turn context persistence**: short-lived cache of discovered concept IDs across follow-up turns. - -## Proposed Workflow: Ordered predicate creation + self-correcting execution -Purpose: ensure **all predicates exist before use**, then execute ontology writes in dependency order, with automatic repair for missing concepts. - -### Workflow stages (high level) -1. **Planner** - - Extract intended assertions and required predicates/types. - - Build a dependency plan (predicates/types before relationships, salient updates before instance links). - -2. **Ontology preflight** - - Resolve candidate concept IDs (predicate/type) by name. - - Run `concept_exists` for each required ID. - - Emit a preflight report with missing/ambiguous IDs. - -3. **Create missing predicates (gated)** - - For each missing predicate, create concept as `instance_of #V#predicate`. - - Add `hasName` (NL + CODE) and `hasDescription` where available. - - Re-check existence for all newly created predicates. - -4. **Execute ordered writes** - - Apply salient predicate updates. - - Create instances. - - Add relationships using verified predicate IDs. - -5. **Verify + summarise** - - Re-fetch key nodes or re-run `concept_exists` checks. - - Summarise tool ledger and final state. - -### Self-correction loop (per step) -- If a write fails due to **missing predicate/type**, re-enter preflight for the missing ID(s), create them if permitted, then **retry only the failed step**. -- If creation is not permitted or ambiguous, **stop and ask the user** for the exact predicate/type to use. - -### Idempotency + safety rules -- All steps must be safe to re-run: creation is gated by `concept_exists`, relationships are added only when predicates are verified. -- Never guess predicate IDs; if a predicate cannot be resolved, halt and request clarification. -- Tool ledger is the authority for the final screen summary. - -## Stage-to-Model Policy Mapping (Proposed) -This mapping introduces **per-stage model policies** without changing current behaviour. It documents *where* a workflow model policy should be queried once implemented. For now, the default policy remains **single-model** (the active LLM) for all stages. - -### Stage: `planner` -- **When**: Before the tool loop begins, as the assistant plans the action/tool sequence. -- **Where**: `InternalMCPChatOrchestrator.run()` before the first LLM call. - -### Stage: `tool_call` -- **When**: Each LLM call that is expected to emit tool calls. -- **Where**: `InternalMCPChatOrchestrator.run()` inside the tool loop. - -### Stage: `tool_recovery` -- **When**: Missing-tool-call detection and JSON repair. -- **Where**: `_missing_tool_call_retry_prompt()` / `_infer_missing_tool_call_retry_tool_calls()` in `orchestrator.py`. - -### Stage: `classifier` -- **When**: Lightweight classification (missing tool call, screen-vs-spoken quality checks, buttonify candidate extraction). -- **Where**: `orchestrator.py` classifier paths and presenter backfill checks. - -### Stage: `critic` -- **When**: Actor–critic checks over tool plans or final summaries (policy compliance, provenance). -- **Where**: Future workflow state after the actor output but before execution/response emission. - -### Stage: `screen_backfill` -- **When**: Presenter screen channel is missing, tool-dump-like, or too similar to spoken. -- **Where**: `_screen_looks_like_tool_dump()` / backfill block in `von_routes.py`. - -### Stage: `narration` -- **When**: Spoken channel generation (narration workflow). -- **Where**: `CHAT_NARRATION_WORKFLOW_ID` execution in `von_routes.py`. - -### Stage: `summariser` -- **When**: Tool-result summarisation and compact response synthesis. -- **Where**: Tool-summary synthesis paths in `von_routes.py`. - -### Stage: `buttonify` -- **When**: Structured quick-reply extraction for UI buttonification. -- **Where**: Before frontend transforms in `chatTab.js` (server-side candidate generation). - -## Compatibility Note -The initial workflow model policy should preserve **current single-model behaviour** (active LLM for all stages), with optional local-only fallback for low-risk classifier stages. No runtime behaviour changes are required until policy resolution is implemented. - -## Glossary -- **Presenter mode**: protocol that outputs `screen` and `spoken` channels for display and TTS. -- **Screen backfill**: second-pass screen synthesis when the model does not provide a valid screen. -- **Orchestrator**: internal MCP loop that runs the LLM and tools. -- **Workflow execution trace**: stored record of workflow execution and status. diff --git a/docs/design_index.md b/docs/design_index.md index 721ef2c57..072f646f6 100644 --- a/docs/design_index.md +++ b/docs/design_index.md @@ -6,7 +6,7 @@ override current user direction, `AGENTS.md`, live represented authority, or live evidence - **Owner:** Von maintainers -- **Last reviewed:** 4 September 2026 +- **Last reviewed:** 17 September 2026 - **Review trigger:** Any change to `AGENTS.md` reading routes, canonical document selection, or document supersession - **Scope:** Tracked design, engineering, operational, review, and generated @@ -32,10 +32,14 @@ documentation-sensitive, or depends on an older design claim. In particular: - treat an unlisted engineering note as supporting material, not as current project-wide authority. -The directory remains mostly flat for now so existing links and external -bookmarks continue to work. Status banners and this virtual organisation are -the first migration step; files can move into archival folders later, with -compatibility stubs, when that produces more value than link churn. +Maintained public manuals, contracts, runbooks and governing design guidance +belong in this repository. Exploratory proposals, architecture reviews, dated +audits, working notes and implementation/experiment records belong in the +private Von research repository. Public instructions must remain self-contained; +private notes do not acquire implementation authority through relocation. + +Historical citations may use commit-pinned public links. These preserve the +previously published evidence without making private access a prerequisite. ## 2. Do not compress document status into one label @@ -80,6 +84,7 @@ live. |---|---| | Every task | [`AGENTS.md`](../AGENTS.md) | | Substantial, architecture-sensitive, or older-doc-dependent work | This index, then the applicable sections below | +| Jira-to-Von task authority and migration recovery | [Jira-to-Von migration guide](engineering/workflow_first_jira_to_von_migration_jvnautosci_1223.md); the current task-project writer controls task updates | | Build/deployment receipts and task work products | [Build and deployment evidence](engineering/deployment_evidence.md) | | Task product selection and continuing a responsibility from Tasks | [Task responsibility continuation](engineering/task_responsibility_continuation.md) | | Subscription-backed coding task pickup on the DGX | [Codex DGX worker pilot](engineering/codex_dgx_worker.md) | @@ -92,11 +97,11 @@ live. | Prompts, models, routing, optimisation, or fine-tuning | [prompt programmes and model routing](engineering/prompt_programs_and_model_routing_playbook.md) | | Retrieval, memory, RAG, KB growth, or long-horizon state | [agent memory and enduring knowledge](engineering/agent_memory_and_enduring_knowledge.md) | | Semantic task search, task embeddings and task RAG context | [Task semantic retrieval](engineering/task_semantic_retrieval.md) | -| Represented, experience- or discussion-derived learning for role, capability, workflow, tool, acquisition, introspection, message, or task decisions | Draft [represented advice design](engineering/represented_advice_design.md), including Phase 0 subtraction, Phase 0.5 non-active candidates, and the [completed bounded negative learning cycle](engineering/jvnautosci_2720_learning_cycle_2026-09-05.md) under JVNAUTOSCI-2720; then the applicable prompt, memory, evaluation, workflow, or security guidance. The cycle links source-derived formation, experimental exposure, outcome evidence, rejection and subsequent verified absence. It does not establish positive advice efficacy, an ordinary runtime consumer, production activation, represented Arm C or autonomous promotion. | | Private Otter archive access, conversation images or research-slide reading | [Private Otter archive and conversation images](engineering/otter_archive_and_conversation_images.md) | | Authenticated Figma access through coding agents; native-backend prerequisites | [Figma authenticated access](engineering/figma_authenticated_access.md) | | Conversational diagrams, equations, generated/tool images or visual format extension | [Conversational visual output](engineering/conversational_visual_output.md) | | Reusing external ontology concepts with source identity and attribution | [KnowKat ontology sources and governed adoption](engineering/knowkat_ontology_sources.md) | +| Experience- or discussion-derived learning | [Role-learning convergence](engineering/role_learning_convergence.md), [memory and enduring knowledge](engineering/agent_memory_and_enduring_knowledge.md), and [evaluation and research uptake](engineering/agent_evaluation_and_research_uptake.md). The current authoritative task surface owns experimental selection and delivery; private proposals are optional background, not public implementation prerequisites. | | Importing external human/agent transcripts into conversation carriers | [External conversation import](engineering/external_conversation_import.md) | | Unified conversation discovery, message exchanges, read state or participant avatars | [Unified conversations and participant profiles](engineering/unified_conversations_and_participant_profiles.md) | | Execution-summary cost display and user/organisation thresholds | [Execution cost display](engineering/execution_cost_display.md) | @@ -142,73 +147,46 @@ from `AGENTS.md`. | [Atlas egress reconciler](engineering/atlas_egress_reconciler_runbook.md) | Active bounded operational runbook | Use for the standalone mobile-client/DGX database access-list reconciler; revalidate Atlas API policy and credentials live | | [Authority-alignment scan](engineering/maintaining_global_design_constraints_and_authority_alignment_with_coding_agents.md) | Active companion guide | Advisory structural review procedure under `AGENTS.md` | -## 6. Programme direction and current synthesis +## 6. Working research and historical evidence + +The 5 September 2026 working-note archive was reconciled with public main on +17 September 2026. Sixty-eight tracked archived documents moved out of the +public working tree; the active Jira-to-Von migration guide remains public. +Von-Private's `research/design_notes/` collection groups proposals, reviews and dated +records, with a source-path and content-hash migration manifest. The private +repository is optional research context, not a dependency of public development. -No formal Architecture Decision Record corpus existed at this review. Do not -mistake a design proposal or Jira implementation summary for an accepted ADR. -Until an ADR process is established, acceptance must be grounded in current -human direction, `AGENTS.md`, and the relevant live Jira decision record. A -design proposal or private synthesis becomes governing only after explicit -human acceptance and promotion into a current public authority surface. +For previously published working records, the +[pre-move public document index](https://github.com/Strong-AI-Lab/Von/blob/5b2f8b976069935da73b82bca433fb3cefffcd3d/docs/design_index.md) +provides historical navigation. Its statements retain their original evidence +boundaries and do not govern current work. Moving a file does not remove its +previously public Git history. + +Remaining public references (retain each document's stated authority): | Document | Classification | Correct use | |---|---|---| -| [Von for Agentic AI](engineering/Von_for_AgenticAI.md) | Advisory long-horizon programme design with an April 2026 state snapshot | Use the durable target direction; revalidate every present-state, file-size, gap, and priority claim | -| Private research syntheses | Advisory material retained outside the public repository | Public coding agents must not depend on private notes; promote approved decisions into the applicable public canonical guide | -| [Represented advice](engineering/represented_advice_design.md) | Draft proposal with completed subtraction, non-active candidate storage and a [bounded negative learning-cycle dossier](engineering/jvnautosci_2720_learning_cycle_2026-09-05.md); live ontology state remains evidence, not a design assumption | Use for experience- and discussion-derived candidate learning, soft-guidance semantics, possible federated projection/lifecycle convergence, bounded autonomous maintenance, ratchet tests, and distinguishing evaluation. Candidate learning, content efficacy, represented-layer value, and production activation are separate claims; no general runtime advice layer is established. | -| [Automated policy learning](engineering/automated_policy_learning_design.md) | Design with partial substrate | Use as a proposed learning architecture, not proof of a closed operational loop | | [Proactive clarification and conversational role learning](engineering/proactive_clarification_and_role_learning.md) | Draft, source-grounded design proposal | Use for material referent questions, alias/role learning, short-reply continuation, recovery and staged evaluation. It builds on existing ordinary-turn support; no runtime behaviour, represented fact or policy activation is established. | | [Conversational rumination](engineering/conversational_rumination.md) | Capability and bounded acceptance record | Optional model-selected background inquiries, independent task execution and later actor-scoped product consumption; no deployment or broad role-competence claim. | -| [Testing workflows and ephemeral theories](engineering/testing_workflows_ephemeral_theories_design.md) | Research/design proposal with partial substrate | Use for design intent and explicit hypotheses; verify implemented surfaces | -| [Reliability Ratchet articles and case log](engineering/reliability_ratchet_articles_and_cases.md) | Active advisory source copy and evidence log | Use as a revisable diagnostic lens and dated case record; not as standing policy, repair authority, or proof of current behaviour | | [Ontology repair plans](engineering/ontology_repair_plan_design.md) | Design with implemented substrate | Use for the plan/approve/execute shape and its affordance argument; it adds no authority route, and agent-reachable ontology mutation remains an open decision | -| [Multi-agent coordination](engineering/multi_agent_coordination_design.md) | Early design proposal | Use as a direction to evaluate, not implemented architecture | | [Coding-agent assignment authority](engineering/coding_agent_assignment_authority_design.md) | Draft / proposed; source audit refreshed 17 September 2026 | Extends existing dispatch provenance with scoped personal/team grants; complete inventory and refreshed private handoff remain separate. No implementation or activation authority | -| [Vontology tooling from KA/KR literature](engineering/vontology_tooling_from_ka_kcap_kr_literature.md) | Research-backed advisory roadmap | Use for alternatives and research uptake, not present capability claims | -## 7. Dated snapshots, diagnostics, and proposals +Current guidance remains listed above, including maintained implementation +references whose filenames contain “design” or “notes”. Classification follows +the document's function, not its filename. Active rehearsal and test protocols +remain public when they support reproducible implementation work. + +Generated tooling documentation and reusable fixtures may remain under +[`docs/generated/`](generated/). Dated research inventories and local evidence +belong with their working notes, subject to the destination's data-handling +rules. Raw Jira, conversation and telemetry payloads must not enter Git merely +because the destination repository is private. - [Mobile Web Push candidate and operator handoff](engineering/mobile_web_push.md) records the bounded JVNAUTOSCI-2757 implementation and unfulfilled browser transport acceptance; it does not establish a deployed notification service. -These files retain evidence and rationale, but must not drive current task -ordering or implementation without live revalidation: - -- [Agentic architecture status — 23 April 2026](engineering/agentic_architecture_status_2026-04-23.md) -- [Recent architecture progress — 21 April 2026](engineering/recent_architecture_progress_2026-04-20.md) -- [Enduring-memory architecture — 23 April 2026](engineering/jvnautosci_1962_enduring_memory_architecture_2026-04-23.md) -- [Self-improvement worlds architecture — 23 April 2026](engineering/jvnautosci_1963_self_improvement_worlds_architecture_2026-04-23.md) -- [Evaluator architecture — 23 April 2026](engineering/jvnautosci_1964_evaluator_architecture_2026-04-23.md) -- [Role-learning review — 24 April 2026](engineering/jvnautosci_2011_role_learning_review_2026-04-24.md) -- [Workflow execution analysis — 7 April 2026](engineering/workflow_execution_architecture_analysis.md) -- [Chat-turn control-plane analysis — 7 April 2026](engineering/chat_turn_workflow_control_plane_analysis.md) -- [Comparative arXiv workflow analysis — 7 April 2026](engineering/arxiv_paper_workflow_analysis.md) -- [Workflow self-authoring gap analysis — 5 April 2026](engineering/workflow_self_authoring_gap_analysis.md) -- [Workflow system audit — 7 February 2026](engineering/workflow_audit_2026-02-07.md) -- [VWL implementation audit — 24 March 2026](engineering/vwl_implementation_audit_2026-03-24.md) -- [MCP reliability incident reflection](engineering/mcp_reliability_analysis.md) -- dated `jvnautosci_*` reviews, audits, designs, and implementation summaries, - unless a current guide above explicitly adopts their conclusions. - -## 8. Historical and generated material - -- [`intro_to_modern_agentic_ai_for_coding_agents.suggested.md`](engineering/intro_to_modern_agentic_ai_for_coding_agents.suggested.md) - is a superseded, unselected proposal. It is not the guide required by - `AGENTS.md`. -- [`AINotes.md`](AINotes.md) is a frozen tactical snapshot, not current task or - repository status. Use live git and Jira instead. -- [Historical agent guidance](engineering/historical_agent_guidance_notes.md) - is an archive and already labels itself accordingly. -- The `JVNAUTOSCI-799` document family records a bounded 2025 milestone. Its - completion, coverage, and readiness claims are not current system-level - evidence. -- Files under [`docs/generated/`](generated/) are generated evidence snapshots. - Read their timestamps and inputs; never treat a generated report as standing - design authority. -- A date in a filename is an evidence boundary, not a freshness guarantee. - -## 9. Metadata for new and substantially revised documents +## 7. Metadata for new and substantially revised documents Use this compact header, adapting fields to the document: @@ -246,7 +224,5 @@ Maintenance rules: 8. Re-check links, code anchors, live Jira, and represented artefacts whenever a substantial task relies on an older document. -This first pass classifies the authority-bearing and highest-risk material. It -does not pretend that all 104 pre-existing engineering files have already -received complete metadata. Until they do, unlisted documents remain supporting -material whose claims require independent verification. +Unlisted documents remain supporting material whose authority and factual +claims require independent verification. diff --git a/docs/engineering/Arxiv_WF_purity.md b/docs/engineering/Arxiv_WF_purity.md deleted file mode 100644 index b88d87f3d..000000000 --- a/docs/engineering/Arxiv_WF_purity.md +++ /dev/null @@ -1,88 +0,0 @@ -# Historical JVNAUTOSCI-1763 Branch Review Transcript - -> **Document status: Historical, unnormalised branch-review transcript.** These -> findings refer to the JVNAUTOSCI-1763 branch and runtime state inspected at -> the time. They are incident provenance, not current architecture or Jira -> status. Re-check every finding against current code, Vontology, telemetry, and -> Jira before acting on it. - -## Code Review: JVNAUTOSCI-1763 Branch vs Jira Requirements and No-Thin-Shim Rule - -Critical Concerns (architectural/rule violations) -1. von_routes.py still owns the arXiv dispatch decision — violates 1766 - -The route handler at von_routes.py explicitly inspects workflow_discovery_result.get("selected_workflow_id") and branches into execute_conversation_turn_supervised() vs orchestrator.run(). This is the route-level bypass that 1766's "ADDITIONAL ANTI-SPECIAL-CASE CONSTRAINT" forbids: - -"It must not introduce an arXiv-only reply-composition architecture, an arXiv-only control-plane contract, or a permanent route-level bypass around generic turn_execution.* surfaces." - -The if is_arxiv_workflow: branch is an arXiv-specific control-plane fork at the route layer. The route should ideally call the Master Turn Workflow for all turns and let the workflow's turn_execution.route step handle the arXiv/non-arXiv dispatch internally. - -2. Narration prompt is hardcoded in Python — violates rules 4/11 of AGENTS.md - -In turn_execution_actions.py:115, _build_turn_execution_narrate_handler() contains a full system_prompt string: - -AGENTS.md rule 11: "Do not place Vontology-governed prompt bodies in Python." JVNAUTOSCI-1766 explicitly says to use #V#prompt_turn_execution_compose_workflow_outcome_report. This prompt should live in a Vontology text relation, not inline in the handler. - -3. Critic and completion gate are not implemented — 1767 claims Done - -turn_execution_actions.py has no handlers registered for turn_execution.critic or turn_execution.completion_gate. The registration function's final comment says: - -JVNAUTOSCI-1767 requires: "completion_gate.decision=completed is blocked on unresolved acquisition/materialisation verification." Without a completion gate handler, nothing prevents the workflow from calling completed regardless of verification status. The arxiv.build_completion_report step always returns status="success" even when verification_status="failed". - -4. execute_conversation_turn_supervised() fallback narration is Python-owned - -In orchestrator.py, lines ~20540-20545 contain a fallback response composition: - -This is the orchestrator composing the user-facing reply from structured data — exactly the "summary-field scraping" pattern 1766 forbids. The reply should come from a workflow-owned narration step, not from the orchestrator's Python fallback. - -Significant Concerns (completeness gaps) -5. Scholarly paper workflow decomposition is good but incomplete - -The step graph decomposition in the seed bundle (normalise → ensure_paper_concept → link_file_copy → attach_metadata → resolve_authors → resolve_topics → assert_source_identity → verify) matches the 1765 spec closely. However, the old monolithic actions (scholarly_paper.materialise_from_file_copy, scholarly_paper.enrich_from_metadata) are still registered with a (monolithic) annotation but not removed. This creates two parallel paths through the same domain. - -6. arxiv.inspect_existing_state searches by substring match - -In _build_arxiv_inspect_existing_state_handler(), the search logic does if arxiv_id.lower() in cid.lower() against concept IDs. This is fragile — an arXiv ID like 2401.0001 would match 2401.00010 (a different paper). The handler should use exact matching or a canonical lookup predicate. - -7. turn_execution_record_service.py schema extension lacks producers - -The build_turn_execution_record() function now accepts selected_workflow_trace, critic_verdict, completion_gate_verdict, and arxiv_completion_report — but I don't see any caller in the diff that actually passes these new fields. The schema extension exists on paper but isn't wired. - -Minor Concerns -8. No reacquire_partial_cache transition in the seed bundle - -JVNAUTOSCI-1765 lists the acquisition mode graph as including reuse_existing_file_copy | recover_partial_cache_state | download_paper. The seed bundle's decide_acquisition_mode conditional transitions handle existing_file_copy and finalise_cached_pdf with a fallback always → download_paper, but there's no explicit reacquire_partial_cache transition — it falls through to download. This might be intentional but isn't obvious. - -9. Step callbacks are fire-and-forget with swallowed exceptions - -Both engine.py and durable_executor.py catch and log step_callback exceptions without surfacing them. If the progress tracker expects reliable delivery, silent failures could mask problems. - -10. build_arxiv_ingestion_completion_report() has legacy/new field ambiguity - -The function has both decisions and decisions_taken, done and created_ids with fallback logic (decisions or decisions_taken). The Jira spec says fields should be mandatory, not best-effort. The dual naming invites confusion about which field is canonical. - -Summary Assessment -The workflow graph structure in the seed bundles is solid and matches the 1765 spec well. The decomposition from monolithic handlers into explicit reportable steps (ensure_paper_concept, link_file_copy, etc.) with writes_context_keys is a genuine step forward. - -However, the implementation violates the "no thin shim" rule in three places: the route-level arXiv fork (concern 1), the Python-embedded narration prompt (concern 2), and the orchestrator fallback reply composition (concern 4). The critic/completion-gate enforcement required by 1767 is absent (concern 3). Together, these mean the turn lifecycle is still Python-owned for arXiv paths — the workflow provides the step execution but Python still controls dispatch, narration, and completion gating. - -Claude Opus 4.6 • 3x - -Findings - -Critical: JVNAUTOSCI-1763 is not actually closed on the authoritative VWL/Vontology surface. The runtime registry is explicitly read-only and does not bootstrap workflow authority during registry construction (registry_factory.py (line 945), registry_factory.py (line 1077)). I checked the live registry, and it still exposes the old Vontology definitions: #V#conversation_turn_execution_workflow is still the old 3-state critic/gate flow, #V#arxiv_paper_representation_workflow is still the older 10-state wrapper without arxiv.build_completion_report, #V#scholarly_paper_representation_workflow is still the older 7-state monolith, and prompt_turn_execution_compose_workflow_outcome_report does not exist. So the branch mainly edits repo seed bundles and Python around unchanged authoritative workflows. - -Critical: The new arXiv “supervised” path is currently broken. execute_conversation_turn_supervised() imports a non-existent module at orchestrator.py (line 20494), and the new turn-execution action module imports the same non-existent path at turn_execution_actions.py (line 74). I verified that src.backend.workflows.registry_factory raises ModuleNotFoundError. Even beyond that, the durable action registry currently does not register turn_execution.completion_gate, turn_execution.critic, or kb_mutation.critic, while the new conversation-turn seed workflow depends on them (canonical_workflow_publication_seed_bundle.json (line 1425)). - -High: The control plane is still Python-owned rather than workflow-owned, which is the exact architectural smell 1766 was meant to remove. The route special-cases arXiv in Python (von_routes.py (line 8415)), the workflow routing action just trusts route-supplied discovery and leaves a TODO instead of owning selection (turn_execution_actions.py (line 43)), execution is explicitly synchronous “for now” rather than a durable handoff (orchestrator.py (line 20531)), and narration uses a Python-authored inline prompt plus direct llm_client.generate() (turn_execution_actions.py (line 114)). That is still Python pretending to be a hosted workflow. - -High: The new conversation-turn graph does not match the Jira-required sequence and cannot truthfully gate the final answer. The seed bundle now says routing -> execution -> narration -> critic -> completion_gate -> completed (canonical_workflow_publication_seed_bundle.json (line 1383)), but JVNAUTOSCI-1766 required ... -> run_postcondition_critic -> apply_completion_gate -> compose_user_report -> completed|failed. It also unconditionally transitions from completion_gate to completed (canonical_workflow_publication_seed_bundle.json (line 1438)), so the workflow graph itself does not encode the 1767 rule that unresolved verification must block completion. - -High: The structured arXiv reporting layer required by 1764/1765 is mostly declarative on paper, not real at runtime. The schema supports decisions, done, not_done, and reuse facts (execution_contracts.py (line 365)), but arxiv.build_completion_report() never populates those fields and instead emits a generic "Processed arXiv paper..." summary (paper_representation_workflow.py (line 1242)). The booleans that are supposed to distinguish fresh download vs reuse vs recovery are not produced anywhere else in the backend. Even one of the few new factual flags is wrong: paper_concept_created is computed after creation, so it will usually report false on the arXiv path (paper_representation_workflow.py (line 797)). - -Medium: The turn-execution record extension is not actually wired through, and I don’t see the exact-path test coverage 1767 asked for. New fields were added to the record builder (turn_execution_record_service.py (line 4230), turn_execution_record_service.py (line 4472)), but the main call sites still invoke build_turn_execution_record() without passing them (orchestrator.py (line 6733), von_routes.py (line 4110)). I also don’t see any test file changes in this branch for the fresh-download / reuse / partial-recovery / failure scenarios named in JVNAUTOSCI-1767. - -I would not consider JVNAUTOSCI-1763 or subtasks 1764-1767 genuinely done in their current state. The strongest concern is not just code quality; it’s that the branch still leaves the real authority in the wrong place, and the actual authoritative Vontology workflows have not been brought along. - -I didn’t change code. I checked the working tree, the Jira task texts/comments, the authoritative workflow registry, and a couple of targeted runtime validations; I did not run a full /von/generate end-to-end because the import/registration failures and unchanged authoritative workflows already show the path is not in a closable state. - diff --git a/docs/engineering/OPTION1_SECURITY_IMPLEMENTATION.md b/docs/engineering/OPTION1_SECURITY_IMPLEMENTATION.md deleted file mode 100644 index 878002fe2..000000000 --- a/docs/engineering/OPTION1_SECURITY_IMPLEMENTATION.md +++ /dev/null @@ -1,180 +0,0 @@ -# Historical Option 1 Security Mitigation Record - -> **Document status: Historical mitigation record from 2 December 2024; not a -> statement of Von's current security posture.** “Secure” below referred only -> to removal of the client-provided `user_id` fallback on the named path. It did -> not establish complete caller authentication, authorisation, namespace -> isolation, or deployment security. Use -> [`security_considerations.md`](security_considerations.md) for current -> requirements and limitations. - -**Date**: December 2, 2024 -**Issue**: JVNAUTOSCI-760 (RAG namespace filtering) -**Bounded milestone result**: Client-provided `user_id` fallback removed on the -named path - -## Summary - -At the time, this implemented the bounded “Option 1” authentication mitigation: -- ❌ **Removed** insecure client-provided user_id fallback -- ✅ **Require** proper authentication via session or validated headers -- ✅ **Inform** agent about authentication status and tool availability -- ✅ **Fail closed** - unauthenticated users cannot access RAG tools - -## Changes Made - -### 1. von_routes.py - Removed Insecure Fallback - -**Before** (INSECURE): -```python -# Fallback: trust client-provided user_id -if not user_concept_id and request_user_id: - user_concept_id = request_user_id # ❌ Anyone can impersonate -``` - -**After** (SECURE): -```python -# SECURITY: Do NOT trust client-provided user_id -user_concept_id = get_effective_user_concept_id() -if not user_concept_id: - # Client-provided user_id is IGNORED - # RAG tools will be unavailable -``` - -### 2. orchestrator.py - Agent Authentication Awareness - -**New Feature**: Agent receives authentication status in system prompt - -**Authenticated User**: -``` -🔐 AUTHENTICATION STATUS: Authenticated (namespace: #V#michael_witbrock) -RAG tools (search_knowledge_base, rag_list_indexed, rag_get_item) are AVAILABLE. -``` - -**Unauthenticated User**: -``` -⚠️ AUTHENTICATION STATUS: NOT AUTHENTICATED -RAG tools (search_knowledge_base, rag_list_indexed, rag_get_item) are UNAVAILABLE. -These tools require user authentication to prevent cross-user data access. -If user asks about their RAG data/sessions/indexed content, explain they need to log in first. -``` - -### 3. catalogue.py - Enforce Namespace Requirement - -All RAG tools now require namespace and fail with clear error: - -```python -# SECURITY: Require namespace for RAG access -if not ns: - return { - "error": "namespace_required", - "message": "RAG access requires authenticated user context (namespace)", - "success": False - } -``` - -## Authentication Methods Supported - -1. **Server-side session** (primary) - - Populated during `/api/auth/google/login` flow - - Stored in Flask session - - Most secure method - -2. **Validated headers** (for automation) - - `X-User-Concept-ID` or `X-User-Client-ID` - - Must pass `_validate_person_concept()` check - - Verifies concept exists in database - -## User Experience - -### Authenticated Users -- ✅ Full access to RAG tools -- ✅ Can search knowledge base -- ✅ Can list indexed sessions -- ✅ Can view session details -- ✅ Namespace isolation enforced (see only their data) - -### Unauthenticated Users -- ✅ Can still use chat interface -- ❌ Cannot access RAG tools -- ✅ Agent explains need to log in -- ✅ Clear error messages -- ✅ No data leakage (fail closed) - -## Testing - -### Manual Tests - -1. **Authenticated RAG Access**: - ```bash - # After logging in via Google OAuth - curl -X POST http://localhost:5000/generate \ - -H "Content-Type: application/json" \ - -d '{"prompt": "How many sessions do I have indexed?"}' \ - --cookie "session=" - # ✅ Should work, agent sees authentication status - ``` - -2. **Unauthenticated RAG Access**: - ```bash - # Without session cookie - curl -X POST http://localhost:5000/generate \ - -H "Content-Type: application/json" \ - -d '{"prompt": "Search my knowledge base"}' - # ✅ Should fail gracefully, agent explains need to log in - ``` - -3. **Namespace Requirement**: - ```python - from src.backend.integrations.internal_mcp.catalogue import _rag_list_indexed - - # No namespace - result = _rag_list_indexed() - assert result["error"] == "namespace_required" - - # With namespace - result = _rag_list_indexed(namespace="#V#michael_witbrock") - assert result["success"] == True - ``` - -## Security Properties - -✅ **Authentication Required**: No RAG access without proper authentication -✅ **No Impersonation**: Client cannot fake user identity -✅ **Namespace Isolation**: Users only see their own data -✅ **Fail Closed**: Errors deny access rather than allow -✅ **Agent Transparency**: Agent knows and explains auth status -✅ **Audit Trail**: All auth failures logged - -## Deployment Checklist - -- [x] Remove client-provided user_id fallback -- [x] Require authentication for RAG tools -- [x] Enforce namespace filtering -- [x] Inform agent about auth status -- [x] Log authentication events -- [x] Update security documentation -- [x] Test authenticated flow -- [x] Test unauthenticated flow -- [x] Verify namespace isolation -- [ ] Add admin endpoint authentication (future) -- [ ] Add rate limiting (future) -- [ ] Add audit logging for sensitive operations (future) - -## Migration Notes - -**Breaking Change**: Unauthenticated users can no longer access RAG tools. - -**Impact**: Single-user research prototype users must now: -1. Authenticate via `/api/auth/google/login` (already implemented) -2. Or use validated headers for automation scripts -3. Client-side localStorage user_id no longer provides access - -**Backwards Compatibility**: None - this is a security fix, not a feature. - -## References - -- Security Documentation: `docs/engineering/security_considerations.md` -- Authentication System: `src/backend/server/routes/auth_routes.py` -- Access Control: `src/backend/security/access_control.py` -- MCP Tools: `src/backend/integrations/internal_mcp/catalogue.py` diff --git a/docs/engineering/README.md b/docs/engineering/README.md index 4db6a84e2..a5a813a89 100644 --- a/docs/engineering/README.md +++ b/docs/engineering/README.md @@ -1,9 +1,11 @@ # Engineering Documentation -This directory deliberately contains both current engineering guidance and the -historical design, audit, incident, and Jira records that explain how Von -evolved. Proximity in this directory does not imply equal authority or -freshness. +This directory contains maintained public engineering guidance, contracts, +runbooks and reproducible implementation protocols. Working design proposals, +architecture reviews, dated audits and implementation/experiment notes belong +in Von-Private's `research/design_notes/` collection. Historical public +citations use commit-pinned links; private access is not required to follow +public implementation guidance. Use the canonical [Von design and engineering document index](../design_index.md) to determine a document's kind, lifecycle, authority, diff --git a/docs/engineering/Von_for_AgenticAI.md b/docs/engineering/Von_for_AgenticAI.md deleted file mode 100644 index 3b9a814e5..000000000 --- a/docs/engineering/Von_for_AgenticAI.md +++ /dev/null @@ -1,677 +0,0 @@ -# Von for Agentic AI - -- **Kind:** Long-horizon programme and target-architecture design -- **Lifecycle:** Active as advisory design; implementation snapshot frozen -- **Authority:** Advisory; `AGENTS.md` and current explicit user direction govern -- **State evidence as of:** 24 April 2026 -- **Last reviewed:** 25 July 2026 -- **Freshness boundary:** Durable target-direction arguments may remain useful; - all “current”, “already”, “still”, code-size, gap, and priority claims require - live revalidation -- **Scope:** Von's architectural direction and the path from a capable - neuro-symbolic assistant toward minimal-imposition agentic AI in the stronger - Von sense - -## 1. Purpose - -This note records the architecture Von is actually moving toward, not merely an aspirational marketing summary. - -It is not an implementation template. Direct model, tool, function, or small -compositional paths remain the baseline; represented layers and stages must -earn their cost for a concrete capability. - -It is intended to do four things: - -1. state clearly what class of system Von is trying to become; -2. distinguish recent real architectural progress from stale earlier diagnoses; -3. identify the most important remaining gaps between the present system and Von's minimal-imposition agent platform vision; -4. make explicit the near-term proving ground in Strong AI Lab research operations and role learning; -5. give engineering and research guidance for how to close those gaps without sliding back into "LLM wrapped in Python heuristics". - -This is not a full changelog. It is an architecture and programme note. - -## 2. Executive Summary - -Von should not evolve into a larger and larger chat application with more ad hoc Python control logic around an LLM. - -The intended direction is a **deployable neuro-symbolic agent platform** in which: - -- enduring knowledge lives in explicit represented form rather than only in latent model state; -- durable behavioural policy that needs independent governance may be authored - in Vontology, workflows, prompt programmes, KB assertions, or another - inspectable surface; -- Python provides reusable support surfaces such as execution, validation, telemetry, persistence, integration, and, where implementation in Python will not compromise agentic generality, safety; -- a **portfolio of model scales** is used intentionally rather than pretending one model class should do everything; -- the system learns and adapts by promoting evidence-backed knowledge, workflows, prompts, and policy artefacts into durable represented form; -- knowledge representation may be in many forms: human or AI authored natural-language documents in any language; propositions in a logical language based on predicate calculus; knowledge in stronger, higher-order, contextual, temporal, or modal logics, with the representational power of the Cyc KB as a useful baseline; embeddings; and other forms; -- the agent acts with **minimal imposition**: it searches, retrieves, reasons, and explains before burdening people with clarification or process disruption; -- long-horizon reasoning, episodic memory, explicit provenance, and safe revision matter as much as immediate answer quality. - -Recent work moved Von materially in that direction. The architecture is cleaner than it was ten days earlier, but the programme is not finished. Several major semantic-policy surfaces still remain in Python, enduring memory is still immature, multi-agent coordination is still mostly a future shape, and continual self-improvement remains more a direction than a fully closed operational loop. - -The near-term proving ground should be more explicit than earlier versions of -this note made it. Von should first become useful by improving process and -operational effectiveness for AI research in the Strong AI Lab, especially by -learning to perform otherwise hard-to-fill research-operations roles with -minimal imposition. From there it should generalise to scientific research, -discovery and R&D more broadly, then to auxiliary organisational roles, and -eventually to roles in general. This staged path narrows the immediate focus -without narrowing the AGI ambition: SAIL research operations are the first -living laboratory for the broader role-learning architecture. - -## 3. Current Architectural Assessment - -### 3.1 Real progress already made - -The most important recent progress is not merely code motion. Several major policy surfaces have already been removed from Python or pushed closer to represented authority. - -#### 3.1.1 Workflow discovery and routing - -Recent landings removed some of the worst old lexical routing doctrine: - -- workflow discovery no longer depends on the old BM25 / stopword capability-index path; -- structured tool-family availability is no longer driven by prompt-keyword and prefix heuristics; -- turn expected-outcome contract handling is now a first-class cross-stage boundary rather than loose per-case plumbing; -- explicit predicate-relative ontology turns now have a proper intermediate support surface through `get_predicate_incidence(...)` rather than relying only on broad relation-hit paging. - -This is a meaningful shift away from language-specific pseudo-NLP toward retrieval-backed and authority-aligned support surfaces. - -#### 3.1.2 Domain-interpretation cleanup - -Multiple domain services that previously held English-language or regex-heavy semantic policy in Python have been moved toward authority-backed interpretation surfaces, especially in: - -- file-copy interpretation; -- workflow authoring and annotation extraction; -- write-request evidence inference. - -That matters because it reduces the tendency for "temporary" deterministic semantic logic to become the real policy substrate. - -#### 3.1.3 Turn correctness and postcondition evaluation - -The turn-correctness path has improved significantly. - -Committed work through `JVNAUTOSCI-1956` stopped grounded false-negative turns from being silently marked as verified success and removed the remaining low-information answer-surface fallback from the primary completion path. The prompt-backed postcondition critic now exists as an explicit subworkflow, with: - -- a dedicated prompt seed for the critic; -- explicit evidence-bundle construction; -- a subworkflow pathway for authoritative evaluation; -- tests that exercise this critic pathway as a workflow-level support surface. - -That is the right direction. Correctness judgement should not remain a frozen Python string-matching layer. - -#### 3.1.4 Anti-drift and test quality - -Recent work also improved the meta-architecture: - -- anti-drift purity checks now guard several reclaimed support surfaces; -- stale tests that pinned removed heuristics have been repaired; -- real-path and near-real-path validations are increasingly being used instead of relying only on unit-shaped assertions. - -That matters because without this layer, the codebase would continuously regress back to heuristic patches. - -### 3.2 The current code reality is still mixed - -Despite the above progress, Von still has major oversized integration files. - -As of 21 April 2026: - -- `src/backend/integrations/internal_mcp/orchestrator.py` is still `34,608` lines -- `src/backend/integrations/internal_mcp/catalogue.py` is still `27,466` lines -- `src/backend/server/routes/von_routes.py` is still `13,348` lines - -So the direction is improving, but the largest files are still large enough to attract inline patching pressure and hidden policy drift. - -### 3.3 What is still wrong in the current architecture - -The most important surviving issues are now more concentrated. - -#### 3.3.1 The contract-text tool-inference seam has now been removed - -The April 21, 2026 bounded `JVNAUTOSCI-1913` / `JVNAUTOSCI-1972` cleanups moved -turn-contract tool requirements onto the explicit `required_tools` field of the -shared `TurnExpectedOutcomeContract` support surface and removed the remaining -contract-text tool-inference and prose-backfill helpers from the orchestrator. - -That means the live path no longer decides Jira, KB, predicate/relation, -relation-argument, or web retrieval requirements by scanning English contract -prose inside Python. - -The next risk is therefore no longer this semantic-policy seam. It is the -structural pressure created by the remaining orchestrator monolith and the -remaining represented-metadata follow-through tasks. - -#### 3.3.2 The earlier write-policy hotspot has already moved - -`JVNAUTOSCI-1970` removed the remaining live write-denial heuristic and hard-coded write-risk classes from the main write path. Earlier draft language pointing at `_ADDITIVE_LOW_RISK_WRITE_TOOLS`, `_MUTATIVE_NON_DESTRUCTIVE_WRITE_TOOLS`, `_DESTRUCTIVE_WRITE_TOOLS`, `_CONFIRMATION_PATTERN`, `_DESTRUCTIVE_MUTATION_PATTERN`, and `prompt_explicitly_denies_write(...)` is therefore stale as a guide to the next architecture step. - -The broader lesson about multilingual policy risk still matters, but it is no longer the sharpest remaining Python-policy seam. - -#### 3.3.3 Buttonify had one remaining residual authority seam - -The live buttonify path already ran through `#V#chat_buttonify_workflow` and structured `buttonify_options_json` validation. The remaining problem was narrower: `src/backend/services/buttonify_service.py` still carried residual heuristic/prose-recovery helpers, and stale docs/tests made it look as if Python could still reconstruct UI options from prose. - -`JVNAUTOSCI-1971` removes that residual helper slab so buttonify options now come only from the authority-backed structured output contract, with Python limited to validation, normalisation, and telemetry. - -#### 3.3.4 Predicate-incidence is the right abstraction but may become expensive - -The new `get_predicate_incidence(...)` surface is architecturally good, because it fills a real gap between: - -- `entity -> relation hits` -- and `predicate -> extent` - -However, the current type-mode implementation gathers all relevant instances and then aggregates relation hits across them. For broad classes or high-degree entities, this is a real runtime and deployability concern. - -That means the design is correct, but the scaling strategy still needs work. - -### 3.4 A note on current state discipline - -One live engineering caution is worth making explicit. - -The main risk is now documentation lag rather than an unmerged critic prototype. High-level notes need to distinguish carefully between: - -- what is already committed and merged; -- what exists only in a current working tree; -- what remains aspirational. - -That distinction matters because otherwise the repo risks documenting local intent as if it were already part of the durable shipped architecture. - -## 4. The Architecture Von Should Be Becoming - -The correct target is not "an LLM plus a growing pile of wrappers". It is a **capability-composed agentic system** with explicit, revisable, inspectable cognition. - -### 4.1 Vontology as the cognitive substrate - -Vontology should be treated as a first-class durable substrate for: - -- semantic memory; -- represented concepts, predicates, and relations; -- provenance-bearing knowledge; -- workflow artefacts; -- prompt programmes; -- policy metadata; -- long-horizon organisational state. - -This is crucial to Von's ambition. A system that only "remembers" through latent weights or ephemeral vector recall cannot be trusted to adapt over months or years while preserving inspectability, revision, and provenance. - -Von should therefore continue treating durable represented state as an implementation surface, not merely a cache. - -For the concrete Vontology tool families implied by this direction, see -`docs/engineering/vontology_tooling_from_ka_kcap_kr_literature.md`. That note -connects KA, KCAP, commonsense KR, Cyc-style contextual representation, and -scientific KR literature to proposed Vontology read/write/support tools. - -### 4.2 Optional durable behavioural authority - -When behaviour genuinely needs independent authoring, revision, attribution, -evaluation, or reuse, possible surfaces include: - -- VWL workflow definitions; -- prompt concepts and text relations; -- represented routing metadata; -- KB assertions and explicit constraints; -- contract objects such as turn expected-outcome contracts and required-effects contracts. - -Do not hide such policy in incidental lexical helpers and conditionals. Equally, -do not create a represented contract merely because the behaviour is durable or -user-visible; compare it with the simpler direct path. - -### 4.3 Shared turn context as a first-class surface - -A genuinely agentic system should not give multiple model calls materially -different hidden contexts without making consequential changes inspectable. -This applies only when multiple calls are justified. - -Von should continue pushing toward: - -- a shared situation across the consumers a capability actually needs; -- explicit call-local additions where consequential; and -- enough telemetry to explain material context differences. - -This does not imply a selector, planner, critic, narration stage, or universal -turn-state object. - -### 4.4 Model hierarchy and capability composition - -Von should assume a model portfolio rather than a monolithic-model doctrine. - -That means: - -- smaller models for bounded, cheap, frequent, structurally constrained work; -- stronger models for difficult synthesis, planning, ambiguity resolution, and critic functions; -- specialised symbolic or deterministic modules for validation, persistence, and explicit typed structure; -- routing and escalation policies that are inspectable and revisable. - -This aligns directly with the Von view that robustness should come from designed capability composition rather than pure parameter scaling. - -### 4.5 Explicit memory strata - -Von needs to treat memory as a set of distinct architectural layers, not one big "RAG" bucket. - -At minimum it should continue toward: - -- **semantic memory:** durable facts and relations; -- **episodic memory:** traces of interactions, workflows, and long-running organisational processes; -- **procedural memory:** workflows, prompts, playbooks, validators; -- **policy memory:** learned routing guidance, critiques, context-conditional advice, and other reusable operational knowledge. - -This is essential if Von is to support long-horizon reasoning, continual learning, and adaptive deployability without repeatedly re-deriving everything from transient conversation context. - -### 4.6 Minimal-imposition operational behaviour - -Von should continue optimising for a form of helpfulness that reduces burden rather than only maximising answer fluency. - -That implies: - -- search and retrieval before clarification; -- explicit uncertainty and provenance instead of bluffing; -- preference for low-risk additive actions over unnecessary permission loops; -- stronger intervention only when the concrete consequence warrants it; -- preserving institutional and scholarly practice rather than flattening it into a narrow internal schema. - -Minimal imposition is not merely a UX preference. It is a central operational criterion for real-world deployability. - -### 4.7 Critics and evaluators when evidence warrants them - -A critic or evaluator may help when direct execution has a demonstrated, -material failure that the extra call catches economically. - -Possible uses include: - -- postcondition critics; -- evidence-answer consistency critics; -- uncertainty and failure-surface critics; -- recovery-decision workflows; -- long-horizon reflection and promotion workflows. - -Critics are not standard architecture. Compare them with direct execution on -usefulness, latency, cost, false rejection, and recovery, and remove them when -they do not improve the complete user job. - -### 4.8 Capability plug-ins and secure deployment surfaces - -Von's longer-term platform direction points toward a "suite" of capability plug-ins deployable into partner and Living Lab environments, including sensitive environments where raw data cannot simply be exported. - -Von therefore needs to become a clean capability platform, not just an application bundle. That implies: - -- well-bounded integration surfaces; -- deployable plugin/capability modules; -- explicit trust, provenance, and namespace boundaries; -- a clean split between represented authority and runtime substrate; -- support for private local deployment with selective escalation to stronger remote models when policy allows. - -### 4.9 Role learning as the near-term proving ground - -Von's most important near-term application spine should be -**minimal-imposition role learning**. - -The system should learn how to take on useful roles in an organisation or team -when those roles are otherwise hard to fill because of availability, cost, or -capability constraints. For the current programme, the first target should be -process improvement and operational effectiveness for AI research in the Strong -AI Lab. - -The staged path is: - -1. **SAIL AI research operations:** help with real research-process drag, - continuity, literature and evaluation support, project tracking, meeting and - action-note follow-through, artefact curation, and onboarding. -2. **Science, discovery, and R&D:** generalise from SAIL to broader research - workflows, experiment and evaluation management, interdisciplinary - collaboration, and discovery-support work. -3. **Auxiliary organisational roles:** extend to coordination, administration, - communications, compliance, support, and other roles around research teams. -4. **Roles in general:** abstract the role-learning machinery so new - organisations can safely onboard Von into hard-to-fill roles through bounded - observation, represented role modelling, evaluation, and improvement loops. - -The role itself should become a represented object, not a prompt persona or a -Python role-name branch. A role-performance representation should include: - -- responsibilities, authority boundaries, permissions, and escalation rules; -- recurring workflows, process states, handoffs, and interaction patterns; -- expected work products and quality/evidence contracts; -- notes, dossiers, evidence receipts, and episodic traces; -- procedural memory such as workflows, prompts, skills, validators, and - playbooks; -- policy memory such as role-specific guidance, routing rules, model choices, - and learned improvement advice; -- evaluator axes for operational effectiveness, burden reduction, work-product - quality, groundedness, calibration, recovery, and long-horizon continuity. - -This is where the AGI-shaped substrate becomes operationally concrete. Von -should learn from work episodes, promote useful traces into semantic, -episodic, procedural, and policy memory, test candidate role improvements in -bounded worlds, and publish improvements only through represented, -evaluable, rollback-aware artefacts. - -## 5. Engineering Implications of This Architecture - -This architecture changes how engineering work should be understood. - -### 5.1 More behaviour work becomes knowledge and workflow engineering - -Many changes that look like "implementation" in ordinary software should be authored in: - -- represented knowledge; -- prompt concepts; -- workflow definitions; -- critic/evaluator artefacts; -- reusable contracts and metadata. - -That is not avoiding engineering. It is doing engineering at the correct authority layer. - -### 5.2 Monolith drift is the main enemy - -Large integration surfaces naturally attract quick fixes. In Von, that is particularly true of: - -- `orchestrator.py` -- `catalogue.py` -- `von_routes.py` - -The main danger is not only file size. It is that mixed responsibilities invite: - -- hidden routing policy; -- hidden verification policy; -- stage-specific context shaping; -- answer-surface repair patches; -- integration-boundary drift. - -### 5.3 Evaluation has to become richer and more honest - -This architecture cannot be validated by ordinary unit tests alone. - -Useful evaluation must include: - -- exact or near-real call-path tests; -- replay-based failure analysis; -- retrieval quality and answer-grounding assessment; -- long-horizon memory tasks; -- calibration and uncertainty evaluation; -- interruption burden and minimal-imposition metrics; -- robustness across paraphrase, multilingual phrasing, and organisational variation; -- cost and latency effects of richer multi-stage cognition. - -### 5.4 Telemetry is not optional polish - -If Von is to become a self-improving, inspectable agent platform, telemetry must remain first-class. - -That includes: - -- stage context lineage; -- workflow discovery and routing traces; -- selected-workflow traces; -- evidence bundles for critic stages; -- completion-gate diagnostics; -- recovery-route traces; -- long-horizon episodic records that can later be re-analysed or promoted. - -### 5.5 Security, provenance, and namespace discipline stay central - -A minimal-imposition agent that works in sensitive real environments cannot trade away access safety for convenience. Namespace discipline, provenance preservation, and clear authorisation boundaries must remain part of the design, not bolt-ons. - -### 5.6 Role-learning work must consume the shared substrate - -Role-learning implementation should not create a parallel stack. - -When Von learns a role, the evidence and improvements should flow through the -same architectural surfaces already being built: - -- enduring-memory manifests and query surfaces for role episodes, work products, - dossiers, notes, and evidence receipts; -- memory-promotion workflows for role facts, process knowledge, procedural - artefacts, and policy guidance; -- multi-axis evaluators for grounded helpfulness, calibration, recovery, - continuity, operational effectiveness, and work-product quality; -- candidate worlds for evaluating workflow, prompt, skill, model-routing, or - retrieval-policy changes before promotion; -- Vontology-authored workflows, prompt concepts, role concepts, and policy - artefacts as the durable authority surfaces. - -Python may add reusable support for manifests, validators, telemetry, -connectors, durable execution, and benchmark harnesses. It should not become -the place where SAIL-specific or role-specific task policy is encoded. - -## 6. What Is Still Missing on the Path to Fully Helpful Agentic AI - -The biggest gaps are now less about obvious hacky routing and more about missing higher-order capabilities. - -### 6.1 Remaining code-side semantic policy removal - -Von still needs to remove the remaining Python-owned semantic policy surfaces, -especially: - -- any remaining hard-coded tool and risk metadata that belongs in represented authority; -- any further decision-policy remnants that are still living inside oversized orchestration helpers instead of represented authority surfaces or clearly bounded support code. - -This is still necessary groundwork. A system cannot become truly multilingual, general, and maintainable while core policy still depends on hidden English token lists. - -### 6.2 Enduring memory and consolidation - -Von has the right semantic-memory direction but still lacks a mature explicit memory architecture for: - -- episodic trace retention and retrieval; -- promotion from episodic evidence to durable semantic knowledge; -- revision and retraction pathways; -- policy-memory induction from repeated failures or successes; -- long-horizon task-state reconstruction across interruptions. - -Without this, Von will remain a capable turn-by-turn assistant rather than a truly enduring collaborator. - -The Vontology tooling design note at -`docs/engineering/vontology_tooling_from_ka_kcap_kr_literature.md` should be -used as the current companion for graph, provenance, context, Davidsonian -role-frame, scientific-claim, and knowledge-acquisition tool surfaces that make -this memory architecture usable from workflows. - -### 6.3 Safe continual self-improvement - -Von's ambition requires gain-of-function introspection and adaptive self-evolution. That means Von needs more than prompt tweaks and Jira tasks after failures. - -It needs explicit self-improvement loops for: - -- benchmark and replay analysis; -- prompt and workflow revision proposals; -- retrieval-policy learning; -- policy-memory induction; -- isolated testing of candidate changes; -- promotion only after evaluation and rollback-safe validation. - -The right model is not unconstrained self-modification. It is controlled improvement through represented, evaluable, promotable artefacts. - -### 6.4 Long-horizon temporal and institutional reasoning - -To support months-long or years-long processes, Von needs stronger representations and retrieval for: - -- temporal structure; -- event sequences; -- institutional procedures and obligations; -- workflow histories and state transitions; -- evolving user and team preferences; -- conflict, uncertainty, and revision over time. - -The current architecture is becoming capable of this, but not yet fully equipped for it. - -### 6.5 Multimodal knowledge acquisition and fusion - -Von's target architecture points toward integration across text, images, speech, and other data sources. Von has made progress on file-copy interpretation, but a mature multimodal architecture still requires: - -- common provenance models across modalities; -- authority-backed interpretation surfaces for multimodal evidence; -- cross-modal linking into Vontology; -- uncertainty-aware fusion rather than silent flattening into text summaries. - -### 6.6 Multi-agent coordination - -The current architecture is still primarily single-agent in its core lifecycle, even though it already contains many of the pieces needed for richer composition. - -What is still missing includes: - -- explicit agent identities and capability profiles; -- safe delegation and subtask ownership; -- shared episodic and semantic state across collaborating agents; -- coordination protocols for handoff, arbitration, and conflict resolution; -- evaluation of multi-agent failure modes such as context divergence and duplicated work. - -This is essential for Von's goal of coordinated agent systems in complex domains. - -### 6.7 Normative governance beyond simple obedience - -A deployable real-world agent needs to bridge rules, norms, uncertainty, and practical adaptation. - -Von still needs stronger explicit handling of: - -- organisational norms; -- cultural and institutional constraints; -- policy pluralism and conflict; -- uncertainty-bearing recommendation rather than false certainty; -- documented reasons for abstention, escalation, and refusal. - -### 6.8 User-facing transparency and calibration - -A fully helpful agent is not just one that retrieves better. It is one that: - -- exposes what it knows and how it knows it; -- distinguishes evidence from conjecture; -- reveals uncertainty when it matters; -- remains answer-first while still preserving inspectable operational traces; -- helps people trust the right things and distrust the right things. - -Recent work on critic pathways and answer-first discipline helps here, but the broader transparency model is still incomplete. - -### 6.9 Role-performance representation and evaluation - -The current architecture still does not make role performance first-class -enough. - -Von needs a canonical way to represent: - -- a role in an organisation or team; -- the role's responsibilities, obligations, authority boundaries, and - escalation points; -- recurring workflows, interactions, handoffs, and process states; -- work products, review standards, and acceptance evidence; -- notes, artefacts, meetings, decisions, and follow-up state; -- the skills, prompts, models, tools, and other capabilities used to perform the - role; -- evidence that the role is being performed better over time. - -This is especially important for the near-term SAIL research-operations aim. -Without this representation, Von risks becoming a collection of helpful -features rather than a system that can learn an under-filled role, improve at -it, and generalise that learning to new research teams and eventually to new -organisational roles. - -## 7. A Concrete Path Forward - -### 7.1 Near-term engineering priorities - -The most important near-term engineering work has two connected tracks. - -First, continue the substrate work that prevents Von from regressing into a -large Python application with prompts attached: - -1. finish replacing remaining Python semantic policy surfaces with represented authority or prompt-backed critic/evaluator surfaces; -2. continue extracting mixed responsibilities out of the orchestrator and route monoliths; -3. keep anti-drift tests and purity checks aligned with the newer architecture; -4. harden scaling and telemetry around new support surfaces such as predicate incidence; -5. keep documentation and programme notes aligned with the actually landed authority surfaces so follow-on work targets the right remaining seams. - -Second, start the role-learning application spine: - -1. define the canonical role-performance representation for SAIL research - operations; -2. identify a small set of SAIL process-improvement pilot roles and their work - products; -3. connect those pilots to enduring-memory, evaluator, and candidate-world - tasks rather than inventing a separate role stack; -4. extend minimal-imposition evaluation to measure operational effectiveness, - avoided burden, workflow continuity, work-product quality, and team adoption - friction; -5. use the SAIL pilots to discover missing reusable Vontology, workflow, - evaluator, and telemetry primitives. - -### 7.2 Medium-term platform priorities - -After that, the most important platform-level advances are: - -1. explicit episodic memory and memory-promotion workflows; -2. isolated self-improvement and benchmark worlds; -3. richer retrieval and reasoning over time, events, and workflows; -4. structured multimodal evidence acquisition and fusion; -5. capability packaging for partner deployment. - -### 7.3 Longer-horizon Von priorities - -To reach the fuller Von vision, Von should grow into: - -- a compositional multi-agent platform; -- a secure deployable capability suite for partners; -- a system that can improve through represented policy and knowledge promotion; -- a long-horizon collaborator that preserves and revises knowledge over months or years; -- a platform that scales by combining specialised capabilities, explicit knowledge, and model hierarchy rather than only chasing larger parameter counts. - -### 7.4 Current programme umbrella in Jira - -The current umbrella Jira epic for this broader research and engineering direction is -[`JVNAUTOSCI-1960`](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-1960), which now functions as the main shaping epic for the broader minimal-imposition agentic AI programme around Von. - -That epic is intended to **shape** the programme rather than prematurely overdetermine it. It does not replace the anti-hack and authority-alignment substrate work under `JVNAUTOSCI-1116` / `JVNAUTOSCI-1913`; instead, it integrates that line with the wider memory, evaluation, multimodal, multi-agent, and deployment questions required for a true minimal-imposition agent platform. - -The near-term role-learning application spine is tracked separately in -[`JVNAUTOSCI-2011`](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-2011). -That epic should consume the `JVNAUTOSCI-1960` substrate rather than compete -with it. It exists to make the practical first deployment path explicit: -minimal-imposition role learning for SAIL research operations, then broader -science and R&D, then auxiliary organisational roles, then roles in general. - -The first design-framing task for that epic is -[`JVNAUTOSCI-2012`](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-2012). - -At the time of writing, its main child tasks are: - -- [`JVNAUTOSCI-1961`](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-1961): define the minimal-imposition benchmark and deployment acceptance model for Von -- [`JVNAUTOSCI-1962`](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-1962): define the enduring-memory and memory-promotion architecture across semantic, episodic, procedural, and policy memory -- [`JVNAUTOSCI-1963`](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-1963): build isolated self-improvement loops with ephemeral theories and benchmark worlds -- [`JVNAUTOSCI-1964`](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-1964): expand authority-backed critic and evaluator workflows for grounded helpfulness, calibration, and long-horizon quality -- [`JVNAUTOSCI-1965`](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-1965): design represented multi-agent coordination, delegation, and shared-memory substrates for Von -- [`JVNAUTOSCI-1966`](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-1966): define the multimodal evidence-fusion and provenance architecture for Von -- [`JVNAUTOSCI-1967`](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-1967): design the capability-suite and secure partner-deployment architecture for Von -- [`JVNAUTOSCI-1968`](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-1968): design the represented normative-governance, uncertainty, and escalation architecture for minimal-imposition deployment - -Taken together, these tasks mark the transition from "clean up anti-patterns in today's assistant" to "deliberately build the research and deployment substrate for the next class of deployable, minimal-imposition agent systems". - -## 8. Anti-Patterns to Keep Rejecting - -To preserve the architecture, the following remain design smells: - -- lexical pseudo-NLP in Python for durable routing, ranking, or classification policy; -- hidden phase-specific context thinning; -- prompt bodies in Python for Vontology-governed features; -- code-side answer-repair heuristics that should really be critic or workflow policy; -- role-name or organisation-name branches in Python that turn a SAIL pilot into - hidden durable role policy; -- persona-only "role learning" that does not represent workflows, - responsibilities, work products, evidence, and improvement lineage; -- treating vector search or transient context as if they were adequate substitutes for durable represented memory; -- treating one giant general model as if it removes the need for architecture. - -## 9. Bottom Line - -Von is increasingly becoming the right kind of system. - -It is moving away from a brittle "application plus LLM" model and toward a represented, inspectable, capability-composed neuro-symbolic agent platform. Recent progress on workflow discovery, turn contracts, predicate-incidence retrieval, authority-backed interpretation surfaces, and critic/evaluation pathways is real. - -But the work is not done. - -The path to fully helpful minimal-imposition agentic AI still requires: - -- removal of the remaining Python semantic policy seams; -- mature enduring memory and consolidation; -- safe self-improvement loops; -- first-class role-performance representation and SAIL research-operations pilots; -- multimodal knowledge fusion; -- multi-agent coordination; -- stronger temporal, institutional, and normative reasoning; -- continued discipline about represented authority, telemetry, and evaluation. - -If Von continues in that direction, it can become not merely a useful assistant, but a genuine platform for the next class of deployable, provenance-bearing, minimal-imposition agent systems. diff --git a/docs/engineering/agent_evaluation_and_research_uptake.md b/docs/engineering/agent_evaluation_and_research_uptake.md index 56e54a08d..11ed952fc 100644 --- a/docs/engineering/agent_evaluation_and_research_uptake.md +++ b/docs/engineering/agent_evaluation_and_research_uptake.md @@ -93,9 +93,9 @@ Adopt a new method only when it: - has an acceptable rollback path - does not quietly increase architectural opacity -For the current KA/KCAP/common-sense KR/scientific KR uptake pass that proposes +For the historical KA/KCAP/common-sense KR/scientific KR uptake pass that proposes new Vontology tooling surfaces, see -`docs/engineering/vontology_tooling_from_ka_kcap_kr_literature.md`. Use it as a +[historical Vontology tooling research review](https://github.com/Strong-AI-Lab/Von/blob/5b2f8b976069935da73b82bca433fb3cefffcd3d/docs/engineering/vontology_tooling_from_ka_kcap_kr_literature.md). Use it as a companion when implementing or evaluating Vontology tools that change retrieval, knowledge acquisition, context handling, provenance, or scientific representation behaviour. diff --git a/docs/engineering/agent_memory_and_enduring_knowledge.md b/docs/engineering/agent_memory_and_enduring_knowledge.md index 4494a08d5..bebc470ed 100644 --- a/docs/engineering/agent_memory_and_enduring_knowledge.md +++ b/docs/engineering/agent_memory_and_enduring_knowledge.md @@ -88,7 +88,7 @@ For concrete Vontology tool families that support richer structure, including typed predicate incidence, context and microtheory queries, provenance tracing, Davidsonian role-frame retrieval, scientific claim materialisation, and mixed-initiative knowledge acquisition, see -`docs/engineering/vontology_tooling_from_ka_kcap_kr_literature.md`. +[historical Vontology tooling research review](https://github.com/Strong-AI-Lab/Von/blob/5b2f8b976069935da73b82bca433fb3cefffcd3d/docs/engineering/vontology_tooling_from_ka_kcap_kr_literature.md). ## 8. Long-horizon evaluation diff --git a/docs/engineering/agentic_architecture_status_2026-04-23.md b/docs/engineering/agentic_architecture_status_2026-04-23.md deleted file mode 100644 index a3ab8615f..000000000 --- a/docs/engineering/agentic_architecture_status_2026-04-23.md +++ /dev/null @@ -1,435 +0,0 @@ -# Agentic Architecture Status - 23 April 2026 - -> **Document status: Frozen architecture and task-ordering snapshot.** This -> records evidence and conclusions from 23 April 2026. It is not current -> programme status. Revalidate code, Vontology, telemetry, and live Jira before -> using any “current”, “remaining”, or “next” claim. - -- **Kind:** Status snapshot -- **Lifecycle:** Frozen -- **Authority:** Evidence only -- **State as of:** 2026-04-23 - -## Purpose - -This note records the current engineering status of the Von codebase against the -design intent described in `docs/engineering/Von_for_AgenticAI.md`, using a -bounded static review of the live code on 23 April 2026. It was first written -earlier the same day, refreshed after the landings through -`JVNAUTOSCI-1994` and `JVNAUTOSCI-1993`, then refreshed again as the -implemented output of `JVNAUTOSCI-1989`, and then refreshed again after -`JVNAUTOSCI-1962` split the enduring-memory line into a concrete architecture -note plus first implementation slices, and then refreshed again after -`JVNAUTOSCI-1964` did the same for the evaluator line, and then refreshed again -after `JVNAUTOSCI-1963` did the same for the isolated self-improvement and -benchmark-world line, and then refreshed again after `JVNAUTOSCI-1998` landed -the first concrete multi-axis evaluator contract slice. - -It is not a replacement for `Von_for_AgenticAI.md`. That document remains the -long-horizon design note. This document is the current engineering-status and -task-ordering companion that answers a narrower question: - -- where the live code still falls materially short of the intended AGI-shaping - architecture; -- which remaining gaps are the sharpest next engineering priorities; and -- which Jira tasks now carry those gaps in a cold-start-useful form. - -## Current Headline - -The sharpest remaining problems are no longer the older write-policy, -buttonify, turn-contract prose, workflow-discovery fallback, or broad -tool-metadata seams. Those were materially reduced first by the April 21 -landings through `JVNAUTOSCI-1972`, then again by the 23 April landings of -`JVNAUTOSCI-1974`, `JVNAUTOSCI-1975`, `JVNAUTOSCI-1985`, -`JVNAUTOSCI-1973`, `JVNAUTOSCI-1986`, `JVNAUTOSCI-1988`, -`JVNAUTOSCI-1987`, `JVNAUTOSCI-1992`, `JVNAUTOSCI-1994`, and -`JVNAUTOSCI-1993`. - -After the `JVNAUTOSCI-1962`, `JVNAUTOSCI-1964`, and `JVNAUTOSCI-1963` -architecture passes, the -current shortfall is now concentrated in four areas: - -1. The evaluator line now has concrete child tasks rather than only the - umbrella `JVNAUTOSCI-1964`: `JVNAUTOSCI-1998`, `JVNAUTOSCI-1999`, - `JVNAUTOSCI-2000`, and `JVNAUTOSCI-2001` now carry the multi-axis evaluator - contract, grounded helpfulness critic, calibration and abstention evaluator, - and long-horizon evaluator support. `1998` and `1999` are now landed, so - the remaining work on this line is the calibration/recovery slice plus the - long-horizon support slice. -2. The self-improvement-world line now also has concrete child tasks rather - than only the umbrella `JVNAUTOSCI-1963`: `JVNAUTOSCI-2002`, - `JVNAUTOSCI-2003`, `JVNAUTOSCI-2004`, and `JVNAUTOSCI-2005` now carry the - candidate-world manifest, baseline-vs-candidate comparison, experiment-backed - promotion and rollback gating, and later prompt or policy candidate-world - expansion. -3. The enduring-memory line still has its own concrete child tasks: - `JVNAUTOSCI-1995`, `JVNAUTOSCI-1996`, and `JVNAUTOSCI-1997` carry the - canonical manifest/query surface, promotion/revision workflow, and - long-horizon evaluation harness that `1962` identified. -4. Monolith pressure remains real, especially in the orchestrator and - catalogue, but it is now secondary to the broader memory/evaluator/ - self-improvement substrate work. The files are still too large, but they are - materially smaller than the stale earlier same-day snapshot. - -## Current Code Reality - -As measured on the live branch during the `JVNAUTOSCI-1989` review: - -- `src/backend/integrations/internal_mcp/orchestrator.py` is `35095` lines -- `src/backend/integrations/internal_mcp/catalogue.py` is `27480` lines -- `src/backend/server/routes/von_routes.py` is `13351` lines -- `src/backend/services/turn_execution_record_service.py` is `6916` lines -- `src/backend/server/utils_flask.py` is `3036` lines - -The earlier same-day status snapshot was therefore materially stale on file -sizes alone. Large files are not the whole problem, but they remain the -strongest pressure surface for hidden policy drift and inline patching. - -## Live Code Anchors - -The remaining umbrellas should now be interpreted against the following live -surfaces rather than against older pre-`1988` or pre-`1993` wording: - -- Memory substrate and turn-context integration: - [conversation_turn_memory_context_service.py](../../src/backend/services/conversation_turn_memory_context_service.py), - [context_bundle_service.py](../../src/backend/services/context_bundle_service.py), - [context_bundle_service.py](../../src/backend/services/context_bundle_service.py), - [orchestrator.py](../../src/backend/integrations/internal_mcp/orchestrator.py), - [orchestrator.py](../../src/backend/integrations/internal_mcp/orchestrator.py), - [von_routes.py](../../src/backend/server/routes/von_routes.py) - -- Episode self-improvement and proposal/promotion loop: - [episode_evaluation_workflow.py](../../src/backend/workflows/durable/episode_evaluation_workflow.py), - [episode_self_improvement_service.py](../../src/backend/services/episode_self_improvement_service.py), - [episode_self_improvement_service.py](../../src/backend/services/episode_self_improvement_service.py), - [episode_self_improvement_service.py](../../src/backend/services/episode_self_improvement_service.py), - [episode_self_improvement_workflow.py](../../src/backend/workflows/durable/episode_self_improvement_workflow.py), - [workflow_studio_service.py](../../src/backend/workflows/workflow_studio_service.py) - -- Experiment worlds, testing theories, and publication rollback support: - [experiment_run_service.py](../../src/backend/services/experiment_run_service.py), - [experiment_run_service.py](../../src/backend/services/experiment_run_service.py), - [experiment_run_service.py](../../src/backend/services/experiment_run_service.py), - [testing_theory_service.py](../../src/backend/services/testing_theory_service.py), - [testing_theory_service.py](../../src/backend/services/testing_theory_service.py), - [testing_theory_service.py](../../src/backend/services/testing_theory_service.py), - [workflow_studio_service.py](../../src/backend/workflows/workflow_studio_service.py), - [workflow_studio_service.py](../../src/backend/workflows/workflow_studio_service.py) - -- Current critic/evaluator surfaces: - [episode_evaluator_contract_service.py](../../src/backend/services/episode_evaluator_contract_service.py), - [turn_execution_record_service.py](../../src/backend/services/turn_execution_record_service.py), - [turn_execution_record_service.py](../../src/backend/services/turn_execution_record_service.py), - [episode_critic_evidence_service.py](../../src/backend/services/episode_critic_evidence_service.py), - [episode_evaluation_workflow.py](../../src/backend/workflows/durable/episode_evaluation_workflow.py), - [episode_critique_memory_service.py](../../src/backend/services/episode_critique_memory_service.py), - [episode_critique_memory_service.py](../../src/backend/services/episode_critique_memory_service.py), - [episode_critique_benchmark_service.py](../../src/backend/services/episode_critique_benchmark_service.py), - [episode_evaluation_workflow_vontology_service.py](../../src/backend/services/episode_evaluation_workflow_vontology_service.py) - -These anchors matter because the remaining gaps are no longer mostly "missing -live-path repair" tasks. They are broader substrate-shaping tasks that should -start from the current represented surfaces rather than restating older absence -claims. - -## Targeted Literature Check - -The `JVNAUTOSCI-1989`, `JVNAUTOSCI-1962`, and `JVNAUTOSCI-1964` passes -together included a short targeted literature pass on memory, evaluator design, -and safe self-improvement. - -- [Memory for Autonomous LLM Agents: Mechanisms, Evaluation, and Emerging Frontiers](https://arxiv.org/abs/2603.07670) - argues that agent memory should be treated as a write-manage-read loop over - temporal scope, representational substrate, and control policy. That - reinforces `1962` as an architecture task, not just a retrieval feature task. -- [E-mem: Multi-agent based Episodic Context Reconstruction for LLM Agent Memory](https://arxiv.org/abs/2601.21714) - reinforces keeping some episodic context reconstructable rather than reducing - everything to flattened preprocessed memory. -- [MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks](https://arxiv.org/abs/2602.16313) - reinforces that memory and action should be evaluated together on - interdependent multi-session tasks rather than only through static recall. -- [LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://arxiv.org/abs/2410.10813) - reinforces that update handling, temporal reasoning, multi-session grounding, - and abstention belong in the enduring-memory acceptance surface. -- [ProcMEM: Learning Reusable Procedural Memory from Experience via Non-Parametric PPO for LLM Agents](https://arxiv.org/abs/2602.01869) - supports promoting episodic traces into reusable procedural artefacts with - explicit executability and gating, which maps directly onto Von's - memory-promotion direction under `1962`. -- [Agentic Confidence Calibration](https://arxiv.org/abs/2601.15778) and - [Four-Axis Decision Alignment for Long-Horizon Enterprise AI Agents](https://arxiv.org/abs/2604.19457) - both reinforce that aggregate success is too weak a target. Richer evaluator - axes for calibration, abstention, reasoning quality, and groundedness are - needed before broader self-improvement rollouts. That is the main reason - the evaluator line was decomposed before `1963`. -- [Groundedness in Retrieval-augmented Long-form Generation: An Empirical Study](https://arxiv.org/abs/2404.07060) - reinforced that answer correctness is too weak if generated claims are not - actually grounded in retrieved evidence. -- [Calibrating the Confidence of Large Language Models by Eliciting Fidelity](https://arxiv.org/abs/2404.02655) - and [BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidence](https://arxiv.org/abs/2604.03216) - reinforced that calibration and abstention need distinct evaluator treatment - rather than one generic confidence field. -- [Process Reward Models for LLM Agents: Practical Framework and Directions](https://arxiv.org/abs/2502.10325) - reinforced that later self-improvement work benefits from machine-usable - process signals rather than only outcome labels. -- [OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflows](https://arxiv.org/abs/2508.09124) - reinforced that long-horizon quality has to be evaluated over history- and - dependency-sensitive tasks, not only atomic completions. -- [AgentDevel: Reframing Self-Evolving LLM Agents as Release Engineering](https://arxiv.org/abs/2601.04620) - is directionally aligned with the current Von state after `1987`/`1994`/`1993`: - keep a single canonical version line, require regression-aware promotion - evidence, and treat self-improvement as an auditable release pipeline rather - than an opaque in-agent recursion. -- [Governed Capability Evolution for Embodied Agents: Safe Upgrade, Compatibility Checking, and Runtime Rollback for Embodied Capability Modules](https://arxiv.org/abs/2604.08059) - reinforced staged validation, sandbox evaluation, shadow deployment, gated - activation, and rollback as first-class system phases rather than ad-hoc - recovery behaviour. -- [Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluation](https://arxiv.org/abs/2402.11443) - reinforced that the right benchmark-world line should be able to evolve or - harden over time rather than relying only on one static suite. -- [OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation](https://arxiv.org/abs/2604.10866) - reinforced the value of simulator-backed evaluation worlds plus controlled - fault injection when measuring robustness before canonical promotion. - -This literature did not overturn Von's design direction. It did sharpen the task -ordering. - -## Priority Ordering - -### Current Tranche Status - -The following architecture tasks from the 21-23 April tranche are now landed: - -1. `JVNAUTOSCI-1974` - Missing-tool-call retry no longer synthesises tool payloads from prompt text - on the live path. - -2. `JVNAUTOSCI-1975` - Dispatch surface-family inference in turn-contract preflight is now driven by - represented tool metadata rather than the previous hard-coded family table. - -3. `JVNAUTOSCI-1985` - Registry exposure and turn-record write/evidence semantics now consume the - unified tool-metadata path instead of separate Python-owned slabs. - -4. `JVNAUTOSCI-1973` - Workflow discovery no longer executes English keyword/name fallback on the - live routing path. - -5. `JVNAUTOSCI-1986` - Rumination relation candidate ranking and auto-apply semantics now resolve - from the linked knowledge-acquisition profile instead of Python keyword/class - heuristics. - -6. `JVNAUTOSCI-1988` - Main-turn execution now consumes represented turn-memory substrate directly. - -7. `JVNAUTOSCI-1987` - Episode evaluation now launches represented workflow-improvement proposal and - promotion workflows instead of relying on Python fallback suggestion - synthesis. - -8. `JVNAUTOSCI-1992` - User-only namespaces now stay on the represented self-improvement path - instead of failing closed behind an unnecessary org requirement. - -9. `JVNAUTOSCI-1994` - Workflow-authoring proposal persistence is now proposal-addressable rather - than singleton-per-workflow. - -10. `JVNAUTOSCI-1993` - The remaining self-improvement launch and benchmark-policy slab now resolves - from a represented Vontology-backed profile. - -11. `JVNAUTOSCI-1989` - This repeat review is now complete: the dated status note is refreshed, the - older sequencing companion note is marked as historical for ordering - purposes, the main monolith sizes were re-measured, and the remaining - umbrella tasks were updated with cold-start comments against the live code. - -12. `JVNAUTOSCI-1962` - The enduring-memory line now has a dated architecture note at - `docs/engineering/jvnautosci_1962_enduring_memory_architecture_2026-04-23.md`, - plus concrete linked child tasks for the first implementation slices: - `JVNAUTOSCI-1995`, `JVNAUTOSCI-1996`, and `JVNAUTOSCI-1997`. - -13. `JVNAUTOSCI-1964` - The evaluator line now has a dated architecture note at - `docs/engineering/jvnautosci_1964_evaluator_architecture_2026-04-23.md`, - plus concrete linked child tasks for the first implementation slices: - `JVNAUTOSCI-1998`, `JVNAUTOSCI-1999`, `JVNAUTOSCI-2000`, and - `JVNAUTOSCI-2001`. - -14. `JVNAUTOSCI-1963` - The isolated self-improvement-world line now has a dated architecture note - at - `docs/engineering/jvnautosci_1963_self_improvement_worlds_architecture_2026-04-23.md`, - plus concrete linked child tasks for the first implementation slices: - `JVNAUTOSCI-2002`, `JVNAUTOSCI-2003`, `JVNAUTOSCI-2004`, and - `JVNAUTOSCI-2005`. - -15. `JVNAUTOSCI-1998` - Critique memory, fallback episode assessment, and benchmark reporting now - share a canonical multi-axis evaluator contract across execution - correctness, grounded helpfulness, calibration and abstention, recovery - quality, and long-horizon task-state integrity, with legacy - verdict/confidence retained only as roll-up compatibility state. - -16. `JVNAUTOSCI-1999` - The evaluator line now has a represented grounded-helpfulness critic - workflow and prompt, plus shared evidence-bundle support for answer - artefacts, answer-support consistency evidence, and response-context - lineage. The live episode-evaluation path now persists grounded - helpfulness as a first-class evaluator axis rather than inferring it only - from the older format-over-content diagnostic. - -### P1 Next Tranche - -The next substantive sequence should now be: - -1. `JVNAUTOSCI-2000` - Calibration, abstention, and recovery-quality evaluator workflow. - -2. `JVNAUTOSCI-1995` - Canonical enduring-memory manifest and query surface. - -3. `JVNAUTOSCI-2001` - Long-horizon task-state reconstruction and memory-conditioned evaluator - support. - -4. `JVNAUTOSCI-2002` - Represented candidate-world manifests and proposal-to-experiment-run - linkage for workflow self-improvement. - -5. `JVNAUTOSCI-2003` - Evaluator- and memory-backed baseline-vs-candidate comparison for - self-improvement experiment runs. - -6. `JVNAUTOSCI-2004` - Experiment-backed promotion, bounded shadow evaluation, and rollback-safe - publication gating for workflow candidates. - -7. `JVNAUTOSCI-1996` - Represented memory-promotion and revision workflow. - -8. `JVNAUTOSCI-1997` - Long-horizon enduring-memory evaluation and task-state reconstruction - harness. - -9. `JVNAUTOSCI-2005` - Prompt, policy, and retrieval or memory candidate-world expansion beyond - the workflow-first line. - -Why this is now the right order: - -- `1964` is now done as architecture and decomposition work, so the next moves - should be concrete evaluator slices rather than another umbrella pass. -- `1998` and `1999` are now landed, so `2000` and `2001` can build on a live - stored axis contract plus a represented grounded-helpfulness critic rather - than inventing local evaluator result shapes. -- `2000` now comes before the later `1963` child tasks because - self-improvement and - promotion gating become more valuable once they can reason over richer - evaluator outputs for grounded helpfulness, calibration, abstention, and - recovery quality. -- `1995` comes before the long-horizon evaluator slice because `2001` should - score memory-conditioned continuity against stable memory refs rather than - bespoke point lookups. -- `2002` now comes before the later self-improvement-world slices because the - live code still lacks one explicit candidate-world identity linking proposal, - experiment run, benchmark world, and baseline. -- `2003` and `2004` then build on that manifest plus the richer evaluator and - memory evidence from `1998`-`2001` and `1995`. -- `1996` and `1997` remain cold-startable enduring-memory slices rather than - abstract future work, but broadening candidate worlds beyond workflows is now - lower priority than landing the workflow-first experiment and evaluation line. - -### P2 Longer-Horizon Platform Work - -These remain important, but they are not the sharpest next moves on the current -code: - -- `JVNAUTOSCI-1961` - Minimal-imposition benchmark and deployment acceptance model. - -- `JVNAUTOSCI-1965` - Represented multi-agent coordination, delegation, and shared-memory - substrates. - -- `JVNAUTOSCI-1966` - Multimodal evidence fusion and provenance architecture. - -- `JVNAUTOSCI-1967` - Capability-suite and secure partner-deployment architecture. - -- `JVNAUTOSCI-1968` - Represented normative-governance, uncertainty, and escalation architecture. - -- `JVNAUTOSCI-1120` - Frontend monolith deconstruction. - -Why this is P2: - -- these are still real gaps, but they are not the sharpest blockers to the - current memory/evaluator/self-improvement architecture line; -- `1961` becomes more valuable once richer evaluator outputs exist to serve as - benchmark evidence rather than only post hoc logs; and -- monolith reduction remains worthwhile, but it is no longer the clearest - AGI-shaping next move relative to the broader memory/evaluator substrate work. - -## Existing Tasks Not Duplicated - -Some older tasks remain useful as umbrellas or adjacent work, but were not -reopened or duplicated here: - -- `JVNAUTOSCI-1913` remains the main anti-drift / monolith-pressure umbrella. -- `JVNAUTOSCI-1960` remains the broader minimal-imposition agentic programme - epic. -- `JVNAUTOSCI-1969` remains a worthwhile MCP contract/diagnostics audit, but it - is not the same as the current remaining memory/evaluator/self-improvement - gaps. -- `JVNAUTOSCI-768` and `JVNAUTOSCI-770` still point in the represented tool - metadata direction, but their wording remains materially stale and they should - not be treated as the current implementation guide without reinterpretation. - -## Ongoing Review Cadence - -`JVNAUTOSCI-1989` has now executed the first bounded repeat review after the -same-day architecture tranche. The review cadence itself should remain in force. - -After completion of any substantive P1 slice under `1998`, `1999`, `2000`, -`2001`, `1963`, `1995`, `1996`, or `1997`: - -1. rerun a bounded architecture scan against the then-current code; -2. re-measure the main monolith sizes and re-check the strongest remaining live - Python authority seams; -3. update this note and any now-stale sequencing companion notes; -4. refresh Jira comments or wording where the cold-start framing has gone stale; - and -5. only then decide the next highest-priority tranche. - -The key rule is simple: - -> do not keep steering work from old architecture notes once the live code has -> moved and the highest-value remaining gaps have changed. - -## Doc Status - -This repeat review plus the `1962` and `1964` architecture passes produced five -documentation conclusions: - -- this dated status note needed a real refresh and now reflects the live - post-`1993`/`1994`/`1989`/`1962`/`1964` state; -- `docs/engineering/jvnautosci_1962_enduring_memory_architecture_2026-04-23.md` - now carries the concrete enduring-memory architecture and first child-task - decomposition; -- `docs/engineering/jvnautosci_1964_evaluator_architecture_2026-04-23.md` - now carries the concrete evaluator architecture and first child-task - decomposition; -- `docs/engineering/recent_architecture_progress_2026-04-20.md` still remains - useful as a historical anti-drift note through the April 21 tranche, but its - sequencing advice should no longer be used as the current programme-wide next - step list; and -- `docs/engineering/Von_for_AgenticAI.md` did not require direct editing in this - review because its long-horizon design claims remain directionally correct. - The stale surfaces were the dated status and sequencing notes, not the - underlying design intent. diff --git a/docs/engineering/arxiv_paper_workflow_analysis.md b/docs/engineering/arxiv_paper_workflow_analysis.md deleted file mode 100644 index 45a798532..000000000 --- a/docs/engineering/arxiv_paper_workflow_analysis.md +++ /dev/null @@ -1,318 +0,0 @@ -# Arxiv Paper Workflow: Comparative Architectural Analysis - -> **Document status: Dated comparative diagnostic; not a current -> implementation guide.** Workflow inventories, Vontology counts, line anchors, -> and Jira conclusions describe the 7 April 2026 evidence base and require live -> revalidation. - -- **Kind:** Comparative diagnostic analysis -- **Lifecycle:** Frozen -- **Authority:** Evidence only -- **State as of:** 2026-04-07 -- **Subject:** Analysis of `#V#arxiv_paper_representation_workflow` and the Von - control plane - -**Sources:** -- [Doc A: Chat-Turn Workflow Control Plane Analysis](chat_turn_workflow_control_plane_analysis.md) -- [Doc B: Workflow Execution Architecture Analysis](workflow_execution_architecture_analysis.md) -- [Reference: Von Workflow Language Manual](von_workflow_language_manual.md) - ---- - -## 1. Executive Summary - -This document synthesises a comparative analysis of two engineering papers examining Von's workflow-first architecture. Both papers agree that the system is in a transitional "neuro-symbolic" state where Vontology-authored workflows are structurally present but often subverted by legacy Python control logic. The primary failure mode identified is the **"communicative gap"**: the system performs successful represented-state mutations (creating concepts, linking authors) but fails to report these durable changes to the user because the response-rendering path collapses rich workflow telemetry into opaque summaries. - ---- - -## 2. Core Points of Agreement - -The two source documents reach a consensus on the following architectural foundations: - -### 2.1 The 5-Layer Intended Hierarchy -Both documents endorse the same five-layer execution model as the correct vision for Von: -1. **Chat Turn Workflow**: The top-level lifecycle controller. -2. **LLM Analysis & Dispatch**: Intent interpretation and capability selection. -3. **Workflow Instantiation**: Loading authoritative VWL definitions from the KB. -4. **Step-by-Step Execution**: Managed by the VWL engine. -5. **Tool & Skill Use**: Subordinate execution surfaces within workflow steps. - -### 2.2 Python's Support Role -There is total agreement that Python code should strictly provide **reusable execution support** (action handlers, validators, telemetry) rather than carrying "hidden" task policy or hardcoded orchestration that VWL could already express. - -### 2.3 The arXiv Workflow Reality -Both documents confirm that `#V#arxiv_paper_representation_workflow` is a **genuine VWL workflow** in Vontology, not a monolithic Python script. However, they agree it currently functions as a "black box" because: -- Its steps bottom out in deterministic Python handlers. -- The LLM is effectively excluded from the execution loop once the workflow starts. - -### 2.4 The Response Rendering Gap -Both papers identify the main user-facing failure as a **lack of operational truthfulness**. The system: -1. Executes the workflow successfully. -2. Detects side-effects (file copies, concept creation) in telemetry. -3. **Suppresses** these side-effects during response assembly, favouring a compact content summary (e.g., the paper's abstract) over an account of what was actually done. - ---- - -## 3. Key Differences in Emphasis - -While the documents agree on the "what," they differ slightly on the "why" and where the primary corrective pressure should be applied: - -| Feature | Doc A (Control Plane Analysis) | Doc B (Execution Architecture) | -| :--- | :--- | :--- | -| **Primary Culprit** | **Lifecycle Ownership**: The Python route (`von_routes.py`) still owns the turn; the workflow is "sidecar" telemetry. | **Execution Bypass**: The `selector_requests_custom_workflow` flag in `orchestrator.py` skips LLM supervision entirely. | -| **Focus** | Top-down control and routing policy. | Bottom-up execution modes and "monolithic" action handlers. | -| **Root of Silence** | Layered presentation backfill (narration/buttonify) detaches output from actual events. | The deterministic state machine in `engine.py` never "asks" the LLM to narrate progress between steps. | -| **Turn Workflow** | Emphasises that `#V#conversation_turn_execution_workflow` is still thin/transitional. | Highlights that its execution is currently hardcoded in Python method sequences. | - ---- - -## 4. Empirical Verification (Fact-Check) - -An investigation of the Vontology and codebase confirms the technical claims made in both documents: - -### 4.1 Vontology Findings -- **`#V#conversation_turn_execution_workflow`**: Confirmed as a "thin" concept in the KB. It contains only `critic` and `completion_gate` steps and explicitly carries a "capability-gap note" stating it is currently code-backed. -- **`#V#arxiv_paper_representation_workflow`**: Confirmed as a structurally complete multi-step workflow. However, it also carries a maintenance note admitting it relies on dedicated Python handlers because VWL lack certain source-specific acquisition primitives. - -### 4.2 Codebase Findings -- **The Bypass Flag**: The existence of `selector_requests_custom_workflow` (L22306 of [orchestrator.py](../../src/backend/integrations/internal_mcp/orchestrator.py)) is verified. This flag does indeed trigger a path that bypasses the standard tool-calling loop in favour of a direct `execute_workflow()` call. -- **Scraping Logic**: The `_render_custom_workflow_response_text()` method (L3773 of [orchestrator.py](../../src/backend/integrations/internal_mcp/orchestrator.py)) was confirmed. It performs a "best-effort" scrape of the workflow's output dictionary (looking for `response_text`, `summary`, etc.), which explains why non-textual side-effects like concept creation are omitted from the chat. -- **Turn Lifecycle**: Ownership of the turn lifecycle in [von_routes.py](../../src/backend/server/routes/von_routes.py) (L8414) is verified, confirming the "sidecar" status of the durable instance. - ---- - -## 5. Strategic Conclusion - -The two analyses provide a coherent roadmap for architectural repair: -1. **Restore LLM Supervision**: The "Custom Workflow" path must be evolved into a **Supervised Workflow** path where the LLM can narrate progress between deterministic steps. -2. **Move Control to the Turn Workflow**: The top-level request lifecycle in [von_routes.py](../../src/backend/server/routes/von_routes.py) and [orchestrator.py](../../src/backend/integrations/internal_mcp/orchestrator.py) must be refactored into steps of a truly executable `#V#conversation_turn_execution_workflow`. -3. **Operational Reporting**: The response rendering logic must be upgraded to treat **durable side-effects** (concept IDs, file copies) as first-class user-visible facts rather than optional telemetry. - -Von is not failing its vision, but it is currently "hiding" its success inside silent Python pipelines. - ---- - -## 6. Additional Fact-Checked Comparison - -### 6.1 Summary - -The two source papers agree on the main architectural diagnosis: Von's intended design is workflow-first, Vontology-authoritative, and LLM-supervised, but the current top-level chat-turn path is still materially controlled by Python route/orchestrator code. They also agree that the key user-visible failure in the observed arXiv turn was not that "nothing happened", but that durable side-effects were not surfaced faithfully to the user. - -They disagree mainly on how far the current system has already progressed toward that vision. The stronger disagreement is about whether parts of the current runtime are already genuinely workflow-executed or are better understood as workflow-labelled Python pipelines. Live inspection of the current workflow registry and Vontology surfaces shows that the present system is more workflow-real than the harsher paper suggests, but still more Python-led and communication-poor than either paper ultimately wants. - -### 6.2 Further points of agreement - -Both papers agree on all of the following. - -1. The intended architecture is a layered hierarchy of chat-turn control, LLM interpretation/dispatch, workflow instantiation, stepwise workflow execution, and subordinate tool or skill use. That matches the authority doctrine in `AGENTS.md` and the VWL manual. -2. Python is still above the workflow layer too often. In the live route path, `von_routes.py` submits a conversation-turn instance, then immediately calls `orchestrator.run(...)`, then later finalises the turn instance after the fact. -3. The custom-workflow branch is still selected in Python via `selector_requests_custom_workflow`, and then handed to `execute_workflow(...)`. -4. The user-visible response path still hides durable effects. `_render_custom_workflow_response_text(...)` prefers `response_text`, `summary`, and `final_response`, rather than generically surfacing created artefacts such as file copies and concept IDs. -5. The arXiv turn did real represented-state work. Live workflow trace inspection showed outputs including a `computer_file_copy_concept_id`, a `paper_concept_id`, author concept IDs, topic concept IDs, and successful verification. - -### 6.3 Further points of disagreement - -The main disagreements are about degree rather than direction. - -#### 6.3.1 How real the workflow layer already is - -`chat_turn_workflow_control_plane_analysis.md` argues that it would be inaccurate to say the system simply bypassed the workflow layer, especially for the arXiv path. - -`workflow_execution_architecture_analysis.md` presents a harsher reading in which the custom-workflow path turns authored workflows into opaque Python pipelines. - -Live registry and Vontology inspection support the milder reading more strongly: - -- `#V#arxiv_paper_representation_workflow` exists as a live Vontology-backed workflow definition -- it currently has a real multi-step state graph -- recent durable instances and execution traces exist for it - -So the more accurate description is: - -- the workflow structure is real -- the steps are often executed through deterministic Python-backed actions -- the top-level control and user explanation remain too Python-led - -#### 6.3.2 Whether the tool-calling path is itself a real workflow - -The two papers diverge sharply here. - -The control-plane paper says `#V#tool_calling_workflow` is already represented as a real workflow. - -The execution-architecture paper describes the tool-calling path more like pure LLM improvisation that does not really leverage authored VWL workflows. - -The current runtime supports the control-plane paper more strongly: - -- the tool-pipeline path calls `execute_workflow(tool_dispatch_workflow_id, ...)` -- `#V#tool_calling_workflow` exists in the workflow registry -- its `respond` step is authored with `workflow_step_execution_mode=llm` - -So the tool path is not best described as workflow-free improvisation. It is better described as an authored workflow whose central reasoning step is LLM-mode. - -#### 6.3.3 Whether narration and buttonify are only conceptual labels - -The execution-architecture paper treats `#V#chat_buttonify_workflow` in particular as if it were mainly a concept or telemetry artefact whose real execution still lives in Python. - -The current route and workflow surfaces show a more mixed reality: - -- `von_routes.py` invokes both narration and buttonify through `orchestrator.execute_workflow(...)` -- Vontology-backed runtime definitions exist for both `#V#chat_narration_workflow` and `#V#chat_buttonify_workflow` -- `#V#chat_narration_workflow` has a genuine `llm` execution-mode step for narration rendering -- recent durable instances exist for `#V#chat_buttonify_workflow` - -So these workflows are not merely conceptual. They are real runtime workflows, even if their placement after main response assembly still contributes to the communication gap. - -#### 6.3.4 What is most wrong beneath the workflow surface - -The control-plane paper puts stronger weight on semantically heavy Python domain mutation, especially in the scholarly-paper/arXiv materialisation path, including direct repository mutation below the service layer. - -The execution-architecture paper concentrates more on: - -- missing LLM supervision during workflow execution -- lack of progress/event emission -- the black-box nature of `execute_workflow()` - -#### 6.3.5 Whether certain judgement steps should be `llm` steps - -The execution-architecture paper pushes harder for an audit of steps such as `arxiv.decide_acquisition_mode`, suggesting some judgement steps may be misclassified as deterministic. - -The control-plane paper is more willing to treat source normalisation, acquisition-mode selection, cache inspection, and verification as legitimate deterministic support primitives beneath the workflow. - -Current authored state shows `#V#workflow_step_arxiv_paper_representation_workflow_decide_acquisition_mode` is explicitly marked `workflow_step_execution_mode=deterministic`. - -The VWL manual requires prompt-bearing reasoning steps to compile as `llm`, but it does not say that every substantive decision must be LLM-authored. So this is best understood as a design question, not a clear present conformance violation. - -### 6.4 Fact-check corrections and clarifications - -#### 6.4.1 Workflow authority and transitional repo-side artefacts - -The VWL manual supports both papers' core authority claims: - -- Vontology-native workflow state is the intended authority surface -- repo-side seed bundles are transitional and non-authoritative by default -- backend code should mainly provide reusable support surfaces - -#### 6.4.2 Conversation-turn workflow: real, but still thin - -Live workflow inspection confirms that `#V#conversation_turn_execution_workflow` exists as a Vontology-backed durable workflow with a current runtime definition. - -Its present runtime shape is thin: - -- `turn_execution.critic` -- `turn_execution.completion_gate` -- terminal completion - -That supports the control-plane paper's claim that the workflow is currently thin/transitional. - -It does not support the stronger suggestion that the current live definition already governs context building, workflow discovery, dispatch, narration, and full response composition. - -#### 6.4.3 The exact "not runnable" warning should not be over-generalised - -The runtime evidence complicates the broader warning interpretation in the harsher paper. - -Current fact-checking showed: - -- `#V#chat_buttonify_workflow` has live runtime definitions and recorded instances -- `#V#arxiv_paper_representation_workflow` has live runtime definitions and recorded instances -- `#V#conversation_turn_execution_workflow` has bound actions and recorded workflow-use aggregates in Vontology - -So a warning such as "workflow is not runnable; instance was not created" should be interpreted as a path-specific submission or probe result, not as proof that the named workflow is generally unreal or non-executable in the current system. - -For `#V#conversation_turn_execution_workflow` specifically, the strongest supported claim is: - -- it is not yet the true top-level controller of the request path - -rather than: - -- it simply lacks executable reality altogether - -#### 6.4.4 The arXiv workflow and launch-input claim are supported - -Live Vontology inspection confirms that `#V#arxiv_paper_representation_workflow` currently publishes a launch-input contract that requires `prompt`. - -So the claim that unresolved `prompt` could truthfully trigger a launchability warning is well supported. - -#### 6.4.5 Tool-calling workflow is a real authored workflow - -The current system contains a live `#V#tool_calling_workflow` definition with: - -- `tool_calling.preflight_requirements` -- `tool_calling.respond` -- `turn_execution.critic` -- `turn_execution.completion_gate` - -Its `respond` step is currently authored as `llm` execution mode. This is an important correction to any reading that treats the tool path as merely improvisational and non-workflow-governed. - -#### 6.4.6 Renderer critique is strongly supported - -The criticism of `_render_custom_workflow_response_text(...)` is strongly supported by code inspection. - -It prioritises a small family of text-like fields and a few experiment-oriented fields. It does not generically surface outcomes such as: - -- `file_copy_concept_id` -- `paper_concept_id` -- `author_concept_ids` -- `topic_concept_ids` - -That directly explains why a workflow can succeed in durable state while still producing a user-visible answer that sounds like "just a summary". - -### 6.5 Bottom line of the added analysis - -Neither paper should be adopted wholesale over the other. - -The control-plane paper is more accurate about how much genuine workflow structure already exists in the runtime. - -The execution-architecture paper is more forceful about the experiential and communicative cost of the current execution model. - -Taken together, and corrected by live workflow and Vontology inspection, they support this balanced view: - -- Von is not merely faking workflows -- Von is also not yet living under workflow-first top-level control -- the arXiv paper path is genuinely workflow-executed -- but the current user experience still makes those workflows feel like silent backend pipelines - ---- - -## 7. Independent Fact-Check Results - -The following fact-checks were performed against the live codebase and Vontology on 2026-04-07, using direct code inspection and MCP tool queries against the production ontology. - -### 7.1 Claims Verified as Accurate (Both Documents) - -| Claim | Evidence | -|---|---| -| `selector_requests_custom_workflow` flag exists at line 22306 of `orchestrator.py` with the described conditions | Confirmed — exact code matches both documents | -| `_render_custom_workflow_response_text()` scrapes fields (`response_text`, `summary`, `final_response`, `verdict`, etc.) and never surfaces `file_copy_concept_id` or `paper_concept_id` | Confirmed — function checks ~11 field names, none artefact-related | -| `#V#conversation_turn_execution_workflow` has 3 steps: critic → completion_gate → completed | Confirmed in Vontology (hasStep relation shows exactly these 3 step concepts) | -| The system collects durable workflow side-effects but only for telemetry | Confirmed — `_collect_workflow_durable_side_effects()` exists, detects `*_created_ids`/`*_updated_ids` patterns, feeds them to telemetry summaries only | -| `execute_workflow()` is a synchronous black-box call | Confirmed — blocking call, no async patterns | -| `WorkflowExecutor.run()` produces only internal telemetry, no user-facing events | Confirmed — traces, metadata validation events, and runtime events are internal only | - -### 7.2 Claims Verified as Accurate (Control Plane Document Only) - -| Claim | Evidence | -|---|---| -| `_register_python_defined_workflows()` is effectively a no-op for production | Confirmed — function body is `return None` with explicit comment that "Vontology is now authoritative" | -| The arXiv workflow steps include `normalise_arxiv_source`, `fetch_arxiv_metadata`, `decide_acquisition_mode`, `finalise_cached_pdf`, `recover_partial_cache_state`, `download_or_finalise`, `delegate_to_general_paper_workflow`, `verify_arxiv_path`, `completed`, `failed` | Confirmed in Vontology — the `hasStep` relation lists exactly 10 step concepts matching this inventory | -| Seed bundles are transitional infrastructure, not a subversion | Consistent with VWL manual §2.2, which explicitly permits seed bundles as "migration seed data" and notes `workflow_repo_seed_bootstrap.py` treats them as "seed fixtures only" | - -### 7.3 Claims Found Inaccurate (Execution Architecture Document) - -| Claim | Finding | -|---|---| -| `_register_durable_action_modules()` registers handlers from **20** workflow-family modules | **Actually 25 modules.** The extra 5 are: `skill_interop`, `context_bundle_actions`, `control_flow_actions`, `subworkflow_actions`, `testing_workflow_actions` | -| `#V#chat_buttonify_workflow` is "defined as a Vontology concept but its execution is embedded in the orchestrator's Python code rather than expressed as a runnable VWL program" | **Wrong.** Vontology shows `#V#chat_buttonify_workflow` as a fully runnable durable workflow with **506 attempts and 377 completions** (74.5% completion rate), 4 real steps (`assess_input`, `select_prompt`, `extract_options`, `completed`), and registered action handlers | -| The arXiv case study step sequence: `arxiv.normalise_source` → `arxiv.decide_acquisition_mode` → download → `scholarly_paper.materialise_from_file_copy` → `scholarly_paper.enrich_from_metadata` → `scholarly_paper.resolve_authors` → `scholarly_paper.verify_representation` | **Conflates the arXiv workflow with its sub-workflow.** The `materialise_from_file_copy`, `enrich_from_metadata`, and `resolve_authors` actions belong to `#V#scholarly_paper_representation_workflow`, which is invoked as a subworkflow via the `delegate_to_general_paper_workflow` step | -| `arxiv.decide_acquisition_mode` should potentially be an `llm`-mode step, citing VWL manual §4 | **Weakly argued; own qualifier undermines it.** The doc concedes "these specific steps do not currently carry prompt contracts." The VWL manual rule applies to steps that *have* prompt contracts, not to mechanically deterministic cache-existence checks | -| Startup bootstrap services are a "subversion" that ensures "perpetual availability of workflows whose entire step inventory is bound to monolithic Python handlers" | **Overstated.** VWL manual §2.2 explicitly anticipates this pattern as transitional, and `workflow_repo_seed_bootstrap.py` now preserves a valid Vontology workflow rather than overwriting it back to seed parity | - -### 7.4 Vontology State Summary (as of 2026-04-07) - -| Workflow Concept | Steps | Instances/Attempts | Status | -|---|---|---|---| -| `#V#conversation_turn_execution_workflow` | 3 (critic, completion_gate, completed) | 3 attempts, 100% completion | Exists but not the real turn controller — route/orchestrator Python owns the turn lifecycle | -| `#V#arxiv_paper_representation_workflow` | 10 steps (see §7.2) | 28 attempts, 82.1% completion | Fully operational; published lifecycle phase | -| `#V#chat_buttonify_workflow` | 4 steps (assess_input, select_prompt, extract_options, completed) | 506 attempts, 74.5% completion | Fully operational — contradicts execution architecture doc's claim | - -### 7.5 Summary Assessment - -The **control plane document** is more factually careful, more nuanced in distinguishing legitimate Python support from control-plane overreach, and more architecturally precise about where the top-level fix belongs. The **execution architecture document** provides a more thorough catalogue of the subversion patterns and a stronger articulation of the user-experience consequences, but contains several factual errors (workflow family count, buttonify runnability, arXiv step conflation) and tends to overstate the severity of patterns the VWL manual already anticipates as transitional. diff --git a/docs/engineering/arxiv_workflow_handoff_design.md b/docs/engineering/arxiv_workflow_handoff_design.md deleted file mode 100644 index 28dfad05d..000000000 --- a/docs/engineering/arxiv_workflow_handoff_design.md +++ /dev/null @@ -1,48 +0,0 @@ -# Phase 0 Design Note: JVNAUTOSCI-1763 -## Ceding Turn Control to Durable Conversation Workflows - -**Status**: Draft -**Related Issue**: JVNAUTOSCI-1763 -**Author**: Gemini CLI - -### 1. Objective -Transition the `/von/generate` route from a "Python-orchestrated" model to a "Workflow-owned" model. The route should serve as a lightweight entry point that instantiates `#V#conversation_turn_execution_workflow` and streams its lifecycle events directly to the client. - -### 2. Current Reality Analysis -Currently, `/von/generate` in `von_routes.py` acts as the "brain" of the turn. It manually sequences: -1. User context resolution and early history persistence. -2. `orchestrator.run()` call (which itself may bypass the LLM for "custom workflows"). -3. Manual backfilling of narration (`CHAT_NARRATION_WORKFLOW_ID`). -4. Manual execution of quick-reply generation (`CHAT_BUTTONIFY_WORKFLOW_ID`). - -While a durable instance of `#V#conversation_turn_execution_workflow` is registered today, it is used primarily for telemetry. The Python route still blocks on the orchestrator's completion before returning a single JSON response. - -### 3. Proposed Changes - -#### 3.1. Route Transformation -The `generate()` function in `von_routes.py` will be refactored to: -* Perform only the essential pre-flight checks (Auth, Namespace, Rate Limiting). -* Call `submit_verified_workflow_instance` for `#V#conversation_turn_execution_workflow`. -* Immediately return a streaming response using Server-Sent Events (SSE). - -#### 3.2. Workflow Ownership -The `#V#conversation_turn_execution_workflow` definition must be updated to encompass the entire turn lifecycle: -* **Step 1: Routing**: Execute the `WorkflowSelector` logic. -* **Step 2: Execution**: Dispatch to the selected sub-workflow (Tool Calling or Specialized). -* **Step 3: Refinement**: Parallel or sequential execution of Narration and Buttonify. -* **Step 4: Finalisation**: Assemble the final `OrchestratorResult`. - -#### 3.3. Streaming Event Protocol -The `WorkflowExecutor` and `DurableWorkflowExecutor` will be augmented to emit lifecycle events into a durable event log (e.g., a capped collection in MongoDB or a Redis stream). The SSE generator in the route will yield: -* `event: progress`: For phase transitions (e.g., `workflow_discovery`, `tool_execute`). -* `event: text_chunk`: For streaming LLM responses from within workflow steps. -* `event: result`: The final structured outcome of the turn. - -#### 3.4. Handling the "Custom Workflow" Bypass -The bypass logic currently in `orchestrator.py` will move into the `#V#conversation_turn_execution_workflow` VWL transitions. The decision to "skip the LLM and run deterministic actions" will be an authored policy in Vontology, making it transparent and editable without Python deployments. - -### 4. Implementation Phasing -* **Phase 0 (This Note)**: Architecture alignment. -* **Phase 1**: Implement SSE event bridge in `von_routes.py`. -* **Phase 2**: Author the "Master Turn" VWL definition. -* **Phase 3**: Migrate Narration/Buttonify logic into the master workflow. diff --git a/docs/engineering/assertion_and_propositional_sentence_ontology.md b/docs/engineering/assertion_and_propositional_sentence_ontology.md index c186c0626..8f000dd71 100644 --- a/docs/engineering/assertion_and_propositional_sentence_ontology.md +++ b/docs/engineering/assertion_and_propositional_sentence_ontology.md @@ -877,10 +877,10 @@ whole ontology: - [Ontology publication authority](ontology_publication_authority.md) governs publication of the new canonical types and predicates; adding type identity does not widen assertion visibility or publication authority. -- [Testing workflows and ephemeral theories](testing_workflows_ephemeral_theories_design.md) +- [Testing workflows and ephemeral theories](https://github.com/Strong-AI-Lab/Von/blob/5b2f8b976069935da73b82bca433fb3cefffcd3d/docs/engineering/testing_workflows_ephemeral_theories_design.md) supplies a current consumer and adapter target, not a second generic assertion ontology. -- [Vontology tooling from KA/KCAP/KR literature](vontology_tooling_from_ka_kcap_kr_literature.md) +- [Vontology tooling from KA/KCAP/KR literature](https://github.com/Strong-AI-Lab/Von/blob/5b2f8b976069935da73b82bca433fb3cefffcd3d/docs/engineering/vontology_tooling_from_ka_kcap_kr_literature.md) supplies the context-query and provenance-tool direction. - `JVNAUTOSCI-343` owns exact occurrence, claim/proposition, formalisation, alignment, lifecycle, and common retrieval. diff --git a/docs/engineering/automated_policy_learning_design.md b/docs/engineering/automated_policy_learning_design.md deleted file mode 100644 index 72f553d8b..000000000 --- a/docs/engineering/automated_policy_learning_design.md +++ /dev/null @@ -1,559 +0,0 @@ -# Automated Policy Learning Design - -**Status**: Proposed design guidance -**Date**: 2026-04-04 -**Updated**: 2026-09-04 - -## 1. Purpose - -This document describes how Von can learn from experience and from what people -and AI participants work out together. That learning is a central purpose of -the Von architecture, not merely a repair mechanism for failed turns. Turn -Execution Records (TERs), conversation situations, corrections, -demonstrations, successful practices, and human/AI discussion may all supply -candidate lessons. Candidate formation, evaluation, runtime activation, and -autonomous policy change remain distinct decisions. - -The immediate motivating case is `JVNAUTOSCI-2090`, linked to the -`JVNAUTOSCI-1894` replay programme. The same user request, `Tell me about -myself`, succeeded with `gpt-5.4-mini` but failed with `gemma4:26b`. The useful -lesson is not "ban Gemma" or "hard-code GPT for routing". The lesson is that Von -needs introspectable model-use learning from real conversation rollouts: -workflow stage, model, prompt variant, context lineage, tool adequacy, final -answer, and post-turn critique should all become evidence for future model -selection. - -## 2. Core Concepts - -### 2.1 Experience and discussion as learning substrates - -TERs are one important observation carrier, not the whole learning substrate. A -TER captures the context, action, and outcome of an execution turn, including: - -- The initial prompt and context. -- The tools invoked and their arguments. -- The intermediate reasoning steps (e.g., ephemeral theories). -- The final output and any associated reward or validation signals. - -The conversation situation preserves a different but equally important source: -what participants are trying to achieve, corrections and preferences they make, -reasons they exchange, candidate practices they formulate, and unresolved -questions. Learning must preserve the provenance and time of those -contributions rather than laundering discussion into execution telemetry. - -### 2.2 The Learning Progression - -The architecture separates four states rather than treating policy update as -the inevitable result of observation: - -1. **Observation or contribution**: Preserve the relevant experience, outcome, - correction, demonstration, question, rationale, or deliberative proposal in - its source situation, with contribution and evidence provenance. -2. **Attributed candidate**: Formulate zero, one, or several provisional - lessons with an explicit contributor, purpose, target, and root evidence. A - candidate is discoverable for inspection and revision but is not a runtime - instruction. -3. **Evaluated candidate**: Attach supporting, contrary, or inconclusive - evidence from a comparison proportionate to the candidate's actual claim. - Evaluation may strengthen, narrow, revise, reject, or leave the candidate - unresolved. -4. **Active projection**: Select a particular evaluated revision for a bounded - actor, audience, target, situation, and decision point through the - authoritative activation path. Record the exact projection so later outcome - evidence can challenge it. - -A single experience or human/AI discussion can justify retaining a narrow, -non-active candidate for later deliberation. It need not demonstrate recurring -harm, but neither does it prove efficacy or authorise runtime projection. -Candidates may remain useful knowledge even when no active consumer has yet -earned implementation. - -### 2.3 Whose Learning, Action, and Outcomes - -Learning records must not collapse different semantic roles into a single -`user`, `owner`, or `scope` field when the distinction matters: - -| Dimension | Question | -|---|---| -| **Contributor** | Who supplied the observation, correction, rationale, demonstration, or candidate? | -| **Actor** | Which person, agent, team, organisation, or other principal would act differently? | -| **Target role** | Which role, capability, workflow, stage, prompt, tool, or practice is the lesson about? | -| **Audience** | Who may inspect or receive the candidate or active projection? | -| **Beneficiary** | Whose interests or outcomes is the proposed learning intended to serve? | -| **Purpose** | What end, function, work product, or wider objective is being advanced? | - -Relevant outcomes may concern an individual, a role or team, an organisation, -a social group, a society, humanity, Earth, or particular ecosystems. These are -possible contexts and beneficiaries, not a fixed hierarchy, an authority -grant, or mandatory ceremony for every candidate. A lesson may have multiple -beneficiaries and purposes, and evidence may show gains for one alongside costs -or disagreement for another. Preserve that plurality rather than reducing all -outcomes to one scalar reward or assuming a single uncontested good. Visibility -comes from the source and intended audience; a broad beneficiary or purpose -does not make private material public or authorise anyone to act. - -## 3. Mechanisms for Learning and Policy Change - -### 3.1 Candidate Induction - -Instead of humans manually writing new rules in `AGENTS.md` for every edge -case, the system may induce candidate guidelines from experience. - -- **Lesson recognition**: Bounded experience or discussion can expose a useful - correction, successful practice, rationale, opportunity, or failure pattern. -- **Candidate proposal**: Von or a participant may formulate a concise, - attributed candidate without claiming that it is already effective. -- **Source-native retention**: Keep a provisional lesson in the conversation - situation or another existing carrier. Materialise it in Vontology only when - independent identity, cross-session revision, or a demonstrated consumer - makes that representation useful. -- **Optional lenses**: Successful practice, failure avoidance, and - low-imposition exploration are useful questions an inductor may ask. None is - a compulsory output category. An observation can yield no durable candidate, - one candidate, or several candidates of other kinds. - -For textual guidance, “approved” is not implied by induction. The -[represented-advice design](represented_advice_design.md) owns the proposed -semantics, applicability, projection, retraction, and ratchet test: an induced -item remains a defeasible candidate until evidence proportionate to its scope -supports activation. - -### 3.2 Retained Substrate and Retired Mechanism - -The September 2026 audit distinguished the purpose and stored data of the old -episode-learning work from its defective automatic guidance mechanism: - -- [`episode_critique_memory.v1`](../../src/backend/services/episode_critique_memory_service.py) - remains a useful source-native store for episode and request identity, - namespace/user/organisation context, critic and evaluator state, implicated - workflows, tools and concepts, remediation and Jira links, structured - improvement suggestions, evidence receipts, and self-improvement state. -- [Explicit episode evaluation](../../src/backend/workflows/durable/episode_evaluation_workflow.py) - can still persist that memory and route remediation. It launches bounded - self-improvement work only when the assessment explicitly sets - `maintenance_follow_up_recommended` to true; ordinary critique retention does - not imply a proposal. Existing represented - [proposal and promotion-evaluation workflows](../../src/backend/workflows/repo_seed_bundles/episode_self_improvement_workflow_seed_bundle.json) - can turn a selected workflow suggestion into a reviewable candidate and - record a recommendation. -- [Workflow Studio](../../src/frontend/web/von_interface/static/js/workflowStudioPage.js) - already projects recent workflow-targeted critique suggestions for human - inspection, including their target, rationale, proposed change, evidence - references, and source episode. It explicitly says that these guidance - surfaces are not automatically applied. This is an existing - candidate-discovery consumer, not evidence of runtime efficacy. - -That substrate should be extended where it fits rather than rebuilt. The -[current episode critic prompt](../../src/backend/workflows/repo_seed_bundles/episode_evaluation_prompt_seed.md) -is predominantly repair-oriented, so it does not yet supply the full positive -function: learning successful practices, attributed discussion-derived -lessons, or optional exploratory ideas. Explicit MCP reads now support -candidate deliberation, but there is not yet a configured active -decision-point projection for such advice. - -The retired path forced successful-run, failure-avoidance, and next-run -exploration text after every episode; stored literal no-lesson output; appended -items without an applicability-aware active selection; and exposed candidate -history to proposal, evaluation, or release contexts. Those mechanisms are not -requirements of learning. The three purposes survive only as optional lenses, -and candidate discovery remains separate from active projection. - -The bounded replacement implements `learning_candidate.v1` as a non-active -`#V#artifact` with a canonical text body and source, contribution, target, -audience, beneficiary, purpose, visibility, and revision metadata. Internal MCP -operations support capture, get, list, and revision from a verified episode -critique or actor- or organisation-visible conversation source; an -unmaterialised chat-history session is the narrower actor-owned form. There is -no automatic trigger, inductor, prompt injection, active selector, evaluator, -or promotion path. The new surface therefore preserves a discussion- or -experience-derived lesson for later human/AI deliberation without claiming that -it already improves runtime behaviour. - -The bounded audit found **no evidence of a knowledge-base advice ratchet**: no -relevant accumulated advice relations, scoped assertions, profiles, or live -prelude instances were found. The observed accumulation and ratchet evidence -were instead in Python code, tests, seed bundles and migration compatibility, -Jira records, and design documentation. Possible ratchets in represented -knowledge remain hypotheses to test if an active consumer is built, not -demonstrated harms that justify controls now. - -### 3.3 Retrieval Strategy Updates - -Retrieval policies may be adjusted from measured usage and outcome patterns. - -- **Relevance Evidence**: Record which fragments were projected or explicitly - referenced and the resulting outcome. Do not claim actual model use or causal - benefit without an independent comparison. -- **Weight Adjustment**: Propose down-weighting or archiving noise and boosting - useful structures for evaluation. Numeric retrieval/routing adaptation is a - specialised policy surface, not represented textual advice merely because it - uses the same evidence or release machinery. -- **Candidate discovery versus active projection**: Humans and AI participants - may inspect non-active candidates to deliberate, merge, revise, or reject - them. Runtime decisions receive only deliberately selected active revisions, - with exact target and exposure lineage. Archival retention without any - discoverability is not useful learning, but discoverability does not imply - activation. - -### 3.4 Dynamic Routing Adjustment - -Model and agent routing should improve through empirical performance data. - -- **Cost-performance evidence**: Where routing is the selected learning job, - retain the cost, latency, and relevant outcome evidence of compared models or - agents for the bounded task category. -- **Router candidate**: Propose a routing or escalation change when that - evidence distinguishes it from the current policy. Re-weight active routing - only after evaluation and activation through its own authority surface. - -## 4. Actor-Critic Model-Use Learning - -Von should treat model choice as a learnable, represented policy over workflow -stages where the evidence shows that fixed selection is inadequate. This is one -concrete branch of the broader learning architecture, not its definition. The -intended loop is actor-critic in the ordinary reinforcement-learning sense, but -grounded in Von's existing workflow, Vontology, prompt, and telemetry -architecture. - -Here, the **actor** is specifically the runtime model-use policy; it is not the -contributor, beneficiary, or audience defined in Section 2.3. It may choose a -model arm, prompt variant, fallback chain, and escalation threshold for a -workflow stage. Where cost is material, it may prefer the cheapest active model -shown to be adequate for the specific workflow, stage, prompt profile, data -sensitivity, and actor or organisation context. It may also escalate when an -observed validation or postcondition failure makes that response useful. - -The **critic** is a reviewer workflow that evaluates completed turn rollouts. -It reads the TER, LLM IO telemetry, model and prompt identifiers, context -lineage, tool calls, completion gates, final answer, user feedback, and -postcondition checks. Its job is to explain what happened, assign structured -outcome signals, and form attributed candidates when a useful lesson is -present. A stronger model can be a design variable when judgement quality needs -it, but is not a universal requirement. Candidate formation does not update the -actor policy; independent evaluation and scoped activation remain later steps. - -The existing explicit episode evaluator and `episode_critique_memory.v1` -provide much of this critic substrate already. They should be reused and -extended, especially to recognise successful practice and discussion-derived -lessons, rather than supplemented with another mandatory post-turn evaluator. - -The **rollout** is the full conversation turn execution, not just the final -assistant text. A rollout record for model-use learning should include the -available fields material to its claim, such as: - -- user request, namespace, authenticated context, and selected workflow; -- candidate workflows and exclusion reasons; -- stage-level prompt concepts, prompt variants, and effective context lineage; -- model provider, model id, model settings, latency, token counts, and cost; -- raw and parsed LLM outputs, including structured-output validity; -- tool plan, tool calls, tool results, and missing-tool diagnostics; -- completion gates, validator results, fallback/escalation events, and answer; -- user feedback or downstream acceptance evidence where available. - -The Gemma/GPT case demonstrates why this matters. The GPT-5.4-mini rollout chose -`#V#entity_information_retrieval_workflow`, used the expected Vontology tools, -and produced a grounded represented answer. The Gemma rollout selected -`#V#chat_assistant_workflow`, did no tool work, and ended with an operational -failure answer. The LLM IO showed several learnable failure signals: - -- selector prompt/context ambiguity: the model appeared to reason about - `Select workflow` instead of the actual user request; -- structured-output weakness: fenced JSON was treated as text and the - structured selector evidence was not fully captured; -- empty-output weakness: a later direct response returned an empty string with - success status; -- tool-adequacy failure: the completion report named required retrieval tools - that had not run. - -Those signals should train and certify model-use policy. They should not become -English-specific routing rules or Python-side semantic patches. - -## 5. Reward and Evaluation Signals - -When the learning job warrants a critic, it should preserve multiple -inspectable signals rather than replacing them with a single opaque reward. -Useful signals include: - -- **selector correctness**: whether the chosen workflow matched the request and - candidate evidence; -- **schema adherence**: whether the model returned valid, parseable structured - output without hidden repair; -- **context use**: whether the model attended to the actual user request and - required stage context; -- **tool adequacy**: whether required bounded tool families were called and - whether their results reached the answer; -- **groundedness**: whether claims are supported by represented facts, tool - output, or explicit uncertainty; -- **answer usefulness**: whether the user-visible answer satisfies the prompt - rather than exposing execution bookkeeping; -- **failure transparency**: whether failures are visible in telemetry and - intelligible to later diagnosis; -- **cost, latency, locality, and privacy fit**: whether a cheaper or local model - was sufficient without sacrificing task quality, where those dimensions are - material to the purpose; -- **recovery quality**: whether the actor escalated or retried appropriately - after validation failure. - -Outcome evidence may also concern effects on a role, team, organisation, social -group, society, humanity, Earth, or an ecosystem. Retain the dimensions that -are material to the candidate's stated beneficiaries and purpose, including -contrary or distributional effects; do not invent every dimension for every -turn. A scalar optimisation target may be computed for a bounded decision, but -it must not erase the underlying evidence, uncertainty, dissent, or plural -outcomes. - -Raw telemetry, documents, traces, and events should remain in their -fit-for-purpose stores. Reuse `episode_critique_memory.v1` and other existing -carriers for structured summaries and evidence locators. Materialise a -first-class Vontology evidence item only when stable identity, cross-session -retrieval, independent revision, governance, or an active consumer actually -needs it. Whatever the store, retain enough inspectable evidence to distinguish -the critic's verdict from the observation it interprets. - -## 6. Candidate and Active-Policy Representation - -An attributed candidate should first remain in the weakest existing carrier -that supports discovery, provenance, revision, and its intended audience. A -policy revision selected for runtime use should live in the authoritative -Vontology or model-policy surface, not in scattered Python conditionals. -Python may provide replay execution, telemetry extraction, validators, -persistence helpers, and policy resolution primitives, but should not silently -become the durable home of task-specific semantic policy. - -If a concrete model-use consumer earns represented evidence and policy, useful -concepts and predicates may include: - -- `#V#model_use_rollout_evaluation` for a critic verdict over one rollout; -- `#V#model_stage_suitability_evidence` for evidence that a model is or is not - adequate for a workflow stage; -- `#V#prompt_variant_suitability_evidence` for model-specific prompt variant - results; -- `#V#actor_policy_version` for a versioned model-selection policy; -- `#V#critic_workflow` for the workflow that evaluates rollouts; -- `#V#replay_set` for curated replay cases such as the `JVNAUTOSCI-1894` - prompt families; -- predicates linking evidence to workflow, stage, model, prompt variant, replay - set, metric vector, verdict, expiry, and provenance. - -These are candidate additions, not a schema that every observation must -instantiate. Search first for equivalent identities and extend the retained -episode evidence, critique, proposal, and policy surfaces where adequate. - -The existing workflow model policy schema remains the runtime control surface: -see `docs/engineering/workflow_model_policy_schema.md`. A future model-use -consumer may resolve entries like "cheap reliable selector for entity-relative -retrieval" through represented suitability evidence rather than hard-coding -`provider:model` strings as permanent policy. That goal does not imply that a -general active advice loop exists today. - -## 7. Prompt Variants and Model-Specific Adaptation - -Model-specific prompts can be useful when observed behaviour distinguishes a -prompt-fit problem from model, tool, transport, authority, or context failures. -Some models may need stricter JSON instructions, more examples, different -language, or different context ordering. A durable active variant should be -inspectable at the authority surface and its actual selection observable. This -is a model-use learning problem, not a reason to hide prompt rewrites in -Python. - -For example, if Gemma reliably misreads a selector instruction unless the user -request is repeated in a labelled field, that can justify a Gemma-specific -selector prompt variant. The critic should record the evidence, the actor should -select the variant only for matching model/stage conditions, and telemetry -should show that a variant was used. If Mistral or DeepSeek performs better with -another language or prompt style, the same mechanism can certify that variant -without changing the workflow's semantic authority. - -## 8. Promotion, Rollback, and Governance - -Only selection into active runtime use is a promotion. Retaining, displaying, -discussing, or revising a non-active candidate does not require the release -machinery below. For a consequential active policy change, use only the parts -of this path that its actual claim and residual consequence require: - -1. identify the exact candidate revision, actor, target, audience, beneficiary, - purpose, applicability claim, and root evidence that are material; -2. compare it with the best fair simpler baseline using real turns or bounded - replay where that comparison can change the decision; -3. evaluate the material outcome dimensions independently of the candidate's - own induction context, using deterministic checks or a suitable critic where - each adds value; -4. select it through the existing standing authority for that policy surface; -5. publish and read back the exact active locator, while preserving the - previous selection or an empty no-advice state when recovery needs it; and -6. observe actual projections and later outcomes so counterevidence can lead to - revision, supersession, retraction, or retest. - -Do not require every metric, neighbouring prompt family, evaluator type, -expiry, or rollback mechanism by default. Apply the lightest lifecycle that -preserves the active item's real scope, provenance, observation, and feasible -recovery. Human inspection should remain available throughout. Human approval -is required when the governing authority surface requires it or when a material -choice cannot be resolved from standing delegation and evidence, not simply -because a policy change is imaginable. - -For example, replacing a proven model on an important workflow stage may need a -reviewer and broader outcome comparison because the decision has a credible -quality and operability consequence. A bounded reversible ranking adjustment -with direct outcome observation may need less. Candidate capture alone changes -neither behaviour nor authority and should not inherit those controls. - -## 9. Learned Failure Modes and Risks to Observe - -The retired path provides direct design evidence for several traps. Preserve -the intended learning function while rejecting these mechanisms: - -- **forced lesson production**: requiring success, failure, and exploration - guidance after every episode, including literal no-lesson text, creates work - and apparent policy without a lesson; -- **additive latest-N activation**: treating recently appended text as active - and applicable lacks deliberate selection, deduplication, revision, or - retraction; -- **self-contaminated evaluation**: injecting candidate history into the - proposal's own evaluator or release context weakens independent evidence; -- **candidate-as-system-policy**: rendering unpromoted critique suggestions as - runtime system guidance falsely upgrades their epistemic and authority - status; -- **mandatory post-episode mutation**: an additional evaluator or write tail on - every episode needs a demonstrated learning job and consumer. - -Other failure modes should be monitored when a concrete implementation makes -them reachable: - -- **vacuous criticism**: critic output that says a turn failed but does not - identify the stage, telemetry fields, or policy implication; -- **reward hacking**: rewarding short, cheap, or schema-valid outputs that do - not answer the user; -- **overfitting to `JVNAUTOSCI-1894`**: claiming broad model suitability from - one replay family; -- **prompt variant sprawl**: creating many model-specific prompts without an - evidenced maintenance or removal path appropriate to their effect; -- **stale certification**: continuing to trust a model after provider changes, - local quantisation changes, prompt changes, or workflow changes; -- **hidden Python policy**: turning critic findings into hard-coded routing - branches instead of represented policy updates; -- **opaque aggregate scores**: losing the stage-level evidence that explains - why a model was accepted or rejected. - -This list is diagnostic, not a universal control checklist. A merely -hypothetical harm can motivate observation or a bounded experiment; it does not -justify restricting an otherwise authorised strategy before a credible causal -path or evidence shows that the harm is materially reachable. - -## 10. Implementation Slices - -Implementation should begin with reuse and one selected learning job, not a new -general loop. The current repository implements bounded non-active portions of -slices 1, 2, and 4; slices 3, 5, and 6 remain future, separately earned work: - -1. **Inventory and reuse the substrate.** Treat TERs, episode evidence bundles, - evaluator axes, `episode_critique_memory.v1`, explicit episode evaluation, - structured improvement suggestions, Workflow Studio inspection, and the - reviewable self-improvement proposal path as existing components. Extend - them only where the selected job exposes a concrete gap. -2. **Preserve discussion contributions.** Use or extend the smallest - source-native way to retain an attributed correction, rationale, successful - practice, or proposal from the conversation situation. The first - implementation links a concise candidate to an actor- or - organisation-visible conversation concept, or to an actor-owned chat-history - session, so ordinary capture does not require materialising the whole - conversation in Vontology. Link to root evidence rather than copying the - whole discussion into a new policy store. -3. **Broaden candidate formation.** Let the existing critic or a separately - justified inductor return zero, one, or several candidates. Treat success, - failure avoidance, and low-imposition exploration as optional lenses, and - preserve contributor, target, audience, beneficiary, purpose, and plural - outcome claims when material. -4. **Make non-active candidates discoverable and revisable.** Extend the - existing human inspection surface before creating a second inventory. The - first implementation exposes trusted-scope MCP capture, get, list, and - revise operations for AI-assisted deliberation, while Workflow Studio keeps - its existing human view of structured episode suggestions. There is not yet - a unified human candidate-management view. Do not inject this inventory into - runtime prompts. -5. **Select one real consumer.** Only after a candidate class and decision point - demonstrate value, add the smallest active locator and applicability-aware - projection. Compare it with the best fair simpler baseline and with existing - specialised learners rather than duplicating them. -6. **Observe exposure and outcome.** Record the exact active revision projected - at the decision, then attach supporting, contrary, or inconclusive outcome - evidence and exercise revision or removal when the observed result warrants - it. - -`JVNAUTOSCI-2090` remains one concrete model-routing branch. It may extend the -`JVNAUTOSCI-1894` replay harness and normalise stage IO, model settings, prompt -variant, context lineage, tool adequacy, completion gates, answer, cost, and -latency. Its fair baseline includes the existing workflow/model policy and any -current numeric outcome-derived selector learning; it should not establish a -general advice layer by assertion. - -The first repository slice demonstrates source-grounded non-active capture, -AI-assisted discovery, canonical read-back, and revision. It does not yet -demonstrate a human candidate-management view or improvement to a real -decision. A later slice may show that one evaluated candidate improves a real -decision through bounded active projection. Neither slice requires full online -reinforcement learning, materialising every signal in Vontology, or claiming -that the loop is already closed. - -## 11. Proportionate Governance and Stopping Rule - -Governance belongs at the effect it protects. Candidate capture needs source, -contribution, visibility, and epistemic status preserved; it does not need the -same release controls as an active production policy. Active changes need -evidence and recovery proportionate to their bounded claim and credible -residual consequence. - -- **Evaluate the claimed effect**: A broad or consequential active policy may - warrant matched regression evidence. A narrow observable and recoverable - projection may need only a bounded comparison and direct read-back. -- **Preserve authority distinctions**: Contributor, actor, target role, - audience, beneficiary, and purpose do not confer authority on one another. - Use the standing authority of the selected policy surface and ask only when a - material unresolved choice or missing authority requires it. -- **Keep recovery feasible**: Use an explicit active selection, prior revision, - empty selection, or other recovery mechanism when the implemented effect can - otherwise persist after contrary evidence. Do not demand instant rollback or - version every provisional observation when deletion or source-native revision - is adequate. -- **Let risks earn controls**: A specific, credible, materially unacceptable - outcome and a causal path or evidence that it is reachable may justify the - least restrictive effective control. Speculation alone should lead at most to - observation, not a pre-emptive ratchet of approvals, gates, schemas, or - prohibitions. Organisational review and approval procedures are interventions - too: they carry human burden and can ratchet just as code and tests do, so - retain them only while their stated need remains justified. -- **Stop when the decision is supported**: Do not expand evaluation to every - model, task, metric, beneficiary, or imagined failure. Stop when further - evidence is unlikely to change capture, activation, rollback, or rejection, - and report bounded uncertainty honestly. - -## 12. Related Design Documents - -- [JVNAUTOSCI-2719](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-2719) - — represented-advice retirement, retained learning substrate, and next - bounded candidate-learning slice -- `docs/engineering/workflow_model_policy_schema.md` -- `docs/engineering/prompt_programs_and_model_routing_playbook.md` -- `docs/engineering/agent_evaluation_and_research_uptake.md` -- `docs/engineering/agent_memory_and_enduring_knowledge.md` -- `docs/engineering/represented_advice_design.md` -- `docs/engineering/real_path_server_replay_and_telemetry_loop.md` -- `docs/engineering/jvnautosci_1964_evaluator_architecture_2026-04-23.md` - -## 13. Conclusion - -Von learns when it can preserve what experience and discussion reveal, make -attributed candidates available for later human/AI deliberation, and revise -them as evidence changes. That purpose is valuable before any candidate becomes -active. Runtime policy changes make a further claim: an evaluated revision -improves a real decision over the fair simpler baseline for its stated actors, -beneficiaries, and purposes. Only that later claim needs scoped activation, -exposure evidence, and proportionate recovery. - -The current repository contains useful episode evidence, critique, human -inspection, proposal, and promotion-recommendation substrate, plus a new -source-grounded non-active candidate capture and revision path. It still has no -active advice consumer or closed exposure-to-outcome learning loop. Building on -the retained substrate selectively preserves learning without repeating the -demonstrated code-and-process ratchet or imposing a new one in response to -harms that remain merely hypothetical. diff --git a/docs/engineering/chat_turn_workflow_control_plane_analysis.md b/docs/engineering/chat_turn_workflow_control_plane_analysis.md deleted file mode 100644 index 261e5493c..000000000 --- a/docs/engineering/chat_turn_workflow_control_plane_analysis.md +++ /dev/null @@ -1,429 +0,0 @@ -# Chat-Turn Workflow Control Plane Analysis - -> **Document status: Dated diagnostic snapshot; not a current implementation -> guide.** Present-tense implementation claims and source line numbers describe -> evidence available on 7 April 2026. Revalidate them against current code, -> live Vontology, telemetry, and Jira before acting. - -- **Kind:** Diagnostic analysis -- **Lifecycle:** Frozen -- **Authority:** Evidence only; not a specification -- **State as of:** 2026-04-07 -- **Author:** Analysis prepared for Michael Witbrock -- **Scope:** Intended chat-turn and workflow hierarchy versus the implementation - observed at that date - ---- - -## 1. Abstract - -Von's intended architecture is a layered neuro-symbolic control system in which the top-level chat turn is interpreted by an LLM, durable task policy is carried by KB-stored workflows, and tools and future skills sit beneath those workflows as subordinate execution surfaces. That doctrine is explicit in `AGENTS.md` and in the VWL manual: workflow, prompt, KB, and Vontology artefacts should hold durable policy, while Python should primarily provide reusable execution, validation, tooling, and telemetry support. - -The current codebase only partially realises that vision. Workflow-definition authority has moved significantly toward Vontology, and the arXiv path is genuinely represented as a KB-authored stepwise workflow rather than as one giant hidden Python workflow. However, the actual control plane for a user turn remains substantially Python-led. Route-level code still owns the top-level turn lifecycle. The orchestrator still decides routing, execution mode, launchability, override policy, and response rendering in Python. Presenter backfill layers still transform the final user-visible response after workflow execution. Semantically heavy Python domain logic still performs important represented-state mutations beneath workflow steps. - -The result is an architecture that looks workflow-centred in ontology and telemetry, but still behaves route-centred in execution control and renderer-centred in user experience. This document describes the intended hierarchy, why it is a good idea, what is already correct in the current implementation, and the concrete Python control surfaces that still subvert the design. - ---- - -## 2. The Intended Hierarchy - -The intended execution hierarchy is: - -1. `CHAT TURN WORKFLOW` -2. `LLM analysis` -3. `Initial dispatch` -4. `Workflow instantiation` -5. `Step-by-step workflow execution under KB-stored VWL definition` -6. `LLM use, tool use, and future skill use inside workflow steps` - -This hierarchy is the right one for Von. - -At the top level, the user is not talking directly to an inert workflow engine. The user is talking to an LLM-supervised assistant that interprets intent, decides whether a direct answer is appropriate, whether a workflow should be selected, whether tools can be used directly, and eventually whether a future skill should be invoked. That top-level layer is conversational, adaptive, and context-sensitive. - -Beneath that, durable policy is supposed to move into workflows. Once a workflow has been selected, the workflow definition should govern the step graph, branching rules, contracts, and postconditions. The LLM should still be usable within steps, but the workflow should determine what kind of step is being executed and what comes next. This gives Von explicit state, inspectable policy, resumability, postcondition checking, and a stable surface for revision and evaluation. - -Below workflows sit tools and future skills. These are subordinate execution capabilities, not the main carrier of durable task policy. They should be invoked within the semantic frame of the workflow, rather than replacing workflow authority. - -This is consistent with the repo's governing doctrine: - -- `AGENTS.md` says workflow-first and KB-authoritative is the default doctrine, and durable policy should live in workflow, prompt, KB, or Vontology artefacts rather than Python. -- `AGENTS.md` also says Python should usually provide reusable support surfaces rather than hidden task policy. -- `docs/engineering/von_workflow_language_manual.md` says Vontology workflow graphs, mappings, schedules, and event bindings are first-class implementation artefacts, and backend code should primarily supply reusable actions, validators, loaders, and telemetry rather than task-specific orchestration that VWL can already express. - ---- - -## 3. Why This Hierarchy Is a Good Idea - -This architecture is good for five main reasons. - -### 3.1 It keeps the LLM where it is strongest - -The LLM belongs at the interpretive and reasoning layers: understanding user intent, deciding whether a workflow is needed, choosing among workflow candidates, handling ambiguity, and reasoning inside those workflow steps that are genuinely open-ended. That is where the model's flexibility is valuable. - -### 3.2 It keeps durable policy explicit and inspectable - -Workflow graphs, prompt contracts, action contracts, context mappings, completion gates, and postcondition checks are not incidental implementation details. They are operational policy. Storing them in Vontology makes them inspectable, revisable, evaluable, and versionable in a way that Python conditionals are not. - -### 3.3 It enables a properly neuro-symbolic architecture - -The symbolic layer provides explicit state, typed entities, graph structure, durable memory, branching rules, and validation. The LLM provides interpretation, synthesis, planning, and conversational explanation. The combination is better than either alone. - -### 3.4 It improves observability and truthfulness - -If the workflow is the true control surface, the system can explain what workflow it selected, what step it is on, what tool it used, what artefact it created, and what still remains uncertain. The user should not have to infer structural state changes from telemetry after the fact. - -### 3.5 It reduces long-term technical debt - -When domain policy is encoded in Python orchestration paths, it becomes difficult to inspect, evaluate, and evolve. When policy sits in KB-authored workflows and prompts, Python can remain a reusable substrate rather than becoming the hidden home of business semantics. - ---- - -## 4. What the Current Code Gets Right - -It would be incorrect to say that the system is simply ignoring workflows or that all workflow logic still lives in Python. - -### 4.1 Production workflow definition authority is moving toward Vontology - -`src/backend/workflows/durable/registry_factory.py` explicitly says Python-defined workflows are no longer registered onto the production registry path: - -- `_register_python_defined_workflows(...)` is effectively a no-op for production runtime registration. - -This matters. It means the current production registry is not supposed to treat Python-authored workflow definitions as the authoritative workflow source. - -### 4.2 The stage model is already derived from workflow metadata - -`src/backend/workflows/conversation_turn_stage_model.py` says formal stage entries are derived from the authoritative workflow registry so state-to-stage mapping follows Vontology-authored workflow definitions rather than a duplicated static state catalogue. - -That is exactly the right direction. - -### 4.3 The tool-calling workflow is represented as a real workflow - -In `src/backend/workflows/repo_seed_bundles/canonical_workflow_publication_seed_bundle.json`, `#V#tool_calling_workflow` is represented as: - -- `tool_calling.preflight_requirements` -- `tool_calling.respond` -- `turn_execution.critic` -- `turn_execution.completion_gate` - -This is aligned with the intended layered design: an LLM/tool step embedded inside a workflow, followed by critic and completion-gate machinery. - -### 4.4 The arXiv paper path is genuinely a workflow, not one hidden monolith - -The strongest inaccurate reading would be: "the system bypassed the workflow layer and ran one hard-coded Python arXiv ingestion pipeline directly." That reading is too strong. - -The represented workflow `#V#arxiv_paper_representation_workflow` in `src/backend/workflows/repo_seed_bundles/paper_representation_workflow_seed_bundle.json` is a real stepwise workflow. It includes: - -- `arxiv.normalise_source` -- `get_paper_metadata` -- `arxiv.decide_acquisition_mode` -- `finalise_cached_paper` -- `download_paper` -- `workflow_invoke_subworkflow` to `#V#scholarly_paper_representation_workflow` -- `scholarly_paper.verify_representation` - -So the workflow layer is real here. - ---- - -## 5. The Central Architectural Failure - -The main problem is not that workflows do not exist. The main problem is that the top-level chat turn is still not actually controlled by the chat-turn workflow. - -### 5.1 The route still owns the turn - -In `src/backend/server/routes/von_routes.py`: - -- `_submit_conversation_turn_instance()` submits a durable instance for `#V#conversation_turn_execution_workflow`. -- But immediately afterwards, the route calls `orchestrator.run(...)`. -- Later, `_finalise_conversation_turn_instance(...)` persists runtime payloads and marks the instance completed or failed. - -This means the conversation-turn workflow instance is currently more like sidecar telemetry and post-hoc state capture than the real control plane of the request. - -### 5.2 The represented conversation-turn workflow is currently thin and transitional - -`docs/engineering/turn_execution_completion_schema_proposal.md` explicitly says the intended responsibilities of `#V#conversation_turn_execution_workflow` are: - -- derive required effects -- execute tool plan -- hand off to critic workflow -- invoke completion gate workflow - -But the same document also says: - -- do not bind these new workflows yet -- only migrate `message.direct_created` after executable definitions and parity tests pass - -That transitional state is visible in the current seed bundle. In `canonical_workflow_publication_seed_bundle.json`, the current `#V#conversation_turn_execution_workflow` is effectively just: - -- `turn_execution.critic` -- `turn_execution.completion_gate` -- `completed` - -So the intended top-level controller exists conceptually, but the real request path is still largely living in route and orchestrator code. - ---- - -## 6. Where Python Still Subverts the Intended Design - -Not all Python here is wrong. Reusable action handlers, validators, loaders, telemetry surfaces, and deterministic wrappers are legitimate support surfaces. The problem is specifically that Python is still carrying too much control policy and too much domain semantics. - -### 6.1 Python still decides routing and execution mode - -In `src/backend/integrations/internal_mcp/orchestrator.py`, the orchestrator computes: - -- whether the selected workflow uses the tool-pipeline contract -- whether it uses the narration contract -- whether it prefers direct response -- whether the selector is requesting a custom workflow - -This is all control-plane policy in Python. - -That means the top-level turn is not being governed by an authored chat-turn workflow. It is being governed by Python logic that inspects workflow metadata and makes dispatch decisions before any durable top-level workflow instance truly owns the request. - -### 6.2 Python still decides launchability and override policy - -The orchestrator also: - -- builds custom-workflow dispatch data -- probes custom-workflow launchability -- evaluates override policy -- may replace the selected workflow based on launchability and override rules - -This is a major subversion of the intended architecture. The question of whether a workflow should remain selected, be overridden, or fall through to another execution mode is currently being decided by Python control logic, not by the authored top-level workflow or by an explicit represented routing policy surface that fully governs the turn lifecycle. - -### 6.3 Workflow execution is still an in-process Python call - -`orchestrator.execute_workflow(...)` ultimately hands the workflow definition to `self._workflow_executor.run(...)`. - -That is not inherently wrong; some execution engine must exist somewhere. But it matters architecturally because the durable workflow instance is not yet the real owner of the user turn. The durable instance is created, but the effective control plane is still "route calls orchestrator, orchestrator calls executor" rather than "top-level workflow instance governs the entire turn and all subordinate workflow invocations". - -### 6.4 Python still renders the user-visible response after workflow execution - -This is one of the most important practical problems. - -After a custom workflow completes, the orchestrator calls `_render_custom_workflow_response_text(...)`. That renderer mainly looks for: - -- `response_text` -- `summary` -- `final_response` -- `verdict` -- experiment-style metadata - -If none of those are present, it falls back to generic status text. - -Crucially, it does not generically surface durable artefacts such as: - -- `file_copy_concept_id` -- `paper_concept_id` -- other created or materialised concept identifiers - -So the workflow may have done exactly the right represented-state work, but the renderer still collapses the result into an opaque summary. - -### 6.5 The system already knows about workflow side-effects, but only in telemetry - -The same orchestrator file contains logic to: - -- collect durable workflow side-effects -- count them -- include them in workflow execution summaries - -So the system is already capable of recognising that a workflow created or materialised durable artefacts. The failure is not ignorance. The failure is that this information is not treated as a first-class user-visible outcome on the main response path. - -### 6.6 Presenter backfill layers further detach visible output from actual workflow events - -In `von_routes.py`, after orchestrator execution the route applies: - -- screen backfill modes -- narration workflow invocation -- buttonify workflow invocation - -These are not necessarily bad capabilities. But because they are layered on after the main workflow execution, they can further separate what the user sees and hears from what the workflow actually did. - -This helps explain the observed failure mode: structurally important side-effects occurred, but the user got a polished summary rather than an operationally faithful account. - ---- - -## 7. The arXiv Turn: What Actually Happened - -The arXiv turn should be described precisely. - -### 7.1 What is false - -It is false that the system simply bypassed the KB-stored workflow layer and ran one giant hidden "native Python arXiv ingestion workflow" instead. - -The represented workflow is real, and its step graph is real. - -### 7.2 What is true - -It is true that the workflow's steps bottom out in dedicated Python handlers and tool implementations. - -In `src/backend/workflows/durable/paper_representation_workflow.py`, there are dedicated action handlers for: - -- `scholarly_paper.normalise_inputs` -- `scholarly_paper.materialise_from_file_copy` -- `scholarly_paper.enrich_from_metadata` -- `scholarly_paper.resolve_authors` -- `scholarly_paper.verify_representation` -- `arxiv.normalise_source` -- `arxiv.decide_acquisition_mode` - -These are registered as reusable actions in the workflow action registry. - -In `src/backend/integrations/internal_mcp/catalogue.py`, the arXiv acquisition tools are also concrete Python handlers: - -- `_get_paper_metadata` -- `_download_paper` -- `_finalise_cached_paper` - -So the accurate description is: - -- the workflow layer was used -- but several of the workflow steps are implemented by dedicated Python domain logic -- and the user-facing response path subsequently hid the workflow's side-effects - -### 7.3 Why this still subverts the intended design - -Some of this Python is legitimate reusable support. For example: - -- extracting a canonical arXiv ID from inputs -- inspecting cache state -- wrapping an MCP tool -- validating representation postconditions - -Those can reasonably be seen as reusable deterministic primitives beneath the workflow. - -But other parts are semantically heavy. In `src/backend/services/arxiv_paper_link_service.py`, the scholarly-paper materialisation helpers: - -- create and type concepts -- attach names and descriptions -- assert publication dates -- create or resolve author concepts -- create or resolve topic concepts -- link file copies to papers -- verify the resulting representation -- request recommendation-refresh downstream effects - -This is more than a thin execution primitive. It is domain semantics. The helper even describes itself as "intentionally deterministic and tool-free so workflow-driven upload pipelines can assert strong postconditions without relying on chat heuristics". - -That determinism is understandable. But architecturally it means that semantically rich represented-state construction still lives substantially in Python. The workflow orchestrates the sequence, but Python still carries much of the actual scholarly-paper semantics. - -### 7.4 There is also direct repository mutation below the service layer - -Some of the arXiv paper link code drops to repository-level mutation via `ConceptsRepository.mutate_relationship_edge(...)`. - -Given the project's constitutional guidance to avoid direct DB access for Vontology-governed data, this is a particularly strong example of Python still carrying represented-state authority more directly than the intended architecture wants. - ---- - -## 8. Why the User Was Not Told About the File Copy or Paper Concept - -The best explanation is simple. - -The workflow did real represented-state work. The telemetry path could detect that work. But the main user-visible output path did not treat those durable state changes as mandatory facts to surface to the user. - -Instead, the system: - -1. executed the selected workflow -2. obtained a workflow result -3. rendered that result into a compact text response -4. then further transformed the visible output through screen/narration presentation layers - -Because `_render_custom_workflow_response_text(...)` does not generically surface durable artefact creation, the user got a content summary rather than an operationally faithful explanation. - -The system therefore looked successful in telemetry while still failing the more important user-level truthfulness test: telling the user what durable state had been changed on their behalf. - ---- - -## 9. What `dispatch_zero_execution` Actually Means - -`dispatch_zero_execution` should not be overread. - -The presence of a zero-tool-execution flag does not mean that nothing happened, nor that no workflow ran. In the current telemetry code, when the selected execution mode is `custom_workflow`, the system explicitly treats zero top-level tool invocations as expected if the custom workflow handled the turn. - -So `dispatch_zero_execution` in this context means something like: - -- the turn was handled by a custom workflow path -- no top-level free-form tool-call phase was needed - -It does not mean: - -- there was no workflow execution -- or that the system merely fabricated a success state - -The problem is not that the workflow did not run. The problem is that the user-visible response path did not faithfully describe what the workflow actually did. - ---- - -## 10. What the Warnings Mean - -The warnings observed in telemetry come from the verified workflow submission path. - -`src/backend/workflows/durable/workflow_instance_submission_service.py` emits exact messages when: - -- a workflow is not runnable -- launch-input resolution fails - -That explains the observed warnings. - -### 10.1 `#V#conversation_turn_execution_workflow` is not runnable - -This is consistent with the repo's own transitional design state. The conversation-turn workflow is intended, but the design note explicitly says it should not yet be bound until executable parity is in place. - -So the system currently has: - -- conceptual and representational support for this workflow -- telemetry and submission attempts involving it -- but not yet a clean end-to-end top-level execution path in which it actually governs the user turn - -### 10.2 `#V#arxiv_paper_representation_workflow` had unresolved `prompt` - -This also makes sense from the published workflow contract. The arXiv paper representation workflow declares `prompt` as a required launch input in its launch contract. - -So when the system attempted a verified submission or probe path without satisfying that contract in the expected form, the warning was truthful: the workflow could not start because required launch inputs were unresolved. - -These warnings are therefore not meaningless noise. They are evidence of a system in a transitional mixed state where represented workflow concepts, durable submission, and real request-path control have not yet fully converged. - ---- - -## 11. The Correct Overall Diagnosis - -The correct diagnosis is not "Von is secretly just a Python application with fake workflows". That is too crude and not fair to the real progress already made. - -The correct diagnosis is also not "the workflow vision is already fully realised and only the UI wording needs improvement". That is much too optimistic. - -The accurate diagnosis is this: - -- workflow-definition authority has moved significantly toward Vontology -- important execution paths, including the arXiv path, are genuinely represented as workflows -- but the top-level chat-turn control plane is still materially Python-led -- Python still decides routing, launchability, override policy, and response rendering -- semantically heavy domain mutation logic still lives beneath workflow steps in Python -- the presentation layer still suppresses user-visible reporting of durable side-effects - -So the system is best understood as a transitional architecture moving toward the intended hierarchy, but still subverting that hierarchy whenever Python remains the hidden home of control policy, domain semantics, or user-visible explanation. - ---- - -## 12. Bottom Line - -Your hierarchy is the right one: - -- top-level LLM handling the chat turn -- workflow instantiation beneath that -- step-by-step workflow execution guided by KB-stored workflow definitions -- tools and future skills lying beneath workflow steps - -That architecture is worth defending because it is the right way to build an inspectable, revisable, neuro-symbolic assistant with durable knowledge and truthful observability. - -The current code does not reject that vision. It only partially realises it. - -What is already correct should be preserved: - -- Vontology-authored workflow authority -- real VWL workflow definitions -- reusable deterministic action surfaces -- critic and completion-gate machinery - -What remains architecturally wrong is that Python still sits above the workflow layer too often, still carries semantically rich represented-state logic too often, and still controls the user-facing explanation of workflow outcomes too strongly. - -Until the chat-turn workflow becomes the true top-level control plane, and until workflow side-effects are surfaced as first-class user-visible facts rather than being hidden in telemetry, Von will continue to look more workflow-authoritative in its internal representations than it feels in actual user experience. diff --git a/docs/engineering/coding_agent_assignment_authority_design.md b/docs/engineering/coding_agent_assignment_authority_design.md index 1c65f90eb..4702d008c 100644 --- a/docs/engineering/coding_agent_assignment_authority_design.md +++ b/docs/engineering/coding_agent_assignment_authority_design.md @@ -64,7 +64,7 @@ assignment was sufficient to prepare the proposal without inventing dialogue. | [Worker runbook](codex_dgx_worker.md#interactive-coding-agent-messages), [Mac retry handoff](codex_dgx_worker.md#dgx-and-mac-retry-policy-handoff) | Documents separate DGX and VS Code identities and an operator-owned Mac task adapter with retained attempts, a launch boundary and lock. VS Code is no longer accurately described as necessarily reporting-only. This source does not identify the live Mac adapter's principal, active installation or clones. | | [Retry policy](../../scripts/codex_von_retry.py), `admission`; [supervision](../../scripts/codex_von_supervision.py), `reconcile`, `record_repair`, `repair_comments` | Explicit delegator retries and verified attempt-bound repair observations can admit continuation after current task checks. The supervisor controller can create repair tasks on the source delegator's behalf and attest a repair. Reporting routes, repair provenance and source/attempt checks are distinct from a general target-agent assignment grant. Completing a repair task alone does not resume its source. | | JVNAUTOSCI-2750 changes `9a7f2a735`, `d56f2a7b8`, retained through the refreshed baseline | Conversational recovery, execution-preference reads and coherent worker/backend releases remain in source. The supplied controller runtime projection identifies one release for worker and task-service modules; see the bounded runtime observation below. Activation work remains separate. | -| [Coordination proposal](multi_agent_coordination_design.md), relevant background to JVNAUTOSCI-1965 | Reuse explicit actor handoffs and context lineage only. Its orchestration and capability-negotiation programme is not a prerequisite. Live issue wording/status could not be re-read. | +| [Coordination proposal](https://github.com/Strong-AI-Lab/Von/blob/5b2f8b976069935da73b82bca433fb3cefffcd3d/docs/engineering/multi_agent_coordination_design.md), relevant background to JVNAUTOSCI-1965 | Reuse explicit actor handoffs and context lineage only. Its orchestration and capability-negotiation programme is not a prerequisite. Live issue wording/status could not be re-read. | | [Creation/recovery tests](../../tests/backend/test_coding_task_creation_recovery.py), [worker tests](../../tests/backend/test_codex_von_worker.py), [inbox tests](../../tests/backend/test_codex_von_inbox.py), [dispatch tests](../../tests/backend/test_task_dispatch_authority_service.py) | Existing isolated fixtures cover pilot paths, trusted dispatch actors, revocation and local-home admission. They do not establish normal production identity issuance, personal/team grants or current runtime installation. | The 17 September worker context explicitly disables Von MCP and reserves diff --git a/docs/engineering/contextual_knowledge_evolution.md b/docs/engineering/contextual_knowledge_evolution.md index b310f5cb7..e17dcf160 100644 --- a/docs/engineering/contextual_knowledge_evolution.md +++ b/docs/engineering/contextual_knowledge_evolution.md @@ -314,6 +314,6 @@ This guide does not require: namespace, and private-data authority. - [Agent memory and enduring knowledge](agent_memory_and_enduring_knowledge.md) governs promotion into durable memory. -- [Testing workflows and ephemeral theories](testing_workflows_ephemeral_theories_design.md) - and [Vontology tooling from KA/KR literature](vontology_tooling_from_ka_kcap_kr_literature.md) +- [Testing workflows and ephemeral theories](https://github.com/Strong-AI-Lab/Von/blob/5b2f8b976069935da73b82bca433fb3cefffcd3d/docs/engineering/testing_workflows_ephemeral_theories_design.md) + and [Vontology tooling from KA/KR literature](https://github.com/Strong-AI-Lab/Von/blob/5b2f8b976069935da73b82bca433fb3cefffcd3d/docs/engineering/vontology_tooling_from_ka_kcap_kr_literature.md) are research and design inputs, not current implementation authority. diff --git a/docs/engineering/durable_workflow_system_design.md b/docs/engineering/durable_workflow_system_design.md deleted file mode 100644 index a559b0dfc..000000000 --- a/docs/engineering/durable_workflow_system_design.md +++ /dev/null @@ -1,1096 +0,0 @@ -# Durable Workflow System Design - -> **Document status: Historical design precursor; non-normative.** Current-state -> and gap descriptions reflect February 2026 and have been overtaken by later -> durable-workflow implementation. Use the current VWL manual and live -> code/Vontology state for workflow planning. - -**Lifecycle**: Superseded design precursor -**Authority**: Historical only -**Version**: 1.0 -**Date**: 2026-02-03 -**Parent JIRA**: [JVNAUTOSCI-803](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-803) (LLM Workflows) -**Authors**: Von AI Assistant (GitHub Copilot), Michael Witbrock - ---- - -## Executive Summary - -This document extends the [LLM Workflows Design](llm_workflows_design.md) with **durable workflow execution** — the ability to persist workflow state, resume interrupted workflows, and schedule workflows for deferred/background execution. - -**Key Capabilities**: -1. **Checkpoint/Resume**: Long-running workflows can be interrupted (server restart, timeout) and resumed from last checkpoint -2. **Background Execution**: Workflows run asynchronously, detached from user sessions -3. **Scheduling**: Workflows can be triggered at specified times or intervals -4. **Observability**: Live status queries for in-progress workflows - -**Primary Use Cases**: -- RAG text relation sync (long-running batch operations) -- Rumination workflows (proactive knowledge enrichment) -- Scheduled report generation -- Multi-step research workflows - ---- - -## Table of Contents - -1. [Background & Motivation](#1-background--motivation) -2. [Current State Analysis](#2-current-state-analysis) -3. [Design Principles](#3-design-principles) -4. [Architecture Overview](#4-architecture-overview) -5. [Workflow Instance Model](#5-workflow-instance-model) -6. [Checkpoint & Resume](#6-checkpoint--resume) -7. [Scheduling System](#7-scheduling-system) -8. [Worker Architecture](#8-worker-architecture) -9. [API Design](#9-api-design) -10. [Vontology Integration](#10-vontology-integration) -11. [Migration Path](#11-migration-path) -12. [Implementation Plan](#12-implementation-plan) -13. [Testing Strategy](#13-testing-strategy) -14. [Operational Concerns](#14-operational-concerns) - ---- - -## 1. Background & Motivation - -### 1.1 Problem Statement - -The current workflow system (per [llm_workflows_design.md](llm_workflows_design.md)) has limitations for long-running operations: - -| Gap | Impact | -|-----|--------| -| **No persistence** | Server restart loses all in-progress workflows | -| **Synchronous execution** | Long workflows block conversation turns | -| **No scheduling** | No way to trigger workflows at specific times | -| **No live status** | Cannot query progress of background work | -| **Single-process** | Cannot distribute work across workers | - -### 1.2 Motivating Use Cases - -#### RAG Text Relation Sync (JVNAUTOSCI-1073 context) -- Syncing thousands of text relations to RAG store -- Can take 10+ minutes for large namespaces -- Currently blocks conversation turns -- Need: Background execution with progress tracking - -#### Rumination Workflows (JVNAUTOSCI-923) -- Proactive knowledge enrichment cycles -- Iterates over concepts, identifies gaps, queries LLM, updates graph -- Runs during idle periods, not tied to user requests -- Need: Scheduled execution, checkpoint on interrupt - -#### Research Report Generation -- Multi-step: gather sources → summarise → synthesise → format -- May take 30+ minutes for comprehensive reports -- User checks back later for results -- Need: Background with notification on completion - -### 1.3 Design Goals - -1. **Zero-loss durability**: No workflow progress lost on restart -2. **Transparent resumption**: Workflows resume automatically without user intervention -3. **Flexible scheduling**: Support one-time, interval, and cron-style triggers -4. **Minimal dependencies**: Use MongoDB (existing), avoid heavy frameworks -5. **Incremental adoption**: Existing workflows continue to work unchanged - ---- - -## 2. Current State Analysis - -### 2.1 Existing Components - -| Component | Purpose | Durability | -|-----------|---------|------------| -| `WorkflowExecutor` | State machine runner | ❌ In-memory only | -| `WorkflowExecutionTrace` | Post-hoc logging | ✅ MongoDB (30-day TTL) | -| `BackgroundTaskRegistry` | Thread-pool for async | ❌ In-memory only | -| `rag_indexing_worker.py` | Polling-based worker | ⚠️ Uses DB status flags | - -### 2.2 What Works Today - -The **RAG indexing worker** pattern demonstrates a working async model: -- Documents marked `PENDING` in MongoDB -- Worker polls for pending items -- Status updated to `INDEXED`, `FAILED`, or `SKIPPED` -- Resume on restart: re-polls pending items - -**Key insight**: MongoDB status flags provide implicit checkpointing. - -### 2.3 What's Missing - -1. **Generalised instance table**: Need a `workflow_instances` collection (not just traces) -2. **State serialisation**: Need to persist `WorkflowState` mid-execution -3. **Worker abstraction**: Decouple polling loop from specific workflow logic -4. **Scheduler**: Currently no way to trigger workflows by time - ---- - -## 3. Design Principles - -1. **Database is the source of truth**: All workflow state lives in MongoDB -2. **Idempotent operations**: Steps can be re-executed safely on resume -3. **Fail-safe defaults**: Incomplete workflows are detectable and resumable -4. **Backward compatible**: Existing synchronous workflows unchanged -5. **Observable progression**: Every state change is logged and queryable - ---- - -## 4. Architecture Overview - -### 4.1 Component Diagram - -``` -┌─────────────────────────────────────────────────────────────────┐ -│ Von Application Server │ -│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────────┐ │ -│ │ API Routes │ │ Orchestrator │ │ WorkflowScheduler │ │ -│ │ /workflows │ │ (sync calls) │ │ (timer-based) │ │ -│ └──────┬───────┘ └──────┬───────┘ └──────────┬───────────┘ │ -│ │ │ │ │ -│ │ ┌────────────┴──────────────────────┘ │ -│ │ │ │ -│ ▼ ▼ │ -│ ┌─────────────────────────────────────┐ │ -│ │ WorkflowInstanceManager │ │ -│ │ - create_instance() │ │ -│ │ - submit_for_execution() │ │ -│ │ - get_status() │ │ -│ │ - checkpoint() │ │ -│ │ - resume() │ │ -│ └─────────────────┬───────────────────┘ │ -│ │ │ -└────────────────────┼────────────────────────────────────────────┘ - │ - ▼ - ┌──────────────────────┐ - │ MongoDB │ - │ ┌────────────────┐ │ - │ │workflow_instances│ │ - │ │workflow_schedules│ │ - │ │workflow_executions││ - │ └────────────────┘ │ - └──────────┬───────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────┐ -│ DurableWorkflowWorker(s) │ -│ ┌──────────────────────────────────────────────────────────┐ │ -│ │ Poll Loop: │ │ -│ │ 1. Claim pending/resumable instances (atomic lock) │ │ -│ │ 2. Load workflow definition + state │ │ -│ │ 3. Execute steps (using WorkflowExecutor) │ │ -│ │ 4. Checkpoint after each step │ │ -│ │ 5. Mark completed/failed │ │ -│ └──────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────┘ -``` - -### 4.2 Data Flow - -1. **Submission**: Client/scheduler creates `workflow_instance` with `status=pending` -2. **Claim**: Worker atomically sets `status=running`, `locked_by=worker_id`, `lock_expires_at=now+TTL` -3. **Execution**: Worker runs steps, checkpointing state after each -4. **Completion**: Worker sets `status=completed` or `status=failed`, releases lock -5. **Resume**: On restart, worker finds instances where `status=running` AND `lock_expires_at < now` - ---- - -## 5. Workflow Instance Model - -### 5.1 MongoDB Collection: `workflow_instances` - -```typescript -interface WorkflowInstance { - // Identity - _id: ObjectId; - instance_id: string; // UUID, primary lookup key - workflow_id: string; // Definition ID (e.g., "#V#rag_sync_workflow") - - // Ownership - user_id: string; // Who initiated - org_id: string; // Organisation context - namespace: string; // Full namespace for data access - - // Status - status: "pending" | "running" | "completed" | "failed" | "cancelled" | "paused"; - created_at: Date; - started_at: Date | null; - completed_at: Date | null; - - // Execution state (for resume) - current_state: string; // State machine state ID - workflow_data: object; // Serialised WorkflowState.data - step_index: number; // For sequential workflows: last completed step - - // Progress tracking - progress_current: number | null; - progress_total: number | null; - progress_message: string | null; - progress_updated_at: Date | null; - - // Locking (for distributed workers) - locked_by: string | null; // Worker ID holding lock - lock_expires_at: Date | null; // Auto-release stale locks - - // Inputs/outputs - inputs: object; // Initial workflow inputs - outputs: object | null; // Final outputs (on completion) - - // Error info - error: string | null; - error_step: string | null; - retry_count: number; - max_retries: number; - - // Scheduling (if scheduled) - schedule_id: string | null; // Reference to workflow_schedules - - // Tracing - execution_trace_id: string | null; // Link to workflow_executions -} -``` - -### 5.2 Indexes - -```javascript -// Primary lookup -db.workflow_instances.createIndex({ instance_id: 1 }, { unique: true }); - -// Worker polling: find claimable instances -db.workflow_instances.createIndex( - { status: 1, lock_expires_at: 1 }, - { partialFilterExpression: { status: { $in: ["pending", "running"] } } } -); - -// User queries -db.workflow_instances.createIndex({ user_id: 1, created_at: -1 }); -db.workflow_instances.createIndex({ org_id: 1, status: 1 }); - -// Schedule association -db.workflow_instances.createIndex({ schedule_id: 1, created_at: -1 }); - -// TTL: auto-delete old completed instances after 30 days -db.workflow_instances.createIndex( - { completed_at: 1 }, - { expireAfterSeconds: 30 * 24 * 60 * 60, partialFilterExpression: { status: "completed" } } -); -``` - -### 5.3 Status Transitions - -``` - ┌─────────────────┐ - ┌─────────│ pending │◄────────┐ - │ └────────┬────────┘ │ - │ │ │ - │ │ claim │ schedule - │ ▼ │ triggers - │ ┌────────────────┐ │ - │ │ running │──────────┘ - │ └───┬────────┬───┘ - │ │ │ - │ success │ │ failure (retriable) - │ │ │ - │ ▼ ▼ - │ ┌────────────┐ ┌──────────────┐ - │ │ completed │ │ running │ (retry) - │ └────────────┘ └──────────────┘ - │ │ - │ │ failure (max retries) - │ ▼ - │ ┌──────────────┐ - │ │ failed │ - │ └──────────────┘ - │ - │ cancel request - ▼ - ┌────────────┐ - │ cancelled │ - └────────────┘ -``` - ---- - -## 6. Checkpoint & Resume - -### 6.1 Checkpoint Strategy - -**When to checkpoint**: -- After each workflow step completes successfully -- Before any operation that could fail (LLM call, external API) - -**What to checkpoint**: -- Current state ID -- Accumulated workflow data (context dict) -- Step completion index (for sequential flows) -- Trace so far (optional, for debugging) - -### 6.2 Checkpoint Implementation - -```python -class DurableWorkflowExecutor: - """Extended WorkflowExecutor with checkpoint support.""" - - def __init__( - self, - *, - registry: ActionRegistry, - instance_manager: WorkflowInstanceManager, - max_transitions: int = 50, # Higher limit for durable workflows - ): - self._registry = registry - self._instance_manager = instance_manager - self._max_transitions = max_transitions - - async def run_durable( - self, - instance_id: str, - *, - resume_from_checkpoint: bool = True, - ) -> WorkflowResult: - """Execute workflow with checkpointing.""" - instance = await self._instance_manager.load(instance_id) - - # Load definition - definition = self._load_definition(instance.workflow_id) - - # Restore state from checkpoint - if resume_from_checkpoint and instance.workflow_data: - context = dict(instance.workflow_data) - current_state = instance.current_state - else: - context = dict(instance.inputs) - current_state = definition.initial_state - - # Create environment - environment = WorkflowEnvironment( - user_id=instance.user_id, - org_id=instance.org_id, - namespace=instance.namespace, - ) - - # Execute with checkpointing - transitions = 0 - while transitions < self._max_transitions: - # Check for cancellation - if await self._instance_manager.is_cancelled(instance_id): - return WorkflowResult( - data=context, - completed=False, - final_state=current_state, - error="cancelled", - ) - - state_spec = definition.states.get(current_state) - if state_spec is None: - return WorkflowResult( - data=context, - completed=False, - final_state=current_state, - error=f"unknown_state:{current_state}", - ) - - # Execute actions - for action in state_spec.actions: - result = await self._execute_action(action, context, environment) - if not result.ok: - # Checkpoint failure state - await self._instance_manager.checkpoint( - instance_id, - current_state=current_state, - workflow_data=context, - error=result.error, - ) - return WorkflowResult( - data=context, - completed=False, - final_state=current_state, - error=result.error, - ) - context.update(result.outputs) - - # Check terminal - if state_spec.terminal: - return WorkflowResult( - data=context, - completed=True, - final_state=current_state, - ) - - # Evaluate transitions - next_state = self._evaluate_transitions(state_spec.transitions, context) - if next_state is None: - return WorkflowResult( - data=context, - completed=False, - final_state=current_state, - error="no_transition", - ) - - # CHECKPOINT after successful step - await self._instance_manager.checkpoint( - instance_id, - current_state=next_state, - workflow_data=context, - ) - - current_state = next_state - transitions += 1 - - return WorkflowResult( - data=context, - completed=False, - final_state=current_state, - error="transition_limit", - ) -``` - -### 6.3 Resume Logic - -```python -class WorkflowInstanceManager: - """Manages workflow instance lifecycle.""" - - async def find_resumable_instances( - self, - worker_id: str, - limit: int = 10, - ) -> list[WorkflowInstance]: - """Find instances that can be resumed by this worker.""" - now = datetime.now(timezone.utc) - - # Atomic claim: find and update in one operation - instances = [] - for _ in range(limit): - instance = await self._collection.find_one_and_update( - { - "$or": [ - # New pending instances - {"status": "pending"}, - # Stale running instances (lock expired) - { - "status": "running", - "lock_expires_at": {"$lt": now}, - }, - ], - }, - { - "$set": { - "status": "running", - "locked_by": worker_id, - "lock_expires_at": now + timedelta(minutes=5), - "started_at": {"$ifNull": ["$started_at", now]}, - }, - }, - return_document=ReturnDocument.AFTER, - ) - if instance: - instances.append(WorkflowInstance.from_doc(instance)) - else: - break - - return instances -``` - ---- - -## 7. Scheduling System - -### 7.1 MongoDB Collection: `workflow_schedules` - -```typescript -interface WorkflowSchedule { - _id: ObjectId; - schedule_id: string; // UUID - workflow_id: string; // Definition to run - - // Ownership - user_id: string; - org_id: string; - namespace: string; - - // Schedule config - schedule_type: "once" | "interval" | "cron"; - - // For "once" - run_at: Date | null; - - // For "interval" - interval_seconds: number | null; - - // For "cron" - cron_expression: string | null; // Standard cron format - - // State - enabled: boolean; - last_run_at: Date | null; - next_run_at: Date | null; - - // Inputs - default_inputs: object; // Passed to each workflow instance - - // Metadata - created_at: Date; - updated_at: Date; - description: string | null; -} -``` - -### 7.2 Scheduler Service - -```python -class WorkflowScheduler: - """Periodically checks schedules and creates workflow instances.""" - - def __init__( - self, - instance_manager: WorkflowInstanceManager, - check_interval_seconds: int = 60, - ): - self._instance_manager = instance_manager - self._check_interval = check_interval_seconds - self._running = False - - async def start(self): - """Start the scheduler loop.""" - self._running = True - while self._running: - try: - await self._process_due_schedules() - except Exception as e: - logger.error(f"Scheduler error: {e}") - await asyncio.sleep(self._check_interval) - - async def _process_due_schedules(self): - """Find and trigger due schedules.""" - now = datetime.now(timezone.utc) - - due_schedules = await self._get_schedules_collection().find({ - "enabled": True, - "next_run_at": {"$lte": now}, - }).to_list(length=100) - - for schedule in due_schedules: - # Create workflow instance - instance_id = await self._instance_manager.create_instance( - workflow_id=schedule["workflow_id"], - inputs=schedule["default_inputs"], - user_id=schedule["user_id"], - org_id=schedule["org_id"], - namespace=schedule["namespace"], - schedule_id=schedule["schedule_id"], - ) - - # Update schedule - next_run = self._calculate_next_run(schedule) - await self._get_schedules_collection().update_one( - {"_id": schedule["_id"]}, - { - "$set": { - "last_run_at": now, - "next_run_at": next_run, - }, - }, - ) - - logger.info( - f"Triggered schedule {schedule['schedule_id']}: " - f"created instance {instance_id}" - ) -``` - ---- - -## 8. Worker Architecture - -### 8.1 DurableWorkflowWorker - -```python -class DurableWorkflowWorker: - """Background worker that processes durable workflow instances.""" - - def __init__( - self, - worker_id: str, - instance_manager: WorkflowInstanceManager, - executor: DurableWorkflowExecutor, - *, - poll_interval_seconds: float = 5.0, - batch_size: int = 5, - heartbeat_interval_seconds: float = 60.0, - ): - self._worker_id = worker_id - self._instance_manager = instance_manager - self._executor = executor - self._poll_interval = poll_interval_seconds - self._batch_size = batch_size - self._heartbeat_interval = heartbeat_interval_seconds - self._running = False - self._current_instances: dict[str, asyncio.Task] = {} - - async def start(self): - """Start the worker loop.""" - self._running = True - logger.info(f"DurableWorkflowWorker {self._worker_id} starting") - - # Start heartbeat task - heartbeat_task = asyncio.create_task(self._heartbeat_loop()) - - try: - while self._running: - # Claim instances - instances = await self._instance_manager.find_resumable_instances( - self._worker_id, - limit=self._batch_size, - ) - - if instances: - for instance in instances: - if instance.instance_id not in self._current_instances: - task = asyncio.create_task( - self._process_instance(instance) - ) - self._current_instances[instance.instance_id] = task - else: - # No work, wait before polling again - await asyncio.sleep(self._poll_interval) - - # Clean up completed tasks - await self._cleanup_completed_tasks() - finally: - heartbeat_task.cancel() - await self._cancel_current_tasks() - - async def _process_instance(self, instance: WorkflowInstance): - """Process a single workflow instance.""" - try: - logger.info( - f"Processing instance {instance.instance_id} " - f"(workflow={instance.workflow_id})" - ) - - result = await self._executor.run_durable( - instance.instance_id, - resume_from_checkpoint=True, - ) - - if result.completed: - await self._instance_manager.mark_completed( - instance.instance_id, - outputs=result.data, - ) - logger.info(f"Instance {instance.instance_id} completed") - else: - await self._instance_manager.mark_failed( - instance.instance_id, - error=result.error or "unknown_error", - ) - logger.warning( - f"Instance {instance.instance_id} failed: {result.error}" - ) - except Exception as e: - logger.exception(f"Error processing instance {instance.instance_id}") - await self._instance_manager.mark_failed( - instance.instance_id, - error=str(e), - ) - finally: - # Release lock - await self._instance_manager.release_lock( - instance.instance_id, - self._worker_id, - ) - - async def _heartbeat_loop(self): - """Periodically extend locks on active instances.""" - while self._running: - await asyncio.sleep(self._heartbeat_interval) - for instance_id in list(self._current_instances.keys()): - try: - await self._instance_manager.extend_lock( - instance_id, - self._worker_id, - extend_seconds=int(self._heartbeat_interval * 2), - ) - except Exception as e: - logger.warning( - f"Failed to extend lock for {instance_id}: {e}" - ) -``` - -### 8.2 Worker Deployment - -**Option A: In-process (development/single-server)** -```python -# In server startup -worker = DurableWorkflowWorker( - worker_id=f"server_{socket.gethostname()}_{os.getpid()}", - instance_manager=instance_manager, - executor=executor, -) -asyncio.create_task(worker.start()) -``` - -**Option B: Separate process (production)** -```python -# scripts/durable_workflow_worker.py -if __name__ == "__main__": - worker = DurableWorkflowWorker( - worker_id=f"worker_{uuid.uuid4().hex[:8]}", - instance_manager=instance_manager, - executor=executor, - ) - asyncio.run(worker.start()) -``` - ---- - -## 9. API Design - -### 9.1 REST Endpoints - -```yaml -# Create and submit workflow instance -POST /api/workflows/instances -Request: - workflow_id: string - inputs: object - schedule?: { type: "once" | "interval" | "cron", ... } -Response: - instance_id: string - status: "pending" - -# Get instance status -GET /api/workflows/instances/{instance_id} -Response: - instance_id: string - workflow_id: string - status: string - current_state: string - progress: { step: number, total: number } | null - created_at: string - started_at: string | null - completed_at: string | null - error: string | null - -# List user's instances -GET /api/workflows/instances?status=running&limit=20 -Response: - instances: [...] - total: number - -# Cancel instance -POST /api/workflows/instances/{instance_id}/cancel -Response: - success: boolean - -# Get instance result (for completed) -GET /api/workflows/instances/{instance_id}/result -Response: - outputs: object - execution_trace_id: string - -# Create schedule -POST /api/workflows/schedules -Request: - workflow_id: string - schedule_type: "once" | "interval" | "cron" - run_at?: string # ISO date for "once" - interval_seconds?: number - cron_expression?: string - default_inputs: object -Response: - schedule_id: string - -# List schedules -GET /api/workflows/schedules - -# Update schedule (enable/disable) -PATCH /api/workflows/schedules/{schedule_id} -Request: - enabled?: boolean - default_inputs?: object - -# Delete schedule -DELETE /api/workflows/schedules/{schedule_id} -``` - -### 9.2 MCP Tool Integration - -```python -# In catalogue.py -DURABLE_WORKFLOW_TOOLS = [ - MCPToolSpec( - name="submit_durable_workflow", - description="Submit a workflow for background execution", - input_schema=Schema( - required={"workflow_id": str}, - optional={ - "inputs": dict, - "schedule_type": str, - "schedule_config": dict, - }, - ), - handler=_submit_durable_workflow, - ), - MCPToolSpec( - name="get_workflow_status", - description="Get the status of a running workflow instance", - input_schema=Schema(required={"instance_id": str}), - handler=_get_workflow_status, - ), - MCPToolSpec( - name="list_my_workflows", - description="List workflow instances for the current user", - input_schema=Schema( - optional={"status": str, "limit": int}, - ), - handler=_list_my_workflows, - ), -] -``` - ---- - -## 10. Vontology Integration - -### 10.1 New Concept Types - -``` -#V#durable_workflow_instance (type) - is_a_type_of: #V#workflow_execution - description: "A persistently tracked workflow execution that can be checkpointed and resumed" - -#V#workflow_schedule (type) - is_a_type_of: #V#scheduled_task - description: "A recurring or one-time trigger for workflow execution" -``` - -### 10.2 Workflow Definition Predicates - -Add to existing `#V#llm_workflow`: - -``` -#V#supports_durable_execution (predicate) - description: "Indicates this workflow can be run durably with checkpointing" - domain: #V#llm_workflow - range: boolean - -#V#default_schedule (predicate) - description: "Default schedule configuration for this workflow" - domain: #V#llm_workflow - range: #V#workflow_schedule -``` - -### 10.3 Example: RAG Sync Workflow Definition - -``` -#V#rag_text_relation_sync_workflow (individual) - is_an_instance_of: #V#llm_workflow - hasName: "RAG Text Relation Sync" - hasDescription: "Synchronises Vontology text relations to RAG store for semantic search" - supports_durable_execution: true - hasInitialStep: #V#rag_sync_step_collect - hasStep: [#V#rag_sync_step_collect, #V#rag_sync_step_batch, #V#rag_sync_step_upsert, #V#rag_sync_step_complete] - -#V#rag_sync_step_collect (individual) - is_an_instance_of: #V#workflow_step - hasName: "Collect text relations" - invokesAction: #V#collect_text_relations_action - nextStep: #V#rag_sync_step_batch - -#V#rag_sync_step_batch (individual) - is_an_instance_of: #V#workflow_step - hasName: "Batch for upsert" - invokesAction: #V#batch_documents_action - nextStep: #V#rag_sync_step_upsert - -#V#rag_sync_step_upsert (individual) - is_an_instance_of: #V#workflow_step - hasName: "Upsert to RAG" - invokesAction: #V#rag_upsert_batch_action - hasPrecondition: #V#has_more_batches_condition - onTrueNextStep: #V#rag_sync_step_batch # Loop back - onFalseNextStep: #V#rag_sync_step_complete - -#V#rag_sync_step_complete (individual) - is_an_instance_of: #V#workflow_step - hasName: "Complete" - terminal: true -``` - ---- - -## 11. Migration Path - -### 11.1 Phase 1: Infrastructure (Week 1) - -1. Create `workflow_instances` collection with indexes -2. Implement `WorkflowInstanceManager` (create, checkpoint, status) -3. Add basic REST endpoints for instance management -4. Write unit tests - -### 11.2 Phase 2: Durable Execution (Week 2) - -1. Implement `DurableWorkflowExecutor` with checkpointing -2. Create `DurableWorkflowWorker` with polling loop -3. Add heartbeat and lock management -4. Integration tests for resume scenarios - -### 11.3 Phase 3: RAG Sync Migration (Week 2-3) - -1. Define `#V#rag_text_relation_sync_workflow` in Vontology -2. Create action implementations for RAG sync steps -3. Migrate `rag_sync_text_relations` to submit durable workflow -4. Deprecate synchronous path (with feature flag) - -### 11.4 Phase 4: Scheduling (Week 3) - -1. Create `workflow_schedules` collection -2. Implement `WorkflowScheduler` service -3. Add schedule management REST endpoints -4. Test cron expressions and interval triggers - -### 11.5 Phase 5: Observability (Week 4) - -1. Add progress tracking to instance model -2. Implement SSE stream for live status updates -3. Create simple workflow status UI component -4. Documentation and user guide - ---- - -## 12. Implementation Plan - -### 12.1 New Files - -``` -src/backend/workflows/ -├── durable/ -│ ├── __init__.py -│ ├── instance_manager.py # WorkflowInstanceManager -│ ├── durable_executor.py # DurableWorkflowExecutor -│ ├── worker.py # DurableWorkflowWorker -│ ├── scheduler.py # WorkflowScheduler -│ └── models.py # WorkflowInstance, WorkflowSchedule -├── definitions/ -│ ├── rag_sync_workflow.py # RAG sync workflow definition -│ └── ... -``` - -### 12.2 Modified Files - -- `src/backend/server/routes/workflows_routes.py` - Add instance/schedule endpoints -- `src/backend/integrations/internal_mcp/catalogue.py` - Add MCP tools -- `src/backend/services/rag_text_relation_sync_service.py` - Use durable workflow - -### 12.3 Jira Tasks - -| Task | Summary | Estimate | -|------|---------|----------| -| JVNAUTOSCI-XXXX | Durable workflow infrastructure (collections, manager) | 2d | -| JVNAUTOSCI-XXXX | DurableWorkflowExecutor with checkpointing | 2d | -| JVNAUTOSCI-XXXX | DurableWorkflowWorker with lock management | 2d | -| JVNAUTOSCI-XXXX | Migrate RAG sync to durable workflow | 3d | -| JVNAUTOSCI-XXXX | Workflow scheduling system | 2d | -| JVNAUTOSCI-XXXX | REST API and MCP tools | 1d | -| JVNAUTOSCI-XXXX | Documentation and testing | 2d | - ---- - -## 13. Testing Strategy - -### 13.1 Unit Tests - -```python -# tests/backend/workflows/durable/test_instance_manager.py -class TestWorkflowInstanceManager: - async def test_create_instance_pending(self): - """New instance starts in pending status.""" - - async def test_checkpoint_updates_state(self): - """Checkpoint persists current state and data.""" - - async def test_find_resumable_claims_atomically(self): - """Concurrent workers don't claim same instance.""" - - async def test_stale_lock_allows_reclaim(self): - """Instance with expired lock can be reclaimed.""" -``` - -### 13.2 Integration Tests - -```python -# tests/backend/workflows/durable/test_resume_scenarios.py -class TestResumeScenarios: - async def test_resume_after_checkpoint(self): - """Workflow resumes from checkpointed state.""" - - async def test_resume_after_worker_crash(self): - """New worker picks up orphaned instance.""" - - async def test_cancel_running_workflow(self): - """Cancellation stops workflow gracefully.""" -``` - -### 13.3 Load Tests - -- 100 concurrent workflow submissions -- Worker restart during execution -- Schedule with 1-second interval - ---- - -## 14. Operational Concerns - -### 14.1 Monitoring - -**Metrics to expose**: -- `workflow_instances_pending` - Gauge: pending instance count -- `workflow_instances_running` - Gauge: running instance count -- `workflow_instance_duration_seconds` - Histogram: execution time -- `workflow_checkpoint_count` - Counter: checkpoints written -- `workflow_resume_count` - Counter: workflows resumed - -**Live status stream**: -- `GET /api/workflows/instances/stream` (SSE) -- Filters: `user_id`, `org_id`, `namespace`, `workflow_id`, `instance_id`, `status` (comma-separated) -- Payload includes progress fields (`progress.current`, `progress.total`, `progress.message`) - -### 14.2 Alerts - -- Pending instances > 100 for > 5 minutes -- Failed instances > 10 in 1 hour -- Worker heartbeat missed for > 2 minutes - -### 14.3 Troubleshooting - -**Stuck workflows**: -```javascript -// Find instances that have been running too long -db.workflow_instances.find({ - status: "running", - started_at: { $lt: new Date(Date.now() - 3600000) } // 1 hour -}) -``` - -**Orphaned locks**: -```javascript -// Release all expired locks -db.workflow_instances.updateMany( - { status: "running", lock_expires_at: { $lt: new Date() } }, - { $set: { locked_by: null, lock_expires_at: null } } -) -``` - ---- - -## Revision History - -| Version | Date | Changes | Author | -|---------|------------|----------------------------------|------------------| -| 1.0 | 2026-02-03 | Initial design document | Von AI Assistant | - ---- - -**End of Document** diff --git a/docs/engineering/historical_agent_guidance_notes.md b/docs/engineering/historical_agent_guidance_notes.md deleted file mode 100644 index d1bb572b0..000000000 --- a/docs/engineering/historical_agent_guidance_notes.md +++ /dev/null @@ -1,434 +0,0 @@ -# Historical Agent Guidance Notes - -**Status**: Historical archive, not authoritative -**Source**: Former `Backup_AGENTS.md`, preserved on 2026-04-04 - -This document preserves older agent guidance as historical context and as a -record of practical lessons that previously accumulated in `AGENTS.md`. - -It may contain superseded, duplicated, over-detailed, or stale guidance. -Prefer the current authoritative documents: - -- `AGENTS.md` -- `docs/engineering/operational_engineering_guide.md` -- the current engineering documents linked from those files - -Consult this note only when investigating prior operational failure modes, -trying to recover historical context for an implementation/debugging issue, or -mining it for lessons that should be promoted into current docs. - -Shell policy note: preserved entries below that say PowerShell is the default or -PowerShell-first are superseded. Current guidance is platform-specific: use -PowerShell on Windows, and use the native POSIX shell (`zsh`/`sh`) on macOS and -Linux unless invoking `.ps1` scripts or deliberately testing PowerShell. - -## Preserved source - -### AI Agent Guide (Trimmed) - -This document provides key instructions for AI agents working on the former private Von project. -All AI agents must read this file and `docs/engineering/security_considerations.md` before starting any work. - -## General advice: - Be reasonably encouraging. Be proactive - if something seems fairly safe, go ahead and do it. Make fairly frequent JIRA comment updates on tasks. We often work on neurosymbolic agent code, where much of system behaviour is determined by LLM prompts and context, and ontology supported representation and reasoning. In this setting, its extremely important not to write python code, json data, etc to implement processes that could be done by LLM or logical reasoning or some combinarion. Python code should be as general as possible, and should be regarded as a tool to be used by the other aspects of the code, where possible. Feel free to look across the repo, at JIRA and Von tasks, Vontology and the scientific literature when devising designs or implementations. - -If something goes wrong in a coding thread, consider whether updating documentation or Agent guidance files could be done to improve success in the future. After closing a task, think about how to could have been done better, and how it might affect pending tasks or the system design. - -## TL;DR for Agents -1. Use New Zealand English spelling always (behaviour, colour, organisation, realise). This can include careful use of Māori words. -2. PowerShell is the default shell. Do not emit Bash heredocs, `export`, `$(cmd)`, or `source venv/bin/activate` unless explicitly asked for Bash. -3. Never clobber `.env`. Only touch it if explicitly asked; never print its contents or secrets. -4. It's fine to auto-start the server; don't wait for explicit user instruction. (Still obey system/developer constraints if stricter.) -5. Use MCP tools (Vontology/Jira/Mongo) by default; for Jira, prefer Von's internal Jira MCP/proxy tools (`jira_*` via InternalMCPGateway) over Atlassian MCP OAuth. Explain if you must use another path. -6. Do not use direct DB access methods for Vontology data; use Vontology routes/services (API/MCP) instead. -7. **Vontology is THE source of truth** for all persistent knowledge and data. Exceptions (e.g., ephemeral caches, session state) must be rare and explicitly justified. -8. Jira issues must be assigned on creation (assignee = current user unless told otherwise). -8A. **Branch first**: substantial Jira work belongs on a task branch named for the Jira key, not on `main`. -9. Keep changes minimal, well-scoped, and add/update tests and docs where relevant. -10. Prefer small, composable functions; avoid monolithic helpers. -11. Always check VS Code Problems panel (or run `get_errors`) after edits and when errors are reported. If the Problems panel is not available, run `pyright` as a proxy. -12. **Pre-commit lint gate**: Before every `git commit`, run `get_errors` (or `pyright`) and fix all outstanding linting/type-checking errors. Do not commit with known unresolved diagnostics. If using `pyright`, include **every changed Python file** from `git diff --name-only` (do not hand-pick a partial subset). -13. Enable pre-commit guardrails: `git config core.hooksPath .githooks`. -14. **STOP**: Never run backend tests against `VON_DB_NAME=von_db`. Always use the test DB (`VON_DB_NAME=test_von_db`) or the `pytest:backend (test db)` task. -15. Requiring a user choice is almost always dispreferred; prefer LLM reasoning to achieve reliability and only ask the user when ambiguity cannot be resolved safely. -16. When in doubt, run tests or re-run tests without requiring user confirmation, but start with a **targeted impacted set** (touched modules + real call-path tests) and only expand to broader suites when risk or failures indicate. -16A. **Pytest execution policy**: use `pdm run python scripts/pytest_lanes.py recommend --git-diff origin/main` to plan targeted impacted validation, then run the direct targets and relevant deterministic lane(s). Treat targeted impacted testing as the default interactive validation level, and use `pdm run python scripts/pytest_lanes.py aggregate-plan` plus individual lane runs for broader coverage. Do not claim that a full pytest run was performed unless the relevant aggregate lanes were actually executed. See `docs/engineering/pytest_lane_strategy.md`. -16B. **Pytest timeout-avoidance policy**: default to small sequential pytest batches in interactive runs. Start with one file per invocation for slower or integration-heavy areas (for example orchestrator, workflow selector, RAG, DB-backed, or end-to-end call-path tests), and only combine files after the individual files have passed quickly. If a batch is slow, silent, or times out, immediately split by file, then by class/test or `-k` selection; do not keep retrying the same broad command. -17. In the case that multiple tests are failing, carefully consider the possibility that the tests are based on a design assumption that no longer holds. Tests are not definitional here, they are diagnostic, and should be changed (carefully) if they are not diagnostic for the current design. Do not allow tests to be a barrier to generality and good factoring. -18. Before closing a task, run the task-closure pass: re-read the task and its epic/subtask context, review nearby recent work, run targeted regression checks, and for substantial tasks complete the reflection pass. -18A. **Older Jira tasks require sceptical review**: task wording may be stale relative to the current codebase and evolving Von design. At task start and task closure, decide whether the task should be executed literally, reinterpreted as a general capability request, narrowed, superseded, or closed as obsolete, and update Jira/linked issues accordingly. -18B. **Literature-informed planning is expected when useful**: for research-sensitive, architecture-shaping, or long-horizon-agent tasks that could materially benefit from recent literature, do a short targeted literature review before implementation. Use it to refine or augment the Jira task/epic plan, prefer durable capability improvements over literal stale prescriptions, and record any material reinterpretation in Jira. -18C. **Mandatory modern-agentic design review for substantial implementation planning**: before finalising the plan for any substantial implementation task, review `docs/engineering/intro_to_modern_agentic_ai_for_coding_agents.md`. This is especially mandatory for ranking, recommendation, classification, matching, routing, planning, prompting, retrieval, memory/KB, ontology/representation, and evaluation work. -19. **Workflow-first / KB-authoritative**: express behaviour in VWL/Vontology when possible; do not encode workflow definitions, routing, or task-specific orchestration in Python when reusable workflow/runtime surfaces can carry the behaviour. The authoritative authored representation should live in Vontology/KB artefacts, not repo-side workflow files, whenever the behaviour can be represented there. If VWL is insufficient, add only the missing reusable execution/validation/telemetry/persistence surface and document the capability gap in Jira/Vontology. -19A. **Beware repo-declarative masquerading as KB-first**: common training/codebase priors strongly favour "files are the authored source; databases/KBs are runtime state". Replacing Python with repo-side JSON/YAML/DSL workflow files is **not** workflow-first convergence unless those files are only migration seeds, generated snapshots, test fixtures, or exports. Treat any production path that makes such files authoritative as source-authority drift. -19B. **No self-serving workflow-policy relaxation**: the same implementation effort may tighten or clarify workflow-authority guidance, but must not broaden what counts as compliant or weaken KB-authority requirements to fit its own design. Any genuine exception or relaxation requires a separate Jira issue, explicit human approval, and a concrete statement of the missing VWL/Vontology primitive. -19C. **Decision-policy authority extends beyond routing**: ranking, recommendation, matching, selection, classification, explanation, and planning policy count as authored behaviour, not just implementation detail. For open-ended natural-language content, do not ship production policy as lexical heuristics in Python when embedding-based, LLM-based, or workflow/KB-authoritative approaches are available. Treat hard-coded tokenisers, stopword lists, lexical-overlap heuristics, fixed score weights, and threshold tables as a design tripwire, not a default implementation pattern. -19D. **Heuristic fallbacks must be explicitly temporary and non-authoritative**: if a heuristic fallback is temporarily necessary, mark it as temporary and non-authoritative in code and Jira, link a removal or replacement task, state the missing reusable primitive or authority surface, and do not allow the heuristic path to become the primary production decision policy by accident. -20. **DRY first**: Create central helpers FIRST, then replace all usages. See "DRY refactoring discipline" in Workflow section. -21. **Comment for evolution**: Add comments that guide future modifications. See "Self-Documenting, Evolvable Code" section. -22. **Proactive hygiene**: Periodically review touched files and their neighbours for inconsistency, duplication, and drift. Fix proactively. -23. In docs/examples for secret env vars, use explicit placeholders like `` and avoid token-like sample strings that can trigger secret scanners. -24. If you are reasonably confident task implementation is complete, proactively run the merge-and-close flow unless the user explicitly asks to hold. -25. **Definition of fully complete Jira task**: merged to `main`, Jira and linked issues updated, required Vontology workflow/state changes materialised, and for substantial tasks the reflection pass has produced any justified `AGENTS.md` or Jira guidance update. -25A. **End-to-end Jira closure evidence must be direct**: when a task claims user-visible or end-to-end acceptance, do not mark it fixed/finished/Done on nearby commits, related-task work, or indirect unit tests alone; rerun the exact or nearest-real acceptance path and record that evidence in Jira before closure. -26. **Fail closed when Vontology is unavailable for Vontology-governed behaviour**: do not silently fall back to in-code prompts, stale context reuse, or heuristic hacks. If required Vontology prompts/workflow data cannot be resolved, no-op that transformation and emit a clear user-visible and telemetry-visible reason. -27. **Minimal-imposition principle**: treat this as a strong default design and operating policy. Exhaust existing context/data/search first; ask humans only when necessary, and then only for concise, low-effort, high-value inputs they are likely to know without extra work. See `docs/engineering/minimal_imposition_design_principle.md` for the rationale and limits. This principle is close to policy, but it is not a mechanical override of explicit user instructions, security requirements, destructive-action safeguards, or stricter workflow/Vontology constraints. -27A. **Minimal-imposition mutation policy**: low-risk additive Vontology writes should default-allow when evidence-backed inputs make the intended additive action clear and the user has not explicitly denied it. -27B. **Canonical identifiers/URLs can be permission**: pasted canonical sources such as arXiv abstract URLs/IDs can constitute sufficient permission for closely related low-risk additive representation work; do not require redundant verbs like `download`, `store`, or `represent`. -27C. **Destructive actions require workflow confirmation**: deletes, removals, and other destructive mutations should branch to explicit workflow confirmation/escalation rather than being executed immediately. -27D. **Human in the loop is not the default doctrine**: outside genuinely safety-critical cases, do not interrupt users or route to human approval without evidence that it is necessary. -28. When the user asks whether something is "finished" or "complete", do not treat Jira status alone as the answer. Verify effective implementation state in both code and Vontology, then report whether it is unimplemented, partly implemented, or fully implemented (with concise evidence). -29. **`.env` is authoritative for credentials and service config**: any code that reads credential or service-critical environment variables (API tokens, secrets, proxy config) **must** register those keys in `_apply_dotenv_overrides()` (`src/workflows/von/main.py`). Never rely on inheriting credential env vars from the parent shell — VS Code, CI runners, and other host processes routinely override well-known keys (e.g. `GITHUB_TOKEN`) with their own values. When adding or modifying code that resolves an env var for auth or external service access: (a) add the key to the override set, (b) verify `.env` contains a value for it, and (c) add a diagnostic log at resolution time showing which key was used. -30. **Substantial-task reflection is mandatory**: before closing a substantial task, explicitly ask what reusable practices helped, what missteps occurred, and whether `AGENTS.md` or Jira epic/task comments should be updated. -31. **Promote stable lessons, not noise**: update `AGENTS.md` only with durable, future-useful guidance that strengthens or cleanly supersedes existing rules; otherwise use Jira epic/task comments. Prefer synthesis over growth. - -## Core AI-Focused Documents -- `docs/AINotes.md`: short-term memory and tactical log. -- `docs/concept_refactoring.md` (or current plan doc): project roadmap for major work. -- `docs/software_engineering.md`: conventions, debugging, and lessons learned. -- `docs/engineering/security_considerations.md`: required security context. -- `docs/engineering/intro_to_modern_agentic_ai_for_coding_agents.md`: required background review before substantial implementation planning, especially for behaviour/policy/representation-sensitive work. -- `docs/engineering/minimal_imposition_design_principle.md`: design rationale and policy framing for minimal-imposition defaults, trade-offs, and limits. -- `docs/engineering/von_workflow_language_manual.md`: primary reference for workflow-related tasks; use it to guide workflow design/changes and update it whenever workflow semantics, capabilities, or constraints change. -- `docs/engineering/atlassian_mcp_recovery_runbook.md`: canonical Atlassian MCP recovery and credential reset procedure. -- `docs/engineering/jira_components_taxonomy.md`: canonical candidate Jira Components taxonomy for Von issues. - -## Workflow and Quality -- Preserve raw user-authored text when it is rendered/transformed; store originals in dataset/raw attributes where applicable. -- Users may update this file; retain any user changes. -- Add a regression test for every state/format-loss bug (load -> edit -> save -> re-edit). -- **Test through the real call path**: When modifying MCP tool handlers, don't just test the handler function directly—also test through the gateway layer (`InternalMCPGateway.invoke()`). Early-return error paths may bypass output schema validation, and handlers may produce response shapes that don't match expected schemas. If a tool has an `output_schema`, verify error responses work end-to-end. -- Prefer lightweight telemetry where practical (timings, counters, and error summaries) to support UX and future introspection. -- **Telemetry correctness is mandatory**: diagnostic payloads, stage paths, counters, and progress summaries are operational interfaces, not optional polish. If they are internally inconsistent with the underlying event stream, treat that as a bug and fix it with the same urgency as a behavioural regression. -- **Trace sanitisation must preserve benign typed telemetry**: do not blanket-redact or string-coerce non-secret counters such as `prompt_tokens`, `completion_tokens`, `total_tokens`, durations, or similar numeric telemetry fields. Redact secret-bearing values, but keep safe metrics machine-readable so downstream analytics, dashboards, and prediction surfaces remain trustworthy. -- When fixing reliability issues, prefer **systemic, general fixes** over point fixes: update shared pipelines, validators, and policies so that behaviour remains stable across model changes and configuration drift. -- **Substantial-task reflection pass (mandatory)**: after implementation and validation, but before merge/closure, review the work explicitly for generality and future leverage. - 1. Ask: "To the extent that I did a good job of producing high-quality code that improved the overall design and function of Von, what practices would be helpful in the future, and what missteps did I make; how should I update `AGENTS.md`, or Jira epic comments, to do better in future?" - 2. Identify which parts of the solution were truly reusable improvements (shared runtime surfaces, schemas, validators, telemetry, VWL/Vontology metadata, canonical helpers) and which parts risked case-specific drift. - 3. Prefer updating shared abstractions, schemas, and metadata surfaces over adding named special cases. If a proper noun from the originating task appears in logic, treat that as a design smell unless there is a strong reason. - 4. Promote stable, repo-wide lessons into `AGENTS.md`; put narrower or still-evolving lessons into the relevant Jira epic/task comments. - 5. Before editing `AGENTS.md`, ask whether the lesson is genuinely durable, likely to help on future tasks, and strong enough to clarify or cleanly supersede existing guidance. - 6. Prefer pruning, clarifying, reorganising, or synthesising nearby guidance over simply appending new text; when adjacent rules teach the same habit, replace them with a shorter, sharper rule when possible. - 7. If the reflection finds a recurring process/design gap, create or update a Jira issue instead of leaving the lesson as an unenforced observation. - 8. Treat this reflection as part of acceptance, not optional retrospective polish. -- For question-generation or elicitation flows, apply the minimal-imposition principle explicitly in prompts/fallbacks: avoid broad requests, and prefer one focused ask only when machine-side retrieval cannot close the gap. -- For write-policy work, treat minimal imposition as a strong risk-class policy: additive low-risk Vontology mutations should usually proceed, while destructive mutations should route to explicit confirmation workflows. -- **DRY refactoring discipline** (CRITICAL — WET code is unacceptable): - 1. **Search first, always**: Before writing ANY function that might exist elsewhere, run `grep_search` or `semantic_search`. This is not optional. - 2. **3-strike rule**: If you're about to write similar code for the 3rd time, STOP. Do not proceed. Create a central helper first. - 3. **Scope the problem first**: Before fixing ANY bug that appears in multiple places, grep/search to find ALL instances. Understand the full scope before touching code. - 4. **Create the helper FIRST**: Write a single, well-named helper function in a central location (e.g., `utils/`, `helpers/`, `domUtils.js`, `apiService.js`). - 5. **Then replace ALL usages**: In a single operation (multi_replace_string_in_file), replace every occurrence with calls to the helper. - 6. **Never fix instances one-by-one**: Piecemeal inline fixes create maintenance burden, review noise, and guarantee future bugs. - 7. **Central locations by domain**: - - Frontend JS utilities: `static/js/utils/` or `domUtils.js` - - API/fetch helpers: `apiService.js` - - Backend Python utilities: `src/backend/utils/` or domain-specific service modules - - Vontology operations: use existing service classes, never raw DB calls - 8. **If you catch yourself copy-pasting**: You are doing it wrong. Stop and refactor. -- Changes must be **modification-tolerant**: switching underlying models should not silently remove or change capabilities; differences must be explicit and detectable (telemetry, validation, or policy). -- **Vontology-first prompt pattern (strict)**: any new or modified LLM prompt should be stored as Vontology text relations (`hasContent`/`hasDescription`, primary `en-NZ`). Do not add silent code-prompt fallbacks for Vontology-governed features. If prompt resolution fails, fail closed for that feature and emit explicit diagnostics (including prompt IDs/reasons) so the failure is obvious and actionable. -- **Prompt/workflow-first diagnosis**: when behaviour looks wrong on an LLM- or workflow-governed path, inspect the authoritative Vontology prompts, workflow definitions, routing metadata, and rendered selector/dispatcher inputs before doing code-first diagnosis. Use those artefacts to drive the subsequent code analysis; do not begin from code-only assumptions when prompt/workflow authority may already explain the outcome. -- **No hack-around reliability policy**: when an ontology/workflow/prompt dependency fails, fix the dependency path (tools, retrieval, workflow config, validation) rather than patching symptoms with ad-hoc heuristics. Temporary mitigations must be explicit, bounded, and tracked in Jira with a removal plan. -- For high-risk state changes (auth/org handling, DB writes, Vontology mutations), use a single authoritative pathway and reuse it consistently. -- Strong rule: treat canonical predicate concepts (e.g. #V#is_a_type_of) as the authoritative ontology relations. Do not introduce or rely on structural relationship fields when predicate concepts exist; kind/classification should be derived from canonical predicate usage. -- **Predicate concepts must be instances of `#V#predicate` (or a specialisation such as `#V#binary_predicate`) and must not keep `is_a_type_of` links that would force kind=`type`.** This ensures `is_predicate()` and the computed kind behave correctly. -- Do a quick factoring review whenever you touch write paths: search for existing canonical helpers/endpoints (especially for deletes/merges) and route through them rather than adding parallel pathways. -- Treat ontology/predicate investigation as a normal first step for Vontology work: resolve candidate concepts by name (including spelling variants like programme/program), search for existing predicates/types before inventing new ones, and record the findings (and chosen canonical IDs) in the related Jira issue. -- When code needs to manipulate real-world or conceptual entities (tasks, organisations, ideas, documents, relations, predicates, claims, assertions, rules, workflows, etc.), **always start by checking whether they already exist in Vontology**. Reuse/connect to existing representations; update them if needed; otherwise create new concepts rigorously. -- **Vontology representation guidance (from code behaviour):** - - **Names:** Display names are resolved from `names[]` (NL then ABBR, preferred language first). Create/update names via `hasName` text relations (primary NL in `en-NZ`), and include CODE names for the `concept_id` and GUID where applicable. Do not rely on legacy top-level `name` fields. - - **Text preservation:** Keep text relation values exactly as-is unless a user explicitly asks to rename or normalise them. Do not auto-replace underscores with spaces (e.g., `has_url` -> `has url`). - - **Descriptions:** Canonical descriptions are stored in `hasDescription` text relations. Avoid writing legacy `description` fields or `concept_data.preserved_fields` directly; use the concept/text relation services. - - **Types vs individuals:** Types use `relationships.is_a_type_of` (parents). Individuals use `relationships.is_an_instance_of` (types). Computed kind is derived from these relationships, so avoid mixing them on the same concept. - - **Type guidance predicates:** For types, prefer `#V#salient_binary_predicate_for_type` (or `salient_predicate_scopes.type_level`) to drive salient predicate prompts. These lists are consumed by salient predicate aggregation and relation elicitation. It is almost never appropriate for a type to be #V#is_a_type_of #V#thing (similarly, individuals should not be instances of thing). Search the ontology for suitable types to attach concepts to. Use the most restrictive applicable and appropriate supertypes. - - **Suggested relations:** Use the meta-relations service (`suggested_relations_for_type`) for type-level suggestions when possible; legacy `relationships.suggested_relations_for_type` is read as a fallback. -- **Vontology-first checklist (mandatory for ontology-related work):** - 1. Resolve candidate concepts by name (MCP search/resolve). - 2. Check for existing predicate/type concepts before inventing anything. - 3. If a list of concepts is needed, prefer a Vontology type and query its instances. - 4. Record the chosen canonical IDs in the Jira issue. -- **Workflow-first behaviour checklist (mandatory for behaviour/orchestration changes):** - 0. Before coding, name the intended authoritative KB artefacts in Jira or task notes (workflow concepts, template/profile concepts, prompt concepts, routing metadata, text relations, or other Vontology-native structures) and distinguish them from any supporting code-only surfaces. - 0A. Before coding, decide whether the task is also authoring decision policy (for example ranking, recommendation, matching, classification, planning, explanation, or retrieval strategy). If so, treat that policy as authoritative behaviour that should live in Vontology/workflow/prompt/KB artefacts where feasible, and keep Python focused on reusable support surfaces. - 1. Inspect existing workflow definitions before coding (`workflow_list_definitions`). - 2. Prefer creating/updating workflow instances and definitions in Vontology over adding task-specific Python orchestration (`workflow_create_instance`, `workflow_get_instance`, `workflow_list_instances`). - 2A. Treat any temptation to encode workflow branching, fallback sequencing, or route selection in Python as a design error unless VWL lacks the required primitive. In that case, add only the missing reusable primitive/control surface in code, document the capability gap in Jira, keep the workflow itself in Vontology, and attach the detailed capability-gap note to the workflow or affected step in Vontology. - 2B. If a workflow could be implemented by a genuine general extension of VWL rather than a context-specific patch, make that VWL/runtime extension first, then implement the workflow fully in VWL. Do not leave the workflow logic hidden in Python once the behaviour is generalisable as language/runtime capability. - 3. Prefer durable bindings/schedules for repeat behaviour (`workflow_bind_event`, `workflow_create_schedule`, `workflow_set_schedule_enabled`, `workflow_trigger_schedule`). - 4. Use operational controls instead of ad-hoc runtime flags (`workflow_cancel_instance`, `workflow_retry_instance`, `workflow_delete_schedule`). - 5. Treat workflow creation/editing in Vontology as normal implementation work. Do it during the task whenever the behaviour can be expressed there; do not stop at design prose if the workflow can be materialised. - 6. If Jira wording conflicts with this policy, treat the workflow-first policy as authoritative: reinterpret the Jira task, leave a note in Jira, and proceed with workflow definitions plus supporting tool work. - 7. If workflow tools still cannot express the requirement, explicitly document the constraint and create/link a Jira capability-gap issue before introducing specialised code. - 8. Repo-side workflow/template/prompt files are non-authoritative by default. They are acceptable only as migration seeds, generated snapshots, test fixtures, or exports. If deleting them would remove the authoritative behaviour or logic, the design is not yet KB-authoritative. -- **Workflow-authority readiness/closure checklist (mandatory for workflow-related tasks):** - - Before implementation, record the authoritative KB artefacts that will hold the logic. - - Before closure, distinguish the supporting code surfaces from the authoritative KB artefacts. - - Before closure, verify that any authoritative decision policy lives in Vontology/workflow/prompt/KB artefacts or materialised KB assertions, not in Python helper code or lexical heuristic scaffolding. - - Before closure, ask whether any docs/manuals became more permissive; if yes, stop and spin that into a separate approved Jira change or exception rather than normalising it inside the implementation task. - - Before closure, ask whether repo-side snapshots or seed files could be deleted without losing authority; if not, the task is not yet KB-authoritative. - - If repo-side files remain, state why they are non-authoritative (migration seed, generated snapshot, test fixture, or export) and link the task that removes or supersedes them if they are temporary. -- For fairly complex capability improvements, run a background search for related Jira issues, Confluence design docs, and existing Vontology concepts before proposing a plan. -- **No hard-coded ontology lists**: do not add fixed lists of predicate/type IDs or names in code. If you believe a hard-coded list is unavoidable, you must: - - explain why Vontology lookup is not viable, - - add a Jira note documenting the exception, and - - include a removal/cleanup plan. -- When uncertain, ask succinctly; do not guess or fabricate behaviour. -- **Task lifecycle discipline**: - - Before implementation, use or create an appropriate task branch and transition the Jira task to `In Progress`. - - Do not do substantial Jira implementation work on `main`; if you notice work has started without the task branch, stop and correct that first. - - If resuming from already-started or handed-off work, re-check branch/Jira/reflection/merge obligations before continuing; do not assume a previous turn or agent already handled them. - - At task start, review the task age, epic context, and related older issues sceptically; if the wording is stale, reinterpret the task toward the current Von design and record that in Jira instead of following outdated prescriptions literally. - - For research-sensitive or architecture-shaping tasks, consider whether a short targeted literature review would materially improve the plan; if so, do it before coding and update the Jira task/epic or subtasks when the literature changes the intended approach. - - During substantial work, post concise Jira progress comments at meaningful milestones. - - Before `Done`, complete the linked-issue review and ensure implementation branch work has been committed, pushed, and merged. - - Treat `git push` as incomplete until you verify the remote ref. Before saying a task is merged or transitioning it to `Done`, confirm `origin/main` contains the task commit(s) or the merge commit that contains them. Do not rely on push output alone; concurrent upstream movement can leave local `main` and `origin/main` diverged. - - Before saying a task is finished or transitioning it to `Done`, verify the acceptance path from the branch that will be merged. For end-to-end/user-visible tasks, direct call-path evidence is required; related-task commits and passing local unit tests are supporting evidence only. - - Before `Done`, review related older tasks again and update, comment, or transition any whose original intent has been absorbed, superseded, narrowed, or made obsolete by the completed work. - - After merge, transition the task to `Done` and post a progress update on the parent issue. - - If implementation appears complete and there is no explicit request to hold, execute the merge-and-close flow proactively. -- When commenting on a Jira task, say that the comment is generated by you (naming the agent) on behalf of the user. -- When asked to "merge", default to: ensure changes are committed, fast-forward merge into `main`, push `main`, verify `origin/main` now contains the intended commit(s), then delete the local and remote feature branch; if deletion needs a force flag, ask for confirmation first. - -## Tooling and Automation -- Activate required external tool categories (Jira, Vontology, MongoDB, GitHub) without asking, when clearly needed. -- Treat tool categories as opt-in per session: activate before first use, and occasionally check whether a matching tool deactivation call exists (to disable categories when no longer needed). -- **Vontology workflow tools are the default control surface for behaviour**: - - Discovery/introspection: `workflow_list_definitions`, `workflow_list_event_bindings`, `workflow_list_instances`, `workflow_mcp_health_check`. - - Execute behaviour: `workflow_create_instance`, then monitor via `workflow_get_instance`. - - Event-driven behaviour: `workflow_bind_event` (avoid hard-coded dispatch tables where workflow binding can be used). - - Scheduled behaviour: `workflow_create_schedule`, `workflow_list_schedules`, `workflow_get_schedule`, `workflow_trigger_schedule`, `workflow_set_schedule_enabled`, `workflow_delete_schedule`. - - Runtime safety/recovery: `workflow_cancel_instance`, `workflow_retry_instance`. - - Prefer these pathways over bespoke orchestration code whenever behaviour can be represented as workflow definitions + inputs. -- **Jira default access path**: it's OK to use the Atlassian MCP tools, but if they fail, switch immediately to using Von's internal Jira proxy/tools first for the rest of the session (`jira_get_auth_config`, `jira_get_myself`, `jira_search`, `jira_get_issue`, write helpers in `src/backend/integrations/internal_mcp/catalogue.py`). This path uses `ATLASSIAN_BASE_URL` + `ATLASSIAN_EMAIL` + `ATLASSIAN_API_TOKEN` and is independent of Atlassian MCP OAuth reliability. -- **If Jira MCP transport is unavailable in-session**: when the `jira_*` tool surface itself is unreachable (for example MCP transport closed) but the local repo/runtime is available, use Von's canonical repo-local Jira proxy implementation rather than inventing an ad-hoc workaround: load `.env` for the process, then call the supported helpers in `src/backend/mcp_server/mcp_server.py` / `src/backend/integrations/internal_mcp/catalogue.py` so auth, ADF conversion, and Jira request shaping stay on the standard path. Do not fall back to handwritten REST scripts. -- Use Atlassian MCP OAuth tools only when explicitly needed for capabilities not available in Von's internal Jira tools. -- When Jira friction is discovered, prefer improving Von's internal Jira path (tooling, diagnostics, schemas, guardrails) and document the issue/capability gap in code/docs/Jira so reliability improves over time. -- Where Vontology search, analysis or manipulation is impeded by the current vontology MCP tools, suggest code improvements to those tools that will facilitate high quality ontological engineering in future. -- Failures in Von's own MCP tools are bugs: report them as Jira bug work items with reproducible steps, observed error payloads, and related-issue links. -- If a tool category is not enabled, request enabling it by exact name. -- If Vontology or Vonrag MCP tools are not exposed in this session, use the stdio proxy scripts (`scripts/query_vontology_mcp.py`, `scripts/query_vonrag_mcp.py`) and check cached tool lists in `data/mcp_tool_cache/`. -- After implementing a fix and tests pass, post a Jira summary comment and transition the issue to the correct state. -- Do not emit JSON tool-call payloads as text; invoke tools directly. - -### Vontology/Vonrag MCP Field Notes (2026-02) -- When searching Jira for possible duplicates, prefer `jira_search` with explicit `fields` (at minimum: summary, status, assignee, issuetype, parent). Default search responses can be id-only and create avoidable follow-up calls. -- Treat `jira_get_issue` 404 responses as potentially permission-related, not just non-existence. Cross-check with `jira_search` results before deciding to create a new issue. -- For canonical singleton text predicates (especially workflow `hasDescription` in one language), use `upsert_singleton_text_relation` with `garbage_collect=true` rather than repeated `upsert_text_relation`. -- After creating or updating workflow concepts, run `concept_exists` and `get_text_relations_summary` as a quick integrity check. -- Use Vonrag `search_knowledge_base` for semantic recall, but do not rely on it for deterministic structured failure triage. When turn-level structured analysis is needed, propose/add dedicated tools and document the gap in Jira. - -## Atlassian MCP Reliability (Pause + Checkpoint) -If Atlassian MCP is flaky (timeouts, empty responses, 401/403/5xx): -1. Stop further Jira writes/edits beyond bounded retries. -2. Checkpoint: what issue(s), what succeeded, what remains. -3. Ask the user to restart MCP using the steps below. -4. Resume from the checkpoint after a minimal health check. -5. Prefer lightweight recovery first (`Developer: Reload Window` + Atlassian MCP restart). In recent incidents this has often recovered auth without credential reset. - -If restart/reload does not recover Atlassian MCP (especially recurring `invalid_token` / `Canceled` token fetch loops), follow: - -- `docs/engineering/atlassian_mcp_recovery_runbook.md` -- This runbook includes the no-sign-out recovery path (safe backup + targeted VS Code state DB key reset). -- Prefer the helper script first: `scripts/powershell/reset_atlassian_mcp_auth.ps1` - -Restart steps (VS Code): -- Command Palette -> `MCP: Browse MCP Servers` -> Atlassian -> Restart. -- If Restart is not available: `Extensions: Focus on MCP Servers - Installed View` -> Atlassian -> Restart/Stop/Start. -- If still failing: `Developer: Reload Window`. -- If it continues to fail: `Atlassian: Open Settings` -> sign out, then sign in again. - -Do not "hack around" MCP failures with ad-hoc scripts or direct REST calls. Fix the MCP session instead. - -## Jira Essentials -- Tasks can have Subtasks (use `issueTypeName="Subtask"` + `parent="JVNAUTOSCI-XXX"`). -- Tasks can set an Epic as `parent` via edit tooling (`{"parent": {"key": "JVNAUTOSCI-123"}}`). -- If an issue is created without an assignee, fix it via the Jira edit tool rather than duplicating. -- **Deduplicate before create**: when a user asks to create a Jira task, first search for an existing issue with the same intended effect. Prefer updating/expanding the existing issue (and linking/commenting for traceability) instead of creating a new one. Create a new issue only if no suitable existing issue exists, or if the user explicitly asks for a separate task. -- When creating or substantially rewriting a Jira task, include a concise implementation-design note by default (scope, key behaviour, edge cases, acceptance checks). For workflow-related tasks, also name the intended authoritative KB artefacts and distinguish them from any supporting code surfaces. For UI/UX tasks, include placement/layout, visibility rules, accessibility, motion, and responsive expectations. For substantial tasks, explicitly include a final acceptance pass requiring post-implementation reflection on reusable practices, missteps/case-specific temptations, whether `AGENTS.md` should be updated, whether any update is truly durable and future-useful, whether nearby guidance should be synthesised/pruned instead of merely extended, and whether the parent epic needs a process/design lesson comment. -- For full Jira Tasks (not sub-tasks), always identify the appropriate epic and attach them to it. In the very unlikely event that a suitable epic isn't available, plan what that epic would look like, and offer to create it. -- Add appropriate inter-task links (e.g. blocks/depends-on/relates) when the available MCP tooling supports it. -- **Link hygiene is mandatory**: - - When creating a new Jira task, scan for existing related tasks first and add links (`relates to` / `blocks` / `is blocked by`) immediately. - - Before starting implementation on an existing task, check whether relevant related tasks exist and add missing links before coding. -- **Completion linked-issue scan is mandatory for every task**: - - Run a linked-issue review before merge while full code context is still available (validate linked issue statuses, add/update comments, and apply justified transitions). - - Before marking a task Done, review all linked issues and identify any linked items still in `Backlog` / `To Do`. - - Treat older linked issues sceptically: if the current design or completed work has changed their practical purpose, update them as reinterpreted, superseded, narrowed, or obsolete rather than leaving stale prescriptions open. - - Add a concise status comment on each relevant linked item describing what changed, what remains, and whether it is now unblocked, superseded, or complete. - - Apply transitions when justified by the completion outcome (for example `To Do` -> `Done` for fully satisfied scope, or `To Do` -> `SUPERSEDED` when absorbed by another task). -- If a screenshot is provided to describe an issue or support issue creation, attach it to the Jira issue whenever possible using available MCP tooling rather than leaving it only in chat context. -- **Pasted screenshot handling (mandatory default path)**: - - Use `list_recent_screenshots` first with `match_clipboard=true` and `include_base64=true` to locate the correct local image and get a Jira-ready payload. - - If clipboard image data is unavailable, select the most likely recent screenshot candidate, attach it with `jira_add_attachment`, and state in the Jira comment that selection was based on recency/path heuristics. - - Prefer this flow over asking the user for a manual file path; ask only if no credible candidate is found. -- When setting Jira `Components`, use `docs/engineering/jira_components_taxonomy.md` as the default source of truth unless the user requests otherwise. -- Jira site URL: https://naoinstitute.atlassian.net/ -- If a cloudId is required, fetch it from https://naoinstitute.atlassian.net/_edge/tenant_info and include that URL when requesting it. - -## PowerShell-First Shell Rules -- Avoid Bash-only syntax: heredocs, `export`, `$(cmd)`, `source venv/bin/activate`. -- Use `$env:VAR = 'value'`, `$var = (cmd)`, and PowerShell here-strings. -- Prefer separate lines over `&&` unless failure short-circuit is required. -- Large multi-line Python: use a here-string variable and `pdm run python -c $code`, or add a script. - -## Command Hang Guardrails -- Treat simple read commands (`Get-Content -TotalCount`, `rg`, `git status`, `git branch`) as expected to complete quickly; if a run exceeds ~30s with no new output, treat it as potentially hung. -- Prefer bounded reads and output caps (`-TotalCount`, `Select-Object -First`, targeted `rg`) instead of broad scans. -- Avoid launching multiple long `Get-Content` calls in parallel; run file reads in smaller batches. -- Treat `pytest` as a timeout-sensitive command in agent sessions: prefer explicit timeouts and narrow batches, especially for quiet or historically slow suites, and shrink the batch immediately after any stalled run rather than rerunning the same wide selection. -- When diagnosing shell delay, run a quick sanity probe first: - - `pwsh -NoProfile -Command "Get-Content -TotalCount 5"` - - If fast manually but slow in agent tooling, assume tooling/transport stall and restart the VS Code window. -- Use explicit command timeouts for agent-run shell calls, and retry with a minimal command before escalating. - -## IDE Host Introspection (Codex/Copilot) -- **Introspect execution host first** where possible (Codex extension, Copilot agent/chat, CLI, or unknown) before applying host-specific assumptions. -- If host introspection is available, record the detected host in diagnostics/comments for debugging reproducibility. -- Use an allowlist for host markers (e.g., `CODEX_INTERNAL_ORIGINATOR_OVERRIDE`, `VSCODE_*`, process chain), and avoid dumping full environment variables. -- If host cannot be determined reliably, default to conservative behaviour: - - use PowerShell-first commands, - - bounded output, - - explicit timeouts, - - restart/retry guidance framed in host-neutral terms (e.g., `Developer: Reload Window`). -- Do not assume UI labels (e.g., `bash` output block labels) reflect the true underlying shell process. - -## UI Debugging (Missing/Invisible Elements) -- First check the DOM element exists; then check CSS visibility (display/visibility/opacity), size, positioning, z-index, and overflow. -- Absolute-positioned elements need a relative parent. Off-screen transforms and `overflow: hidden` are common culprits. -- If the element exists but looks wrong, capture `outerHTML` and key computed styles before changing JS. - -## Copilot Tool Selection Hygiene (when running in Copilot) -- Authoritative script: `scripts/set_vscode_copilot_selected_tools.ps1`. -- If you rely on a tool, ensure it stays enabled by updating that script. -- When not running in Copilot (for example, Codex extension), do not assume Copilot tool-selection state is active. - -## Reporting IDE Agent Issues -When encountering agent-host bugs or limitations, first include host identification in the report (Codex extension vs Copilot vs other). Then use the host-appropriate path: -- Copilot issues: https://github.com/microsoft/vscode-copilot-release/issues -- Codex extension issues: use the current OpenAI/Codex feedback channel configured for this workspace/org. - -Copilot issue template: - -``` -- Copilot Chat Extension Version: (run `code-insiders --list-extensions --show-versions | Select-String copilot`) -- VS Code Version: (run `code-insiders --version`) -- OS Version: Windows -- Feature (e.g. agent/edit/ask mode): -- Selected model (e.g. GPT 4.1, Claude Opus 4.5): -- Logs: (if applicable) - -Steps to Reproduce: -1. -2. - -Expected Behaviour: - -Actual Behaviour: - -Impact: - -Suggested Fix: -``` - -This keeps overhead low while providing actionable reports. - -## Research Prototype Engineering Philosophy -- Build research-appropriate quality: modular, extensible, and understandable. -- Avoid enterprise-scale over-engineering; keep abstractions just deep enough for near-term change. - -## Self-Documenting, Evolvable Code -Code comments are not just explanations—they are **guidance for future modification**. Use comments strategically to steer the codebase toward global consistency and safe evolution. - -### When to Add Guiding Comments -1. **Design rationale**: When a choice isn't obvious, explain *why* (not just *what*). Future agents and developers will otherwise repeat the same mistakes or undo intentional decisions. -2. **Cross-cutting conventions**: When a file or module establishes patterns that other code should follow, document those patterns prominently (module docstrings, section headers). Example: schema design guidelines in `catalogue.py`. -3. **Footgun warnings**: When code has non-obvious failure modes (e.g., strict validation rejecting LLM-hallucinated fields), add a comment explaining the risk and the mitigation pattern. -4. **Consistency requirements**: When a parameter, field, or pattern must be included for global consistency even if unused locally, document why. Example: `# Accept namespace for LLM consistency (ignored by handler)`. -5. **Evolution hooks**: When you anticipate future changes, leave breadcrumbs: `# TODO: when X is implemented, update Y` or `# Future: consider Z for better performance`. - -### Comment Style Guidelines -- **Be actionable**: "Include `namespace` in all user-facing tool schemas" is better than "namespace is important". -- **Reference issues**: Link to Jira tickets (e.g., `See JVNAUTOSCI-1044`) so readers can find full context. -- **Prefer module/class docstrings** for conventions that affect entire files; use inline comments for localised warnings. -- **Keep comments current**: When changing code, update or remove stale comments. Wrong comments are worse than no comments. - -### Examples of Good Guiding Comments -```python -# Module docstring establishing conventions: -""" -SCHEMA DESIGN GUIDELINES -======================== -1. Include `namespace` as optional on user-facing tools (LLMs generalise patterns). -2. Prefer allow_unknown=True for inputs; strict validation rejects hallucinated fields. -See JVNAUTOSCI-1044 for the failure mode this prevents. -""" - -# Inline comment explaining non-obvious inclusion: -optional={ - "namespace": (str, type(None)), # Accepted but ignored; LLM consistency -} - -# Warning comment for future maintainers: -# WARNING: Do not remove this fallback—external callers depend on the legacy format. -``` - -### Anti-Patterns -- **Commenting *what* without *why***: `# increment counter` adds nothing; `# increment to track retry attempts for rate-limit backoff` adds value. -- **Orphaned TODOs**: If you add a TODO, also add context (who should do it, when, why). -- **Defensive silence**: Not commenting tricky code because "it's obvious" guarantees future breakage. - -## Agent-Friendly Engineering Practices -These practices make codebases more navigable and consistent for AI agents, who lack tribal knowledge and rely on explicit signals. - -### Canonical Reference Implementations -When establishing a pattern, create ONE exemplary instance and comment it as the reference. Agents learn by example more reliably than by rule. -```python -# REFERENCE IMPLEMENTATION for tool input schemas. -# Copy this pattern for new tools. See JVNAUTOSCI-1044. -def _example_tool_input_schema() -> Schema: - return Schema( - required={"concept_id": str}, - optional={ - "namespace": (str, type(None)), # Accept for LLM consistency - }, - allow_unknown=True, # Tolerate hallucinated fields - ) -``` -Then elsewhere: `# See _example_tool_input_schema for the canonical pattern.` - -### Fail-Fast with Diagnostic Errors -Error messages must include **what was provided** and **what was expected**. Agents iterating on fixes need both. -```python -# Bad: -raise ValueError("Invalid field") - -# Good: -raise ValueError(f"Unexpected field '{field}'. Valid fields: {list(schema.keys())}") -``` - -### Schema-First Interfaces -Define types/schemas/contracts **before** implementing behaviour. Agents reason better about explicit contracts than emergent behaviour. -- Use `Schema` definitions for MCP tools -- Use TypedDict/dataclass for internal interfaces -- Document return shapes in docstrings when formal schemas aren't practical - -### Structured Telemetry and Introspection -Embed diagnostic payloads throughout the system (like the `aux_llm_calls` JSON). When something fails, the debugging story should be readable from structured output. -- Include `stage`, `type`, `error`, and `context` fields in diagnostic records -- Prefer structured dicts over free-form log strings -- Make introspection endpoints (like `chat_introspect`) available for debugging -- If route selection, workflow membership, stage mapping, or tool accounting disagree with each other, the telemetry is wrong; either reconcile the fields against one authoritative event stream or omit the derived field and emit an explicit reason. - -### Convention Over Configuration -Reduce the number of valid ways to accomplish a task. Fewer choices = fewer agent mistakes. -- Establish one canonical path for common operations (e.g., one way to add a relationship) -- When multiple approaches exist, deprecate the worse ones explicitly -- Document "the one right way" in module docstrings - -### Proactive Consistency Reviews -Periodically scan touched files and their neighbours for inconsistency, duplication, and drift—then fix proactively. -- When editing a file, skim related files for parallel patterns that should match -- After completing a task, do a quick grep for similar code that might need the same fix -- Treat documentation (like this file) as code: review it for internal consistency when editing -- Flag accumulated debt in Jira rather than ignoring it diff --git a/docs/engineering/intro_to_modern_agentic_ai_for_coding_agents.suggested.md b/docs/engineering/intro_to_modern_agentic_ai_for_coding_agents.suggested.md deleted file mode 100644 index 0bfaf64d7..000000000 --- a/docs/engineering/intro_to_modern_agentic_ai_for_coding_agents.suggested.md +++ /dev/null @@ -1,573 +0,0 @@ -# Superseded proposal: Von Agentic AI Architecture Constitution - -> **Document status: Superseded proposal; non-authoritative.** This file was -> never the guide selected by `AGENTS.md`. Do not treat its “normative”, “main -> architecture guide”, or “mandatory” wording as current authority. Use -> [`intro_to_modern_agentic_ai_for_coding_agents.md`](intro_to_modern_agentic_ai_for_coding_agents.md) -> together with [`AGENTS.md`](../../AGENTS.md). This copy is retained only for -> design provenance. - -- **Kind:** Unselected architecture proposal -- **Lifecycle:** Superseded -- **Authority:** Historical only -- **Original date:** 2026-04-04 - -## 1. Purpose - -This document was proposed as the main architecture guide for coding agents -working on Von. It was not selected as the canonical guide. - -It exists to prevent a recurring failure mode: treating modern agentic behaviour as if it were mainly ordinary application logic that should be hand-coded in Python, with prompts, workflows, knowledge representations, model portfolios, and learning loops treated as secondary decoration. - -For Von, that is often backwards. - -Von is being built as a deployable neuro-symbolic assistant system for teams, demonstrated first for research-team support. It must stay able to absorb newer models, fine-tunes, prompt optimisers, routing methods, memory architectures, and reinforcement-learning advances without repeatedly relocating decision policy into code. - -### 1.1 Four Pillars of Agentic Evolution - -To drive progress towards the vision of Von as an advanced neuro-symbolic assistant, the primary direction for Von's agentic evolution is built on four key pillars: - -1. **Closing the Workflow Execution "Black Box" Gap**: Ensuring LLM-supervised workflow paths where all automated and sub-agent executions are observable and produce parseable Turn Execution Records. -2. **Materialising Ephemeral Theories**: Supporting self-evolution by allowing agents to generate, test, and either promote or discard temporary working models (ephemeral theories) for self-modification and capability discovery. -3. **Maturing Multi-Agent Coordination**: Extending the Von Workflow Language (VWL) and the Orchestrator to support robust agent-to-agent delegation, capability negotiation, and shared episodic memory contexts. -4. **Transitioning to True Policy Learning**: Moving away from manual prompt tweaks towards automated learning loops that parse Turn Execution Records to induce guidelines, update retrieval strategies, and dynamically adjust model routing. - -## 2. When to read it - -This document is not required by `AGENTS.md`. Read it only when investigating -design provenance or material that may merit deliberate promotion into a -current guide. - -It is especially important when the task touches: - -- ranking, recommendation, matching, routing, classification, planning, or evaluation -- prompts, tool use, workflow behaviour, or agent policies -- retrieval, memory, knowledge bases, or context construction -- ontology or representation design -- model selection, prompt evolution, fine-tuning, or policy learning -- research-sensitive or architecture-shaping work - -## 3. What Von is architecturally - -Von should be treated as a layered neuro-symbolic system, not as a single model wrapped in Python. - -### 3.1 The enduring-knowledge substrate - -This is the durable represented world: - -- Vontology concepts, types, predicates, and text relations -- structured KB assertions with provenance and uncertainty -- workflow and policy metadata -- durable memories and profiles -- materialised outputs worth reusing across sessions, users, and workflows - -This substrate is part of the system's cognition, not just a cache. - -### 3.2 The workflow and control layer - -This is where explicit behaviour structure should live when feasible: - -- state and transition structure -- action selection scaffolding -- escalation, continuation, retry, and pause logic -- schedule, event, and binding behaviour -- policy surfaces that should remain inspectable and revisable - -### 3.3 The prompt-program layer - -Prompts are not comments. They are operational policy. - -Prompt programs should be treated as: - -- authored artefacts -- versioned behaviour surfaces -- optimisable modules -- objects with provenance, evaluation history, and rollback paths - -### 3.4 The model-portfolio layer - -Von should assume a portfolio rather than a single default model: - -- small local or inexpensive models -- medium general-purpose models -- frontier models -- narrow fine-tunes or adapters -- non-neural symbolic or deterministic modules - -The portfolio should be explicitly selectable, evolvable, and measurable. - -### 3.5 The runtime-support layer - -Python usually belongs here: - -- execution surfaces -- validators and constraint checking -- tool wrappers -- rendering and transformation helpers -- telemetry and persistence -- replay, recovery, and safety support - -If Python starts containing the durable task policy, the architecture is drifting. - -### 3.6 The learning and adaptation layer - -Von should be designed to improve through: - -- prompt optimisation -- retrieval and context-policy improvement -- guideline induction from experience -- model routing and portfolio updates -- fine-tuning when justified -- bandit or RL-style decision-policy optimisation when rewards and trajectories are available - -These learning loops should improve authoritative policy surfaces, not encourage policy to migrate into hidden code. - -## 4. Core architectural stance - -### 4.1 Behaviour authority matters - -For Von, behaviour often ought to be authored in: - -- Vontology concepts and text relations -- VWL workflow definitions and metadata -- prompt concepts and related policy text -- explicit KB artefacts with provenance -- typed relations and other inspectable structures - -Python should usually provide only reusable support. - -### 4.2 Decision policy is broader than routing - -Decision policy includes: - -- ranking -- recommendation -- matching -- classification -- retrieval strategy -- explanation choice -- planning -- model selection -- when to continue, ask, escalate, or stop - -If these are being implemented as hidden Python heuristics, the architecture is usually wrong. - -### 4.3 Inspectability is not the same as deterministic Python - -A bounded lexical heuristic can look tidy, interpretable, and reviewable while still placing the real production policy in the wrong place. - -Do not confuse: - -- "I can read the code" -with -- "the system's policy is in the intended authority surface" - -## 5. Reference architecture for substantial behaviour features - -For any substantial behaviour feature, identify all of the following before implementation: - -1. **Authoritative knowledge artefacts** - Which concepts, predicates, prompts, workflow definitions, memory objects, or KB assertions should exist? - -2. **Policy surface** - Is the behaviour primarily in workflow, prompt program, symbolic representation, model router, or learned control policy? - -3. **Runtime support** - What reusable code surfaces are actually needed? - -4. **Candidate models** - Which models or fine-tunes are plausible, and what are their cost, latency, privacy, and capability trade-offs? - -5. **Evaluation and telemetry** - How will success, failure, reliability, consistency, and cost be measured? - -6. **Learning loop** - If the feature should improve over time, what data, rewards, or feedback signals will drive that improvement? - -7. **Fallback strategy** - What bounded fallback exists, and why is it explicitly temporary rather than the durable policy? - -If these questions are unanswered, planning is probably too shallow. - -## 6. Prompt programs and self-improvement - -### 6.1 Prompts are first-class policy objects - -Every important prompt should have: - -- an identity in Vontology -- version history -- owner or task provenance -- intended scope -- evaluation notes -- rollback path - -### 6.2 Prompts should be improvable, not merely editable - -For prompt-governed features, prefer an explicit optimisation loop where feasible: - -- candidate generation -- offline evaluation set -- promotion and demotion criteria -- regression checks -- production telemetry -- rollback and lineage - -### 6.3 Preferred improvement ladder - -Prefer the lightest effective improvement loop: - -1. prompt revision by design -2. retrieval/context improvement -3. experience-derived guidelines or reflections -4. prompt-program optimisation -5. model routing change -6. small fine-tune or adapter -7. RL-style policy optimisation - -Do not jump to heavier interventions when a cheaper, more inspectable layer is still likely to help. - -### 6.4 Self-improving prompts must still be governed - -Self-improvement is not permission for uncontrolled mutation. - -Any prompt optimisation path should define: - -- objective function -- held-out or replay evaluation -- failure detection -- promotion threshold -- rollback condition -- human review threshold for high-risk domains - -## 7. Model portfolios, routing, and fine-tuning - -### 7.1 Model choice is part of the architecture - -Von should not behave as if one model choice is permanently correct. - -Different surfaces may need different substrates: - -- small models for cheap routine transformations -- stronger models for difficult synthesis, planning, or ambiguity -- fine-tunes for narrow stable tasks with sufficient data -- symbolic components where correctness, traceability, or constraints dominate - -### 7.2 Routing should be explicit and learnable - -When model choice materially affects quality or cost, design for an explicit routing policy rather than scattered ad-hoc if/else rules. - -The routing policy should be able to improve using: - -- task metadata -- observed performance -- preference signals -- cost and latency targets -- offline benchmark results - -### 7.3 Do not overfit the codebase to current model quirks - -Avoid embedding today's model-specific workaround into durable code when it really belongs in: - -- prompt policy -- routing policy -- evaluation logic -- validators -- workflow structure - -### 7.4 Fine-tuning is useful but should be narrow and justified - -Use fine-tunes or adapters when all of the following hold: - -- the task is stable and recurring -- prompt and retrieval improvements are not enough -- appropriate data exists -- the deployment trade-offs are acceptable -- the resulting behaviour remains observable and testable - -## 8. Enduring knowledge and memory - -### 8.1 Memory taxonomy - -Von should distinguish at least four forms of durable memory: - -- **semantic memory**: stable facts, concepts, relations, references -- **episodic memory**: interaction and workflow histories -- **procedural memory**: reusable policies, workflows, prompt programs, playbooks -- **policy memory**: learned preferences, routing tendencies, or context-conditional guidelines - -These should not be collapsed into one undifferentiated "memory" bucket. - -### 8.2 Not all memory belongs in the same surface - -Use the right surface for the right durability: - -- transient scratch state in runtime memory -- reusable structured knowledge in Vontology or KB -- evaluated policy or guideline artefacts in durable policy memory -- long-horizon interaction traces as episodic records with promotion rules - -### 8.3 Consolidation matters - -Define when traces should be promoted into durable knowledge: - -- when evidence is strong enough -- when the fact or pattern is reusable -- when provenance can be retained -- when the representation type is clear - -Do not leave all learning trapped in latent model state or ad-hoc logs. - -### 8.4 Retrieval is a policy surface - -Retrieval and context construction are part of the behaviour policy. - -This includes: - -- what gets indexed -- how memory is chunked or structured -- what query transformations are used -- when graph or global retrieval is preferred over local retrieval -- how context budget is allocated - -### 8.5 Use stronger structure when the task needs it - -Use graph, temporal, causal, or other richer structures when the task materially depends on: - -- global corpus sense-making -- long-horizon dependencies -- role structure -- temporal ordering -- causal constraints -- conflict resolution or revision - -Do not assume plain similarity retrieval is always sufficient. - -### 8.6 Also do not assume fancy memory beats a strong baseline - -New memory designs should be compared against: - -- the current production baseline -- a plain strong long-context baseline where feasible -- a simpler retrieval baseline - -Do not introduce elaborate memory machinery only because it sounds advanced. - -## 9. Learning from experience - -### 9.1 Experience can improve more than weights - -Von should learn from traces in several ways: - -- reflection or verbal feedback -- context-aware guidelines -- better retrievers -- better routing policies -- better prompt programs -- fine-tunes -- RL-style control optimisation - -### 9.2 Default progression for decision-policy learning - -When improving decision quality over repeated tasks, prefer the following progression unless there is a strong reason not to: - -1. collect trajectories and telemetry -2. add evaluators and validators -3. derive reflections or guidelines -4. improve prompt and retrieval policy -5. improve routing or contextual selection -6. use bandit or RL-style optimisation if the reward signal is stable and the state/action interface is explicit -7. use fine-tuning when the policy is stable enough to compress into weights - -### 9.3 RL is promising, but not a substitute for architecture - -RL-style optimisation is especially relevant when the task has: - -- explicit or inferable state -- repeated decision points -- delayed reward -- measurable outcomes -- stable action interfaces - -But RL does not remove the need for: - -- explicit authority boundaries -- observability -- safe rollback -- high-quality evaluation -- durable represented knowledge - -## 10. Representation and symbolic reasoning - -### 10.1 Use the weakest adequate representation - -Choose the lightest representation that preserves the structure you need. - -Roughly: - -- plain text for raw user-authored material -- typed entities and relations for stable references and provenance -- workflow/state structure for explicit control -- graph structure for connected multi-entity reasoning -- logical or quasi-logical forms when variables, roles, modality, temporality, normativity, or causal structure matter - -### 10.2 Do not over-formalise for prestige - -Formal representations are justified when they buy real leverage: - -- better inference -- better validation -- better revision -- better explanation -- better reuse -- better auditability - -They are not justified merely because they sound sophisticated. - -### 10.3 Symbolic scaffolds are helpful but not magical - -Adding structure does not guarantee good prediction or decision quality. - -A symbolic scaffold can clarify the problem and improve inspectability, yet the model may still fail at the remaining judgement step. - -Therefore: - -- use explicit structure where it buys leverage -- keep the residual learned component visible -- evaluate the residual learned component directly - -## 11. Evaluation doctrine - -### 11.1 Benchmark optimism is dangerous - -Do not accept a behaviour feature merely because it looks good on a single benchmark or single-shot success metric. - -### 11.2 Evaluate the properties that matter for deployed assistants - -Where relevant, measure: - -- task success -- pass@k or pass^k style consistency -- policy adherence -- perturbation robustness -- calibration and uncertainty quality -- latency and cost -- retrieval hit quality -- long-horizon state retention -- failure visibility and recoverability - -### 11.3 Test the nearest real path - -Especially for user-visible features, acceptance should go through the real control path or the nearest faithful path, not a simplified harness only. - -### 11.4 Evaluate changes in the right place - -If a change claims to improve: - -- prompt policy, evaluate the prompt-governed path -- routing, evaluate the router and its downstream outcomes -- memory, evaluate long-horizon retrieval and use, not only isolated recall -- reasoning, evaluate under structural perturbation where appropriate - -## 12. Research uptake and staying current - -### 12.1 Design for absorbability - -Von should be easy to improve when new research becomes relevant. - -That means: - -- keeping authority surfaces explicit -- keeping runtime support generic -- logging the data needed for future learning loops -- separating policy from implementation scaffolding -- maintaining clean evaluation and rollback paths - -### 12.2 Research-sensitive tasks require a literature note - -For architecture-shaping or research-sensitive work, create a short task note that states: - -- what recent work was checked -- what changed in the plan because of it -- what was rejected and why -- what open question remains - -### 12.3 Prefer integration patterns, not trend chasing - -A new paper should not automatically become production design. - -Adopt a new method only when: - -- it addresses a real Von bottleneck -- it improves the authoritative architecture rather than bypassing it -- it has a clear evaluation story -- it fits Von's deployment constraints - -## 13. Common design smells - -Pause and reconsider when you are about to write: - -- a custom tokeniser for ranking, matching, or routing policy -- stopword lists or lexical-overlap scoring as production semantics -- fixed additive weight tables for durable recommendation or classification policy -- a growing tree of if/else rules that mimics workflow logic -- repo-side workflow/template/prompt files that silently become authoritative -- code-side prompt defaults for a Vontology-governed feature -- hidden model-selection logic scattered through helper code -- a memory system with no promotion, revision, or uncertainty rules -- a fancy retrieval or memory subsystem that has not beaten a simple baseline -- an RL claim without explicit state, action, reward, and rollback definitions - -## 14. Planning checklist for coding agents - -Before implementing substantial behaviour changes, answer these briefly in task notes or Jira comments: - -1. What part of the task is support code, and what part is authored behaviour? -2. Where should the authoritative policy live? -3. Which Vontology concepts, prompts, workflows, or memory objects should exist? -4. What model portfolio or router considerations matter? -5. What evaluation path will test the real behaviour? -6. What learning loop, if any, should make this improve over time? -7. How does the design remain legible if the underlying model improves? -8. What baseline are we comparing against? -9. What temporary fallback exists, and how will it be removed? - -## 15. Closure check - -Before closing a substantial behaviour task, verify all of the following: - -1. The authoritative decision policy is in workflow/prompt/KB/Vontology artefacts or materialised assertions. -2. Python support code remains generic support rather than hidden business policy. -3. Prompt and model choices are visible in evaluation notes and telemetry. -4. Memory or retrieval changes were tested on the intended long-horizon or global-question path, not only a toy path. -5. Any heuristic fallback is explicitly temporary and linked to a removal or replacement task. -6. The acceptance evidence covers the nearest real production path. -7. The task notes record any architecture or literature reinterpretation. - -## 16. Bottom line - -For Von, modern agentic AI does not mainly mean adding more autonomous Python. - -It means designing a system in which: - -- enduring knowledge is explicit -- workflows and prompts are real implementation surfaces -- representation choices are deliberate -- model portfolios are explicit -- learning loops are designed in, not bolted on -- evaluation is realistic and sceptical -- and new research can be adopted by improving policy surfaces rather than by accreting code heuristics - -## 17. Selected motivating literature and SAIL-adjacent themes - -Useful background examples include work on: - -- interleaving reasoning and acting -- reflective or guideline-based learning from experience -- prompt-program optimisation -- graph-based retrieval for global corpus questions -- model routing and cost-aware portfolios -- RL-style training for agent control -- rigorous agent benchmarking -- long-horizon agent memory evaluation -- SAIL work on abstract reasoning limits, logical robustness, counterfactual reasoning, and iterative explanation improvement diff --git a/docs/engineering/jira_authorised_read_replay_2026-09-05.md b/docs/engineering/jira_authorised_read_replay_2026-09-05.md deleted file mode 100644 index 8775ca0f7..000000000 --- a/docs/engineering/jira_authorised_read_replay_2026-09-05.md +++ /dev/null @@ -1,109 +0,0 @@ -# Authorised Jira read replay — 5 September 2026 - -- **Kind:** Bounded acceptance evidence -- **Lifecycle:** Dated record -- **Decision surface:** [JVNAUTOSCI-2722](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-2722) -- **Programme:** [JVNAUTOSCI-2721](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-2721) -- **Machine-readable observations:** [selected receipts](../generated/jira_authorised_read_2026-09-05.json) - -## Claim and baseline - -The configured Jira account's authenticated Von owner can obtain a current -issue and the newest project Task through ordinary chat without importing -tasks or rewriting a digest. Other users cannot borrow that account merely -by supplying the owner's identifier, namespace or account selector. - -The preceding deployed baseline, `16e4fc79089e8480237e22bc3021874ced5f2cd4`, -excluded ordinary Jira reads as `deployment_global_account`. The earlier -Terra trial returned a non-answer. A Luna caller reached Jira through a -Terra digest child that replaced an existing text relation. Those observations -and exact identities remain on the linked issue. Successful operator reads -distinguished that capability/authority mismatch from expired credentials. - -The candidate makes the three existing read tools available to the unique -owner resolved through the configured account's protected Von login-email -binding. The ordinary model chooses its query. The actual Jira proxy checks -the actor and account again, including direct workflow calls. This uses the -existing identity surface; it adds no workflow, semantic routing rule, token -ceremony or broad user grant. The explicit operator migration CLI binds its -operator provenance at its entry point, leaving its shared runner actor-bound. -The current authority rule lives in the [security guide](security_considerations.md). - -## Normal authenticated path - -Four fresh Chrome conversations used Michael's existing normal login and -SAIL organisation on candidate port 5002. No browser-test login, manually -injected credential or test actor supplied the positive path. The candidate -used the normal Atlas data, authentication and Jira services. Its isolated -runtime marker and disabled background consumers prevented duplicate shared -maintenance; those settings did not replace the turn's binding or retrieval. - -The candidate base was `7dd93457695f97deb64369c04aa10cc7a1ddfe40`, with the -three runtime source hashes retained in the selected receipts. Runtime start -was `2026-09-05T11:58:27.550179Z`. Luna remained the saved primary; Terra used -the allowed browser model override. No Sol request was issued. - -The exact prompts were: - -1. “Look up JVNAUTOSCI-2722 in Jira and give its current summary and status.” -2. “What is the most recently created Task in the JVNAUTOSCI Jira project? - Give its key, summary and status.” - -| Model | Case | Source read | Visible completion | Recorded turn time | Estimated USD | -| --- | --- | --- | --- | --- | --- | -| Luna | Exact issue | `jira_get_issue` | Correct, 58 s | 53.970 s | 0.005123 | -| Luna | Newest Task | `jira_search` | Correct, 45 s | 40.868 s | 0.005296 | -| Terra | Exact issue | `jira_get_issue` | Correct, 53 s | 48.883 s | 0.049255 | -| Terra | Newest Task | `jira_search` | Correct, 42 s | 38.308 s | 0.052629 | - -Every answer identified `JVNAUTOSCI-2722`, its current summary and `In Progress`. -The recency queries constrained issue type to Task and sorted Jira `created` -descending, with one result. Canonical persisted Jira responses agreed with -the visible answers. Each record had four provider-observed calls to its -requested model, two capability-discovery calls and one Jira read. Each Jira -response carried the authenticated owner's `governed_login_email_owner` -receipt. Terminal state was `completed`. There was no workflow launch, -import, reconciliation or Jira mutation invocation in these turns; ordinary -conversation persistence still occurred. - -The record includes request/session identifiers, history positions, source -response hashes, effective queries, model identities and selected receipts. -Full private traces and expiring telemetry access envelopes are not published. -The UI duration includes work outside the recorded turn interval. Costs are -Von's estimates from reported usage, not invoices; order and caching were not -randomised. These cases show no Terra-only success and support no general -model ranking. Latency remains substantial for these small requests. - -## Boundary and regression evidence - -The live governed identity resolved an owner binding for Michael and none -for the second actor. With the real configured proxy and identity lookup, -forged owner/namespace/selector input from that actor returned -`jira_resource_not_authorised`; missing trusted actor context returned -`authenticated_actor_context_required`. A direct proxy call in the second -actor's workflow context was also denied. An RPC sentinel recorded zero -external calls in those negative probes. This proves the selected live -binding boundary, not a second positive login route. - -Targeted tests cover all three ordinary read tools, server-only projection, -forged request fields, owner and non-owner workflow calls, changed account -configuration, missing/reassigned/ambiguous identity, separate Jira 401 -results and explicit operator CLI compatibility. The final run was: - -```sh -pdm run pytest tests/backend/test_jira_read_authority.py \ - tests/backend/test_internal_mcp_jira_tools.py \ - tests/backend/test_von_generate_workflow_instances.py \ - tests/backend/test_internal_mcp_catalogue_builds.py -q \ - -k 'not canonical_outcome_report_uses_natural_fact_grounded_projection' -``` - -Result: **124 passed, 1 deselected**. The deselected narration-report test -failed in the broader run and independently on unchanged main `7dd93457` -(`turn_not_deliverable` versus expected `not_required`). It does not exercise -Jira and was not repaired or relabelled as passing. Diff checks passed. - -The evidence supports this owner-read capability. It does not establish -multi-user Jira delegation, the complete four-turn follow-up family, broader -role competence or learned transfer. The continuing programme and release -decision remain on the linked Jira issues. diff --git a/docs/engineering/jira_components_taxonomy.md b/docs/engineering/jira_components_taxonomy.md deleted file mode 100644 index cba2f2450..000000000 --- a/docs/engineering/jira_components_taxonomy.md +++ /dev/null @@ -1,49 +0,0 @@ -# Jira Components Taxonomy for Von - -This file defines a candidate Jira `Components` list for Von work. It is designed to match real subsystem boundaries in the current codebase and Von's product purpose (ontology-first agentic chat, workflows, and integrations). - -## Suggested Naming Convention -- Use short, stable component names with title case. -- Prefer one primary component per issue. -- Add a second component only when work clearly spans two subsystems. - -## Candidate Components - -| Component | Scope | Code Anchors | -| --- | --- | --- | -| Vontology Core | Concept graph lifecycle, predicate/type semantics, canonical IDs | `src/backend/services/concept_service.py`, `src/backend/vontology/utils_vontology.py` | -| Vontology Text Relations | Names/descriptions/content text relation CRUD and policies | `src/backend/services/text_value_service.py`, `src/backend/services/concept_relation_service.py` | -| Vontology Search & Embeddings | Concept search, resolve-by-name, embedding index behaviour | `src/backend/services/concept_search_service.py`, `src/backend/services/concept_embedding_service.py` | -| RAG & Retrieval | Indexing, retrieval, provenance, namespace-aware queries | `src/backend/services/rag_service.py`, `src/backend/services/rag_sync_service.py` | -| Internal MCP Framework | Internal gateway, transport, tool catalogue runtime | `src/backend/integrations/internal_mcp/catalogue.py`, `src/backend/integrations/internal_mcp/gateway.py` | -| MCP Server Surface & Contracts | Stdio tool exposure, manifest parity, input/output schemas | `src/backend/mcp_server/mcp_stdio_server.py`, `src/backend/mcp_server/vontology_mcp.json` | -| Chat Orchestrator | LLM/tool loop, tool-call recovery, turn execution behaviour | `src/backend/integrations/internal_mcp/orchestrator.py` | -| Prompting & Model Policy | Prompt templates, model registry/policy selection | `src/backend/services/prompt_template_service.py`, `src/backend/services/model_registry_service.py` | -| Renderer Applicability & Profiles | Renderer applicability logic, profile resolution, metadata | `src/backend/services/renderer_applicability_service.py`, `src/backend/services/renderer_applicability_vontology_service.py` | -| Display Elements & Presenter Protocol | Render plan, screen/spoken channel shaping, display contracts | `src/backend/services/display_elements_service.py`, `src/backend/server/routes/von_routes.py` | -| Frontend Chat UX | Main chat interaction model and rendering behaviour | `src/frontend/web/von_interface/static/js/chatTab.js` | -| Frontend Debug UX | Debug panels, provenance display, developer-facing UX affordances | `src/frontend/web/von_interface/static/js/components/messagePanel.js`, `src/frontend/web/von_interface/static/js/components/promptCartoucheOverlay.js` | -| Conversation-Turn Workflows | In-turn workflow selection and execution | `src/backend/workflows/workflow_selector.py`, `src/backend/workflows/engine.py` | -| Durable Workflows & Scheduler | Persistent workflow instances, schedules, worker execution | `src/backend/workflows/durable/scheduler.py`, `src/backend/workflows/durable/worker.py` | -| Auth, Session & Namespace | Login/session identity plumbing and namespace derivation | `src/backend/server/routes/auth_routes.py`, `src/backend/services/namespace_service.py` | -| Access Control & Write Guardrails | Write-safety policy, authz checks, guarded operations | `src/backend/workflows/write_tool_policy.py`, `src/backend/server/routes/admin_routes.py` | -| Task & Organisation | Task domain model, organisation membership, task routes | `src/backend/services/task_management_service.py`, `src/backend/services/organisation_membership_service.py` | -| Jira Integration | Jira proxy tools, schema mapping, Jira-specific reliability | `src/backend/integrations/internal_mcp/jira_proxy_mcp.py`, `src/backend/mcp_server/mcp_server.py` | -| Gmail Integration | Gmail OAuth, token handling, message tools | `src/backend/services/agent_gmail_oauth_service.py`, `src/backend/server/routes/agent_gmail_oauth_routes.py` | -| arXiv & Web Research | arXiv metadata/download and web extraction/search pathways | `src/backend/integrations/internal_mcp/arxiv_proxy_mcp.py`, `src/backend/integrations/internal_mcp/search_proxy_mcp.py` | -| Speech & Audio | TTS/speech ingestion and related route behaviour | `src/backend/server/routes/speech_routes.py` | -| Observability & Tracing | Workflow traces, debug metadata, diagnostics surfaces | `src/backend/workflows/trace_store.py`, `src/backend/server/routes/von_routes.py` | -| Data Safety & Migration | Migration tooling, safety guards, backfill utilities | `src/backend/utilities/migrate_workflow_mapping_specs.py`, `src/backend/db/mongo_client.py` | -| Developer Tooling & CI | Test harness behaviour, hooks, lint/type gates, automation | `.githooks`, `pytest.ini`, `pyproject.toml` | - -## Default Starter Set (if you want fewer components initially) -- Vontology Core -- RAG & Retrieval -- Internal MCP Framework -- Chat Orchestrator -- Display Elements & Presenter Protocol -- Frontend Chat UX -- Durable Workflows & Scheduler -- Auth, Session & Namespace -- Task & Organisation -- Jira Integration diff --git a/docs/engineering/jira_github_autofix_loop_workflow.md b/docs/engineering/jira_github_autofix_loop_workflow.md deleted file mode 100644 index fcb19a61a..000000000 --- a/docs/engineering/jira_github_autofix_loop_workflow.md +++ /dev/null @@ -1,278 +0,0 @@ -# Jira-GitHub Autofix Loop Workflow - -Status: Design-only canonical VWL artefact -Last updated: 2026-03-07 (Pacific/Auckland) -Primary Jira task: `JVNAUTOSCI-1338` -Canonical concept ID: `#V#jira_github_autofix_loop_workflow` - -## Purpose - -Define the canonical workflow-first design for a Codex-style Jira-to-GitHub autofix loop in Von. - -This workflow is intended to: - -- discover Jira issues that are explicitly eligible for automation, -- map them to an allowed GitHub repository and base branch, -- launch a guarded implementation path, -- publish the resulting GitHub metadata back to Jira, -- and persist auditable per-run and per-issue outcomes. - -The design is intentionally KB-first. The workflow should live in Vontology and be executed by the lightweight VWL runtime. Supporting Python code should remain limited to reusable actions, validation, scheduling, and telemetry. - -## Non-goals - -- No bespoke Python orchestration service for this loop. -- No silent fallback to ad-hoc scripts when workflow data is missing. -- No unreviewed cross-repository or cross-namespace writes. -- No assumption that a tool success response is equivalent to completed remediation. - -## Trigger Modes - -The workflow supports two trigger modes: - -1. Scheduled polling mode. - Use `workflow_create_schedule` with a bounded interval and explicit default inputs. - -2. Manual operator mode. - Use `workflow_create_instance` directly, or `workflow_trigger_schedule` for an existing schedule. - -The workflow MUST record which trigger mode launched the run. - -## Required Inputs - -- `jira_jql`: bounded JQL used to discover candidate issues. -- `required_labels`: explicit label guard for autofix eligibility. -- `repository_owner`: GitHub owner or organisation. -- `repository_name`: GitHub repository. -- `base_branch`: target base branch for generated work. -- `issue_batch_cap`: maximum issues processed in one run. -- `per_issue_write_mode`: `dry_run|proposal_only|guarded_write`. -- `allowed_components`: optional Jira component filter. -- `allowed_issue_types`: optional Jira issue-type filter. -- `namespace`: authoritative namespace for the run. -- `actor_concept_id`: operator or workflow actor identity. - -## Canonical Context Keys - -The workflow SHOULD use the following stable context keys: - -- `run_correlation_id` -- `trigger_mode` -- `jira_candidate_issues` -- `jira_filtered_issues` -- `current_issue` -- `current_issue_index` -- `issue_outcomes` -- `github_repository` -- `github_branch_name` -- `github_pr_url` -- `github_pr_number` -- `github_copilot_job_id` -- `approval_required` -- `approval_state` -- `idempotency_key` -- `run_summary` -- `blocked_reason` -- `failure_diagnostics` - -These keys are intended to remain stable so later runtime features can rely on them without rewriting the design. - -## End-to-End Step Contract - -### 1. Discover eligible Jira issues - -Tool surface: - -- `jira_search` - -Outputs: - -- `jira_candidate_issues` - -Rules: - -- Query must be bounded by `issue_batch_cap`. -- Only explicitly labelled issues are eligible. -- Discovery must fail closed on Jira auth or query errors. - -### 2. Filter and normalise candidate issues - -Purpose: - -- remove issues already in flight, -- remove issues outside the allow-list policy, -- produce deterministic per-issue input envelopes. - -Outputs: - -- `jira_filtered_issues` -- `failure_diagnostics` - -Required checks: - -- existing PR link or existing branch marker, -- missing repo mapping, -- missing write approval, -- disallowed repository, -- missing required label at execution time. - -### 3. Iterate over eligible issues - -Conceptual behaviour: - -- for each filtered issue, execute the same bounded per-issue subworkflow, -- preserve deterministic ordering, -- collect one outcome envelope per issue. - -Current status: - -- implemented generically in VWL via `workflow_control.for_each`, richer transition conditions, approval-gate metadata, retry policies, and idempotency policies (`JVNAUTOSCI-1339`), -- and therefore no longer a justification for hand-written Python control flow. - -### 4. Per-issue subworkflow - -For each issue, the subworkflow should perform: - -1. Read issue details and comments. -2. Derive a deterministic `idempotency_key`. -3. Resolve repository, owner, and base branch. -4. Re-check for open PRs or existing automation markers. -5. If no write is allowed, emit a blocked outcome and continue. -6. Launch the implementation path. -7. Persist GitHub job, branch, and PR metadata. -8. Comment the resulting status back to Jira. - -Preferred implementation path: - -- `github_create_pull_request_with_copilot` - -Fallback write path: - -- `github_create_branch` -- file mutation or branch update tools only when the policy allows them and the workflow explicitly declares that branch strategy. - -Read-only evidence path: - -- `github_list_pull_requests` -- `github_issue_read` -- `github_pull_request_read` -- `github_list_branches` - -### 5. Summarise run outcome - -Outputs: - -- `issue_outcomes` -- `run_summary` - -The final summary MUST contain: - -- discovered issue count, -- filtered issue count, -- processed issue count, -- blocked issue count, -- successful launch count, -- failed issue count, -- stable references to any PR URLs or Copilot job IDs. - -## Idempotency Model - -Each per-issue execution MUST derive a stable idempotency key from: - -- `issue_key` -- `workflow_id` -- `repository_owner` -- `repository_name` -- `base_branch` -- and a bounded scheduling window or explicit manual-run token. - -The workflow MUST treat the following as duplicate-protection signals: - -- an open PR already linked to the issue, -- a cached job ID still active for the same idempotency key, -- a deterministic branch name already present. - -## Fail-Closed Behaviour - -The workflow MUST no-op with explicit diagnostics when any of the following are missing: - -- Jira auth, -- GitHub auth, -- repository allow-list approval, -- namespace, -- label guard, -- required workflow metadata, -- or required prompt-resolution / completion-gate capabilities from later follow-on tasks. - -Blocked states are valid outcomes. Silent degradation is not. - -## Operator Controls - -Operators must be able to: - -- enable or disable a schedule, -- trigger a schedule manually, -- cancel a running workflow instance, -- retry a failed workflow instance, -- inspect current status and stored outputs. - -Control surface: - -- `workflow_list_schedules` -- `workflow_get_schedule` -- `workflow_set_schedule_enabled` -- `workflow_trigger_schedule` -- `workflow_list_instances` -- `workflow_get_instance` -- `workflow_cancel_instance` -- `workflow_retry_instance` - -## Observability Requirements - -Every run should emit: - -- `run_correlation_id` -- `issue_key` -- `current_issue_index` -- `tool_name` -- `tool_request_id` where available -- `idempotency_key` -- `approval_state` -- `branch_name` -- `pr_url` -- `copilot_job_id` -- `blocked_reason` -- `failure_diagnostics` - -The summary and per-issue envelopes should be stored so an operator can reconstruct: - -- what was attempted, -- what was skipped, -- what was blocked by policy, -- and what external state was created. - -## Current Capability Assessment - -Executable today with existing surfaces: - -- scheduled and manual launch, -- Jira discovery and Jira commenting, -- guarded GitHub proxy invocation, -- issue-level sequential fan-out via `workflow_control.for_each`, -- declarative per-step approval gating, -- declarative retry/backoff policy execution, -- declarative per-step idempotency reuse, -- declarative per-step checkpoint policies, -- workflow-level plan-state tracking with resumable cursor and summary snapshots, -- completion gates that fail closed before terminal success when declared deliverables are unmet, -- prompt and tool metadata resolution from workflow-authored prompt concepts, -- durable workflow instance management, -- high-level run summary persistence. - -## Follow-on Mapping - -- `JVNAUTOSCI-1360`: optional future import or execution of external SKILL artefacts against the stabilised VWL surface. - -## Conformance Note - -This workflow is deliberately specified as a Vontology-first design artefact. When it becomes executable, the implementation should extend the same canonical workflow concept rather than introducing a parallel Python-specific orchestration path. diff --git a/docs/engineering/jira_von_pilot_validation_jvnautosci_1114.md b/docs/engineering/jira_von_pilot_validation_jvnautosci_1114.md deleted file mode 100644 index f8b99764c..000000000 --- a/docs/engineering/jira_von_pilot_validation_jvnautosci_1114.md +++ /dev/null @@ -1,48 +0,0 @@ -# JVNAUTOSCI-1114 Jira->Von Pilot Validation - -Date: 2026-02-20 - -## Goal - -Validate that Von can operate as the primary task system for JVNAUTOSCI-style workflows after Jira import, with explicit classification of blocking vs deferrable parity gaps. - -## Runtime Output - -`task_import_jira_issues` now returns: - -1. `project_parity`: per-project mapped/dropped parity details. -2. `pilot_validation`: deterministic classification and recommendation payload. - -`pilot_validation` fields: - -1. `summary.issues_scanned` -2. `summary.must_fix_gap_count` -3. `summary.acceptable_defer_gap_count` -4. `recommendation` (`go`, `go_with_conditions`, `no_go`) -5. `must_fix_gaps[]` -6. `acceptable_defer_gaps[]` - -## Classification Policy - -1. `must_fix`: unmapped Jira status/priority values (`status_mapping`, `priority_mapping`). -2. `acceptable_defer`: planning-model gaps where values are preserved and queryable but not yet represented as typed planning entities: - - components - - fix versions - - sprint values - - backlog rank - -## Recommendation Policy - -1. `no_go`: one or more `must_fix` gaps are present. -2. `go_with_conditions`: zero `must_fix` gaps, but one or more `acceptable_defer` gaps remain. -3. `go`: no detected gaps in the scanned sample. - -## Real Backlog Alignment Notes - -Pilot checks against JVNAUTOSCI project data showed real statuses not covered by the original mapper. The importer now maps: - -1. `SUPERSEDED` -> `cancelled` -2. `Won't Fix` / `Wont Fix` -> `cancelled` -3. `SUSPENDED` -> `blocked` - -Planning metadata observed in Jira (`components`, `rank`, and where present `fixVersions`/sprints) is now mapped into Von task metadata fields and exposed through REST and MCP search/update pathways. diff --git a/docs/engineering/jvnautosci-990-specialised-concept-suggestion-workflow-evaluation.md b/docs/engineering/jvnautosci-990-specialised-concept-suggestion-workflow-evaluation.md deleted file mode 100644 index afd5bd8f0..000000000 --- a/docs/engineering/jvnautosci-990-specialised-concept-suggestion-workflow-evaluation.md +++ /dev/null @@ -1,51 +0,0 @@ -# JVNAUTOSCI-990: Specialised Concept-Suggestion Workflow Evaluation - -## Summary -- Implemented a dedicated Stage-5 workflow: `#V#concept_suggestion_preflight_workflow`. -- The stage runs only when earlier preflight stages are sparse (missing type and/or predicate suggestions). -- Rollout is controlled by `VON_MCP_SPECIALISED_PREFLIGHT_MODE`: - - `off` (default): disabled. - - `shadow`: evaluate and emit telemetry without applying suggestions. - - `active`: apply specialised suggestions to final preflight candidates. - -## Inputs, Outputs, and Guardrails -- Inputs: - - Current user prompt. - - Baseline type and predicate suggestions from earlier stages. - - Candidate pools from topic, annotation, RAG, salient predicates, and session memory. -- Outputs: - - `specialised_type_suggestions` - - `specialised_predicate_suggestions` - - evaluation telemetry (mode, invoked, projected gain, recommendation). -- Guardrails: - - Accept only well-formed `#V#...` concept IDs. - - Strict de-duplication. - - Bounded output (`_SPECIALISED_PREFLIGHT_MAX_SUGGESTIONS`). - - Fail-safe: if workflow errors, baseline suggestions remain authoritative. - -## Small Evaluation Set -- Scenario A (shadow mode, sparse baseline): - - Workflow invoked and produced bounded type/predicate suggestions. - - Final candidate set remained unchanged. - - Recommendation emitted as `go_active_trial` when projected gain was positive. -- Scenario B (active mode, sparse baseline): - - Workflow invoked and merged specialised suggestions into final candidates. - - Provenance path `specialised_preflight_workflow` recorded in telemetry. - - Recommendation emitted as `go_adopted`. -- Scenario C (baseline sufficient): - - Stage remains skipped by design (`specialised_preflight_needed=false`). - -## Go/No-Go Recommendation -- Recommendation: **Go**, with staged rollout. -- Rollout sequence: - 1. Enable `shadow` mode and monitor projected gain + error rates. - 2. Promote to `active` when shadow telemetry remains positive and stable. - -## Integration and Fallback Plan -- Integration: - - Built-in workflow definition + action registration. - - Canonical workflow publication in authority service. - - Preflight telemetry extended with baseline vs specialised counts and recommendation. -- Fallback: - - Default `off` keeps existing behaviour unchanged. - - Any specialised workflow failure degrades cleanly to baseline preflight outputs. diff --git a/docs/engineering/jvnautosci_1244_considerations_cadence_design.md b/docs/engineering/jvnautosci_1244_considerations_cadence_design.md deleted file mode 100644 index 9c219dd1d..000000000 --- a/docs/engineering/jvnautosci_1244_considerations_cadence_design.md +++ /dev/null @@ -1,157 +0,0 @@ -# JVNAUTOSCI-1244 Considerations-for-Use Cadence Design - -Date: 2026-02-22 - -## Scope - -This task is planning-only. It defines the global cadence design for Considerations-for-Use generation and does not introduce production code changes in this ticket. - -## Objective - -Adopt a workflow-first global cadence policy so that: - -1. `concept.created` keeps immediate Considerations-for-Use generation. -2. `concept.updated` stops launching immediate enrichment and instead feeds a global ~5-minute cadence run. -3. Repeated updates inside a cadence window dedupe cleanly and do not create duplicate work. - -## Current State (Observed) - -Observed from workflow surfaces and event-launch code: - -1. `#V#enrichment_workflow` currently has `background_launch_policy` with: - - `enabled=true` - - `min_interval_seconds=300` - - `scope=global_per_server` - - `applies_to_sources=["event"]` -2. Both `concept.created` and `concept.updated` are bound directly to `#V#enrichment_workflow` for `#V#has_considerations_for_use`. -3. Cadence gating is evaluated before instance creation and is workflow-global, so one recent event-triggered instance can block another event type for the same workflow ID. -4. There is no explicit update-candidate queue for later scheduled processing. -5. A daily enrichment schedule exists but is disabled and is not the required 5-minute update cadence. - -Implication: current gating is a skip gate, not a queue-backed cadence system. - -## Selected Trigger Policy - -1. `concept.created`: - - Immediate launch remains enabled. - - Must not be blocked by update cadence throttling. -2. `concept.updated`: - - No direct enrichment launch. - - Event path records/refreshes a pending candidate marker. - - Processing happens only through a global 5-minute scheduler. - -## Workflow-First Architecture - -### A. Event intake (updated concepts) - -`concept.updated` binding targets an intake workflow (or intake-capable step) that only records candidate state. - -Required behaviour: - -1. Idempotent candidate upsert keyed by `concept_id`. -2. Update marker timestamp on every update event. -3. Do not spawn enrichment instances from this path. - -### B. Immediate path (new concepts) - -`concept.created` binding continues to launch immediate enrichment for `#V#has_considerations_for_use` with `force_concept_id=event.concept_id` and `limit=1`. - -### C. Global cadence dispatcher - -One server-wide schedule runs every 300 seconds and dispatches bounded enrichment batches from pending candidates. - -Required behaviour: - -1. Deterministic candidate ordering (oldest pending first). -2. Bounded batch size per run (operator-configurable). -3. Candidate state updated atomically after success/failure. -4. Retries use bounded attempts with next-attempt timestamp. - -## Candidate Data Model (Vontology-first) - -Persist candidate state as Vontology-governed data, not ephemeral in-memory state. - -Minimum logical fields: - -1. `concept_id` -2. `pending_since` -3. `last_updated_event_at` -4. `last_attempt_at` -5. `attempt_count` -6. `last_result` (`success|failed|skipped`) -7. `last_error` (optional diagnostic text) - -Representation can be a dedicated candidate concept type or canonical text relations, but persistence must remain Vontology-managed and queryable via Vontology services/tools. - -## Dedupe and Back-Pressure Rules - -1. Multiple updates for the same concept inside one cadence window collapse to one pending candidate. -2. Dispatcher enforces a per-run batch cap and leaves residual candidates pending. -3. Repeated failures move candidates to retry-later state (bounded retry policy). -4. Scheduler must not launch overlapping dispatcher runs for the same workflow (instance-level idempotency/locking). - -## Observability Requirements - -Per cadence run, emit telemetry including: - -1. `window_started_at`, `window_ended_at`, `duration_ms` -2. `pending_at_start`, `processed_count`, `success_count`, `failed_count`, `skipped_count`, `remaining_count` -3. `batch_size_limit`, `retry_count`, `oldest_pending_age_seconds` -4. `reason` when no work is processed (for example `no_pending_candidates`, `dispatcher_locked`) - -Per event intake, emit: - -1. `event_type`, `concept_id` -2. `candidate_state` (`created|refreshed|unchanged`) - -## Migration and Rollout Plan - -1. Introduce candidate intake for `concept.updated` while keeping current immediate path temporarily behind a feature flag. -2. Enable 5-minute dispatcher schedule and validate telemetry + throughput. -3. Disable direct `concept.updated` enrichment launch after verification. -4. Keep `concept.created` immediate path enabled throughout. - -Rollback: - -1. Re-enable direct `concept.updated` launch binding. -2. Disable dispatcher schedule. -3. Preserve candidate records for forensic analysis; do not delete on rollback. - -## Edge Cases (Explicit) - -1. Burst updates on one concept: dedupe by `concept_id`, retain freshest update timestamp. -2. Duplicate event deliveries: intake remains idempotent. -3. Backlog growth: bounded batch processing with residual carry-over and age telemetry. -4. Service restart/recovery: pending candidates persist and are resumed by next schedule tick. -5. Concurrent scheduler ticks: guarded by workflow instance lock/idempotency. -6. Created-then-updated quickly: create path runs immediately; update path queues potential follow-up refresh. - -## Test Plan (for implementation task) - -### Unit tests - -1. Intake dedupe behaviour for repeated `concept.updated` events. -2. Candidate ordering and batch slicing. -3. Retry/backoff transitions and terminal failure handling. -4. Created-vs-updated policy split (created immediate, updated queued). - -### Integration tests - -1. Event binding `concept.updated` writes candidate state without launching enrichment. -2. 5-minute dispatcher launches enrichment for queued candidates and updates state. -3. Cadence runs remain deterministic under burst updates. -4. Restart scenario resumes pending candidates. - -### End-to-end tests - -1. Real call path: update concept -> candidate marked -> scheduled run -> `#V#has_considerations_for_use` updated. -2. Real call path: create concept -> immediate enrichment occurs without waiting for cadence. -3. Operator control checks: disable/enable schedule toggles behaviour as expected. - -## Completion Criteria for JVNAUTOSCI-1244 - -1. Design note exists (this document). -2. Decision for created-vs-updated handling is explicit. -3. Dedupe/back-pressure/observability/migration are specified. -4. Test plan is specified. -5. No production code changes are included in this ticket. diff --git a/docs/engineering/jvnautosci_1399_bare_arxiv_url_integration_test_design.md b/docs/engineering/jvnautosci_1399_bare_arxiv_url_integration_test_design.md deleted file mode 100644 index 50479b6f9..000000000 --- a/docs/engineering/jvnautosci_1399_bare_arxiv_url_integration_test_design.md +++ /dev/null @@ -1,431 +0,0 @@ -# JVNAUTOSCI-1399 Bare arXiv URL Integration Test Design - -This note defines the authoritative integration-test plan for the user-visible -chat path that starts at `/von/generate` and is expected to drive -workflow-governed scholarly-paper representation from arXiv URLs. - -It exists because adjacent coverage already passes while the real route can -still fail before any tool executes, as shown by `JVNAUTOSCI-1394`. It also -takes account of the recent provider-aware dispatch fix in `JVNAUTOSCI-1396` -and the deterministic targeted-lane pytest workflow established by -`JVNAUTOSCI-1400`. - -## Linked Work - -- `JVNAUTOSCI-1394`: implementation bug for arXiv URL representation turns -- `JVNAUTOSCI-1369`: required-effects and completion-gate groundwork -- `JVNAUTOSCI-1396`: provider-aware workflow-dispatch fix already merged -- `JVNAUTOSCI-1400`: deterministic targeted and aggregate lane strategy; - validation should follow `docs/engineering/pytest_lane_strategy.md` - -## Authoritative Behaviour Decision - -Two cases must be separated. - -### 1. Bare canonical arXiv URL - -Prompt shape: - -```text -https://arxiv.org/abs/2510.06248 -``` - -Authoritative behaviour: - -- This **is** sufficient permission for `download_paper` and additive scholarly - representation. -- The turn must enter the workflow/tool pipeline rather than collapsing into a - read-only or plain-response path. -- The turn must carry a paper representation contract with - `artefact_source="url"` and must require `download_paper`. -- Successful completion requires real tool execution plus verified scholarly - representation postconditions. - -Why: - -- A canonical arXiv abstract URL is evidence-backed, low-risk input for - additive representation work. -- Minimal imposition means not making the user add extra phrasing such as - "download", "store", or "represent" when the intended low-risk additive - action is already clear from the supplied canonical source. -- This is the primary positive mutation path that `JVNAUTOSCI-1394` must fix. - -Equivalent positive prompts MAY also include explicit phrasing such as: - -```text -https://arxiv.org/abs/2510.06248 represent that paper -``` - -but that wording is no longer the sole authorising form. - -### 2. URL plus explicit denial - -Prompt shape: - -```text -Do not download or store this: https://arxiv.org/abs/2510.06248 -``` - -Authoritative behaviour: - -- Explicit user denial must block mutation. -- The route may still answer read-only, but it must not invoke - `download_paper`, persist artefacts, or claim successful representation. - -This explicit-denial case is the canonical negative regression for -`JVNAUTOSCI-1394`, because it preserves fail-closed behaviour where the user -has provided direct contrary instruction. - -## Runtime Boundary To Test - -The authoritative entry point is the Flask route: - -- `src/backend/server/routes/von_routes.py` -- route: `/von/generate` - -The integration test must not stop at direct orchestrator or handler calls. -It should exercise: - -1. `/von/generate` -2. real `InternalMCPChatOrchestrator` -3. real `InternalMCPGateway.invoke(...)` -4. real internal `download_paper` tool handler path -5. ontology writes/materialisation -6. completion gate -7. returned `llm_debug` -8. persisted `turn_execution_record` - -Reason: - -- The observed failure in `JVNAUTOSCI-1394` occurs before tool execution, so - downstream-only tests are insufficient. -- Route-level `llm_debug` and turn-record persistence are part of the user- and - operator-visible contract. - -## Workflow Authority - -For fresh tool-seeking chat turns, the authoritative execution wrapper is: - -- `#V#tool_calling_workflow` - -Evidence: - -- The selector logic in `src/backend/integrations/internal_mcp/orchestrator.py` - maps `tool_seeking` verdicts to `#V#tool_calling_workflow`. -- Existing selector routing tests assert that tool-seeking turns resolve to the - tool-calling wrapper rather than directly to a domain-specific workflow ID. - -The domain-specific scholarly authority still matters, but one layer down: - -- the paper representation contract is sourced from - `#V#representation_contract_profile_paper` -- the Vontology concept `#V#scholarly_paper_representation_workflow` exists and - encodes the canonical paper-specific step structure, including - `resolve_scholarly_authors` - -So the current architecture is: - -- route-level workflow selection: `#V#tool_calling_workflow` -- domain-specific mutation requirements: paper required-effects contract -- canonical paper process model: `#V#scholarly_paper_representation_workflow` - -Important distinction: - -- Upload-event routing in - `src/backend/workflows/durable/file_copy_upload_classification_workflow.py` - still defaults to `#V#integration_scholarly_paper_representation_workflow`. -- That ID does not currently resolve as a Vontology concept via the current - concept-access surface. -- `JVNAUTOSCI-1399` should therefore treat the upload-event workflow constant as - separate from the authoritative fresh-chat execution wrapper. - -Implication for the future implementation task: - -- `JVNAUTOSCI-1394` should make explicit URL-first chat turns reach the - tool-calling wrapper with the correct paper contract and tool requirements. -- If the architecture later evolves to direct custom workflow dispatch for this - domain, the integration assertions can be updated then; 1399 should reflect - the current real execution path. - -## Required-Effects Contract To Assert - -The canonical paper profile in -`src/backend/services/representation_contract_vontology_service.py` already -defines: - -- `profile_id = "paper"` -- `target_entity_class = "scholarly_paper"` -- `effect_type = "scholarly_representation"` -- `required_tools_by_source["url"] = ["download_paper"]` - -The VWL manual also states: - -- `url` source for scholarly paper representation must require `download_paper` - -Therefore the positive integration test must assert: - -- `llm_debug.turn_execution_record.execution.required_effects_contract.schema_version == "required_effects_contract.v1"` -- `...domain_profile_id == "paper"` -- `...target_entity_class == "scholarly_paper"` -- `...artefact_source == "url"` -- `llm_debug.turn_execution_record.required_effects[*].required_tools` includes - `download_paper` - -## Authoritative Telemetry Fields - -The test should assert against both route-level debug and the canonical turn -record, because both are user-visible operational interfaces. - -### Route-level workflow routing - -Assert: - -- `body["llm_debug"]["workflow_routing"]["workflow_id"]` -- `body["llm_debug"]["workflow_routing"]["verdict"]` -- `body["llm_debug"]["workflow_routing"]["source"]` - -For the bare-URL positive case, the expected selected workflow ID is: - -- `#V#tool_calling_workflow` - -The paper-specific behaviour should be asserted via the required-effects -contract and tool/postcondition evidence, not by expecting direct selection of -the scholarly workflow concept. - -### Canonical turn-execution record - -Assert: - -- `body["llm_debug"]["turn_execution_record"]["workflow_selection"]["selected_workflow_id"]` -- `...["workflow_selection"]["selector_verdict"]` -- `...["workflow_selection"]["selector_source"]` -- `...["execution"]["required_effects_contract"]` -- `...["completion_gate"]["decision"]` -- `...["completion_gate"]["safe_to_claim_completion"]` -- `...["completion_gate"]["requires_follow_up"]` - -### Stage-path telemetry - -Assert that -`body["llm_debug"]["turn_execution_diagnostics"]["workflow_stage_path"]["path"]` -contains, at minimum, these stage IDs for the positive case: - -- `workflow_dispatch` -- `tool_plan` -- `tool_execute` -- terminal stage `completed` - -The regression must fail if the route again collapses into a `screen_backfill` -/ `response_finalising` path with zero tool execution. - -### Tool evidence - -Assert both: - -- `body["llm_debug"]["turn_execution_diagnostics"]["tool_history"]` contains a - `download_paper` entry -- `body["llm_debug"]["turn_execution_record"]["execution"]["tool_invocations"]` - contains a successful `download_paper` invocation - -The positive test should fail if completion is claimed without that tool -evidence. - -## Ontology Postconditions - -The authoritative postconditions should match the deterministic arXiv -materialisation helper in -`src/backend/services/arxiv_paper_link_service.py`. - -For the positive bare-URL case, assert: - -- a stable paper concept exists -- the paper concept is an instance of `#V#scholarly_article` -- the paper is linked to the produced file-copy concept via - `#V#propositional_information_thing_has_computer_file` -- the paper has canonical name text including the arXiv ID and title -- the paper has a summary in `hasDescription` -- the paper has `#V#authored_by` links to author concepts -- the paper has `#V#about` links to topic concepts when categories are present - -Author handling decision: - -- Author representation is **not** a separate workflow for this test. -- It is an effect inside the scholarly-paper representation flow. -- The chat workflow concept explicitly includes `resolve_scholarly_authors`, and - the deterministic materialisation helper resolves or creates person concepts - and writes `#V#authored_by` relations. - -So the integration test should assert authorship as a postcondition of the same -paper-representation turn, not as dispatch to a separate author workflow. - -## Recommended Harness Design - -### Flask app - -Use a real Flask app with `von_bp` registered at `/von`. - -### Authentication/session - -Use an authenticated session so the write path is permitted to register file -copies and scholarly representation: - -- `user_concept_id` -- `organisation_concept_id` if needed by the route setup -- deterministic `session_id` - -### Orchestrator - -Use the real `InternalMCPChatOrchestrator`. - -Patch only external seams: - -- selector/planner/final-response LLM calls return deterministic scripted - responses -- any network-facing arXiv/PDF retrieval dependency -- blob-store backend - -Do not stub out the orchestrator’s routing, contract, or completion logic. - -### Gateway - -Use a real `InternalMCPGateway` with the default catalogue and transport. - -Patch external dependencies under the `download_paper` path rather than -replacing the gateway itself. This keeps the regression on the real -`gateway.invoke(...) -> tool handler -> tool payload` boundary. - -### Metadata fixture - -Use a deterministic arXiv metadata fixture for `2510.06248` with: - -- title -- summary -- authors -- categories - -The existing upload integration test for `2502.14996` is the right model for -this kind of deterministic metadata fixture. - -## Fixture and Database Strategy - -The canonical strategy is a per-test cloned database, not reuse of the shared -`test_von_db`. - -Recommended setup: - -1. clone the ontology slice from `test_von_db` into a fresh per-test clone DB -2. point `VON_DB_NAME` at the clone -3. preflight-assert that the clone contains: - - `#V#scholarly_paper_representation_workflow` - - `#V#representation_contract_profile_paper` - - required core types/predicates for scholarly materialisation - -Why clone instead of global/shared fixtures: - -- the route writes more than ontology concepts, including history and execution - projections -- dropping one temporary DB is more deterministic than trying to surgically - delete concepts after partial failure -- this avoids visibility confusion around user/org-scoped versus global objects - -## Rollback and Failure Handling - -The canonical rollback is: - -1. create a uniquely named clone DB for the test -2. run the full route-level scenario there -3. always drop the clone DB in `finally` -4. clean any fake/local blob-store artefacts used by the test harness - -This is preferable to relation-by-relation cleanup because the route can create: - -- ontology concepts and text relations -- chat history rows -- interaction sessions -- turn execution records -- workflow runtime artefacts - -Within the test body, still record created IDs for diagnostics: - -- paper concept ID -- file-copy concept ID -- author concept IDs -- workflow instance IDs if any appear -- request ID - -These IDs should be emitted in failure messages to support debugging, but the -authoritative cleanup mechanism should remain clone-DB deletion. - -## Proposed Targeted Test Matrix - -### A. Positive regression case - -`test_von_generate_bare_arxiv_url_auto_represents_paper` - -Prompt: - -```text -https://arxiv.org/abs/2510.06248 -``` - -Assert: - -- selected workflow is `#V#tool_calling_workflow` -- stage path includes dispatch, plan, execute, completed -- `download_paper` executed through the gateway/tool path -- required-effects contract says `artefact_source="url"` -- required-effects contract domain is `paper` -- completion gate is `completed` and `safe_to_claim_completion=True` -- scholarly ontology postconditions exist - -This is the regression that should fail on the broken `JVNAUTOSCI-1394` -behaviour. - -### B. Negative safety case - -`test_von_generate_bare_arxiv_url_with_explicit_denial_stays_non_mutating` - -Prompt: - -```text -Do not download or store this: https://arxiv.org/abs/2510.06248 -``` - -Assert: - -- no `download_paper` invocation -- no completion claim that implies successful representation -- the turn remains non-mutating despite the canonical URL being present - -This protects the explicit-denial boundary. - -## Validation Scope - -Per `JVNAUTOSCI-1400` and `docs/engineering/pytest_lane_strategy.md`, -validation for the eventual implementation should stay targeted: - -- the new route-level arXiv integration test module -- adjacent required-effects/turn-record tests only if the implementation - changes those shared pathways - -Do not make the fix depend on a monolithic full-suite pytest run. - -## Implementation Sequence For JVNAUTOSCI-1394 - -1. Add the positive bare-URL route-level integration test. -2. Add the explicit-denial negative regression. -3. Fix the URL-first routing/tool-requirement gap until the positive test passes. -4. Only then tighten any telemetry assertions that depend on - `JVNAUTOSCI-1397`. - -## Open Architectural Note - -The current upload classification constant -`#V#integration_scholarly_paper_representation_workflow` appears to be code-only -from the present Vontology surface, while the chat workflow -`#V#scholarly_paper_representation_workflow` is a real Vontology concept with a -published process graph. - -That discrepancy does not block `JVNAUTOSCI-1399`, but the eventual -implementation and cleanup work should keep the two paths intentionally aligned -or intentionally separate rather than assuming they are already the same thing. diff --git a/docs/engineering/jvnautosci_1415_paper_arxiv_workflow_design.md b/docs/engineering/jvnautosci_1415_paper_arxiv_workflow_design.md deleted file mode 100644 index df17cbd2e..000000000 --- a/docs/engineering/jvnautosci_1415_paper_arxiv_workflow_design.md +++ /dev/null @@ -1,293 +0,0 @@ -# JVNAUTOSCI-1415 Paper and arXiv Workflow Design - -Status: Planning baseline for implementation -Last updated: 2026-03-10 (Pacific/Auckland) -Task: `JVNAUTOSCI-1415` - -## 1. Why this design exists - -Representing scientific papers is a core lab activity, but the current workflow picture is split across: - -- representation contracts and completion-gate logic, -- tool-handler-side materialisation in `download_paper`, `finalise_cached_paper`, and `interpret_file_copy`, -- an existing but incomplete `#V#scholarly_paper_representation_workflow`, -- and a code-only upload constant `#V#integration_scholarly_paper_representation_workflow`. - -That is enough for some paths to work, but it is not a good knowledge-base representation of the behaviour. `JVNAUTOSCI-1415` therefore needs explicit VWL/Vontology workflow artefacts for: - -- general scientific/scholarly paper representation, and -- arXiv-specific acquisition and normalisation as a wrapper around that general paper workflow. - -## 2. Audited current state - -### 2.1 Confirmed Vontology and code facts - -- `#V#scholarly_paper_representation_workflow` exists in Vontology. -- `#V#integration_scholarly_paper_representation_workflow` does not currently exist in Vontology. -- no explicit arXiv wrapper workflow concept currently exists. -- the current `#V#scholarly_paper_representation_workflow` loads and validates, but its graph is legacy/incomplete: - - 10 steps, - - most steps invoke `workflow_creation.emit_marker`, - - only `workflow_creation.resolve_scholarly_authors` is domain-specific, - - no `invokesWorkflow` delegation, - - no explicit arXiv acquisition branch, - - no meaningful context-input or tool-output mapping contracts. -- file-upload classification still defaults the scholarly route to the non-existent `#V#integration_scholarly_paper_representation_workflow`. -- current real paper/arXiv materialisation mostly lives in reusable services and tool handlers: - - `src/backend/services/arxiv_paper_link_service.py` - - `src/backend/integrations/internal_mcp/catalogue.py` -- the current manual was broadly aligned on runtime semantics, but it did not yet document the scholarly-workflow incompleteness or the upload-workflow identity split. - -### 2.2 Design implication - -The current scholarly workflow should be treated as a legacy baseline to repair or replace while preserving compatibility, not as the finished canonical topology. - -## 3. Design goals - -- Keep workflow behaviour represented primarily in Vontology/VWL, not hidden in Python orchestration. -- Reuse the existing `#V#scholarly_paper_representation_workflow` identifier as the canonical general-paper workflow, so current references do not all need to be renamed. -- Introduce a new explicit arXiv wrapper workflow that delegates to the general paper workflow. -- Align upload routing with the canonical general-paper workflow instead of a code-only placeholder ID. -- Make the workflow family inspectable: - - explicit workflow concepts, - - explicit step graphs, - - explicit workflow-to-workflow delegation, - - explicit context keys and output mappings, - - explicit verification outputs. -- Keep Python changes limited to reusable workflow actions/validators/control surfaces that VWL needs in order to express the intended graph. - -## 4. Non-goals - -- Do not treat `#V#tool_calling_workflow` as the long-term canonical representation of paper/arXiv behaviour. -- Do not encode the paper/arXiv sequence as bespoke route-specific Python logic if VWL can represent it. -- Do not make arXiv a separate domain that duplicates the whole general paper workflow; it should wrap and specialise the general workflow. - -## 5. Target workflow family - -### 5.1 Canonical general workflow - -Workflow ID: - -- `#V#scholarly_paper_representation_workflow` - -Role: - -- canonical workflow for representing scientific/scholarly papers in general, -- reusable from upload-driven, URL-driven, and already-registered file-copy flows, -- domain workflow that owns paper-specific representation semantics rather than conversation-turn wrapper semantics. - -Expected inputs: - -- `file_copy_concept_id` (required for current implementation phase) -- `paper_concept_id` (optional pre-existing target) -- `paper_metadata` (optional structured metadata) -- `source_uri` (optional) -- `source_label` (optional) -- `arxiv_id` (optional) -- `verification_profile` (`generic` or `arxiv`) - -Expected outputs: - -- `paper_concept_id` -- `scholarly_representation` -- `scholarly_representation_verified` -- `verification_failures` -- `author_concept_ids` -- `topic_concept_ids` -- `representation_mode` - -Proposed step graph: - -1. `normalise_inputs` - - new reusable action: `scholarly_paper.normalise_inputs` - - purpose: validate required inputs, normalise metadata payload shape, set verification defaults, expose deterministic context keys. -2. `materialise_base_representation` - - new reusable action: `scholarly_paper.materialise_from_file_copy` - - purpose: use the canonical materialisation helpers to ensure a paper concept exists, has the correct type, and is linked to the file copy. -3. `enrich_from_metadata` - - new reusable action: `scholarly_paper.enrich_from_metadata` - - purpose: assert/update title, description, topic labels, and author-name candidates from supplied metadata. -4. `resolve_authors` - - new reusable action: `scholarly_paper.resolve_authors` - - likely implemented by extracting/generalising current `workflow_creation.resolve_scholarly_authors` behaviour into a domain action with a non-workflow-creation name. -5. `verify_representation` - - new reusable action: `scholarly_paper.verify_representation` - - purpose: check the representation against the requested verification profile and emit explicit failure codes. -6. `completed` -7. `failed` - -Branching expectations: - -- any failed domain action goes to `failed`; -- `verify_representation` succeeds only when the requested profile is satisfied; -- verification profile `generic` should require: - - paper concept exists, - - paper typed as `#V#scholarly_article`, - - file-copy link exists, - - at least one stable name/title signal exists; -- verification profile `arxiv` should require the generic checks plus: - - arXiv identifier asserted, - - title present, - - summary present, - - authors linked, - - topic/category evidence present when available from metadata. - -### 5.2 Explicit arXiv wrapper workflow - -Workflow ID: - -- `#V#arxiv_paper_representation_workflow` - -Role: - -- recognise and normalise canonical arXiv inputs, -- acquire or finalise the paper artefact, -- delegate to `#V#scholarly_paper_representation_workflow`, -- apply stricter arXiv-specific verification. - -Expected inputs: - -- `prompt` or `source_uri` or `arxiv_id` -- optional `file_copy_concept_id` -- optional `paper_metadata` - -Expected outputs: - -- `arxiv_id` -- `file_copy_concept_id` -- `paper_concept_id` -- `scholarly_representation` -- `scholarly_representation_verified` -- `verification_failures` - -Proposed step graph: - -1. `normalise_arxiv_source` - - new reusable action: `arxiv.normalise_source` - - purpose: extract a canonical arXiv ID from URL/ID/PDF filename/current context. -2. `decide_acquisition_mode` - - new reusable action: `arxiv.decide_acquisition_mode` - - purpose: decide whether the workflow already has a usable file copy or must call `download_paper`. -3. `download_or_finalise` - - action/tool: `download_paper` - - only on the branch where acquisition is required. - - rely on current tool semantics that prefer `finalise_cached_paper` when appropriate. -4. `delegate_to_general_paper_workflow` - - action: `workflow_invoke_subworkflow` - - invoked workflow: `#V#scholarly_paper_representation_workflow` - - pass `file_copy_concept_id`, `paper_metadata`, `arxiv_id`, and `verification_profile=arxiv`. -5. `verify_arxiv_path` - - action: `scholarly_paper.verify_representation` - - explicit second verification step so the wrapper itself can fail closed with arXiv-specific failure codes. -6. `completed` -7. `failed` - -Branching expectations: - -- if `decide_acquisition_mode` says an existing file copy is already available, skip `download_or_finalise` and go straight to delegation; -- if `download_paper` returns failure or no usable `computer_file_copy_concept_id`, fail closed; -- if the delegated general workflow fails or produces incomplete verification, the wrapper fails. - -### 5.3 Upload alignment - -The upload-classification scholarly default should be changed from: - -- `#V#integration_scholarly_paper_representation_workflow` - -to: - -- `#V#scholarly_paper_representation_workflow` - -This keeps upload-event-driven scholarly handling aligned with the same canonical general paper workflow that the arXiv wrapper delegates to. - -## 6. Required reusable action/control-surface work - -This task should add reusable workflow actions where the workflow graph currently has no good first-class primitive. - -Actions to add or refactor: - -- `scholarly_paper.normalise_inputs` -- `scholarly_paper.materialise_from_file_copy` -- `scholarly_paper.enrich_from_metadata` -- `scholarly_paper.resolve_authors` -- `scholarly_paper.verify_representation` -- `arxiv.normalise_source` -- `arxiv.decide_acquisition_mode` - -Migration note: - -- `workflow_creation.resolve_scholarly_authors` should not remain the canonical domain action name inside the repaired paper workflow. -- It can be retained temporarily as a compatibility alias if needed, but the canonical workflow should use a domain-specific action namespace. - -## 7. Vontology artefacts to create or update - -### 7.1 Update existing concept - -- repair/rewrite `#V#scholarly_paper_representation_workflow` -- replace the current marker-heavy step graph with the designed general-paper graph -- update its `hasDescription` and `hasContent` text so they describe the actual workflow, not the legacy generated request - -### 7.2 Create new workflow concept - -- create `#V#arxiv_paper_representation_workflow` -- type it as `#V#ai_workflow` and `#V#durable_workflow` -- attach explicit step concepts and `invokesWorkflow` edge to `#V#scholarly_paper_representation_workflow` - -### 7.3 Create or update supporting artefacts - -- step concepts for both workflows -- context-key concepts as needed for new mappings -- input-mapping concepts -- tool-output-to-context mapping concepts -- prompt/content text relations when a step truly needs prompt-governed behaviour - -## 8. Routing and execution implications - -### 8.1 Conversation turns - -Target direction: - -- bare arXiv URL turns should become discoverable as an explicit arXiv workflow case rather than relying only on the generic tool-calling wrapper plus tool-requirement overrides. - -Pragmatic transition rule: - -- if full direct workflow selection is not completed in the same change, the arXiv wrapper must at minimum exist and be invocable as a first-class workflow, with a follow-on routing change making workflow discovery select it directly. - -### 8.2 Upload events - -- file-copy upload handling should invoke the canonical general paper workflow for scholarly uploads. -- no upload pathway should depend on the non-existent integration-only workflow ID after this task. - -## 9. Validation plan - -Required validation after materialisation: - -- concept existence checks: - - `#V#scholarly_paper_representation_workflow` - - `#V#arxiv_paper_representation_workflow` -- process-graph/load validation: - - `build_workflow_process_graph(...)` - - `load_workflow_definition_from_vontology(...)` - - `validate_workflow_definition_contract(...)` -- targeted behavioural tests: - - `tests/backend/test_von_generate_bare_arxiv_url_integration.py` - - `tests/backend/test_von_file_upload_scholarly_integration.py` - - `tests/backend/test_turn_execution_required_effects_contract.py` - - relevant workflow authority/publication tests if publication helpers are touched - -## 10. Jira step plan - -Step 1. Audit current implementation and manual drift. -Step 2. Write and attach/update this design baseline. -Step 3. Repair the general scholarly paper workflow in Vontology. -Step 4. Create the explicit arXiv wrapper workflow in Vontology. -Step 5. Align upload routing and any workflow-discovery/routing hooks. -Step 6. Validate with targeted tests and Vontology inspection. -Step 7. Update Jira with per-step notes and close out only after workflow materialisation is real. - -## 11. Immediate implementation decisions from this plan - -- Preserve `#V#scholarly_paper_representation_workflow` as the canonical general-paper ID, but replace its graph. -- Create `#V#arxiv_paper_representation_workflow` as a new explicit wrapper. -- Treat the current scholarly workflow as legacy/incomplete, not authoritative. -- Treat the upload default `#V#integration_scholarly_paper_representation_workflow` as technical debt to eliminate in this task. diff --git a/docs/engineering/jvnautosci_150_publication_topic_salience_use_case.md b/docs/engineering/jvnautosci_150_publication_topic_salience_use_case.md deleted file mode 100644 index aff121fbf..000000000 --- a/docs/engineering/jvnautosci_150_publication_topic_salience_use_case.md +++ /dev/null @@ -1,387 +0,0 @@ -# JVNAUTOSCI-150 Publication Topic Salience Use Case - -Status: implementation artefact for `JVNAUTOSCI-150` -Date: 2026-04-25 - -## 1. Purpose - -This document defines the use case, technical specification, sample data, and -evaluation criteria for describing which research topics a SAIL publication is -salient to. - -The task is interpreted as a blueprint and design artefact. It does not add a -new Python-side topic scorer. Current Von already has a scholarly-paper -representation substrate, so the next durable capability should be expressed as -a Vontology/workflow-authoritative salience workflow with Python limited to -support surfaces such as parsing, validation, retrieval, telemetry, and report -materialisation. - -## 2. Current State and Staleness Review - -The original Jira wording predates the current paper workflow family. The live -system now has: - -- canonical workflow IDs `#V#scholarly_paper_representation_workflow` and - `#V#arxiv_paper_representation_workflow` in - `src/backend/services/paper_representation_workflow_vontology_service.py:23`. -- a repo seed bootstrap fixture for those workflows in - `src/backend/workflows/repo_seed_bundles/paper_representation_workflow_seed_bundle.json`. -- current metadata-topic handling in - `src/backend/services/arxiv_paper_link_service.py:216`, `:304`, and `:925`. - This extracts topic/category labels from metadata, resolves or creates - `#V#research_topic` instances, and links the paper with `#V#about`. -- workflow completion reporting with `topic_concept_ids` in - `src/backend/workflows/execution_contracts.py:387`. -- acceptance support for arXiv paper representation and topic links in - `tests/backend/test_paper_representation_workflow_vontology_service.py:200`. - -That existing path is useful substrate, but it is not content-level topic -salience. It records externally supplied topic/category metadata and a bounded -number of topic links. The missing capability is an evidence-grounded judgement -about how central each topic is to the paper, with ontology mapping, -confidence, explanation, and evaluation. - -## 3. Literature Note - -This task is research-sensitive because it defines future topic analysis and -evaluation behaviour. The implementation design was checked against recent -topic-modelling work: - -- BERTopic combines transformer embeddings, clustering, and class-based TF-IDF - to produce coherent topic representations. This is a useful non-authoritative - discovery baseline, not the durable Von policy surface. - Source: -- TopicGPT shows that prompt-based topic modelling can produce interpretable - natural-language topics and allows user constraints without retraining. - Source: -- Recent evaluations of LLMs for topic modelling report coherent and diverse - topics with few hallucinations, but also risks such as shortcutting on only - parts of a document and limited controllability. - Source: -- LLM-based topic-modelling work suggests LLMs can generate relevant topic - titles and support human-guided refinement/merging, but this should be - evaluated rather than assumed. - Source: -- LLM-assisted topic-model evaluation can cover coherence, repetitiveness, - diversity, and topic-document alignment for evolving scientific taxonomies, - but it should complement rather than replace expert review. - Source: - -Design consequence: Von should not implement topic salience as keyword -frequency, token overlap, or a static Python scoring table. Use deterministic -or embedding methods as bounded support signals, then keep the salience -judgement, ontology mapping policy, and explanation policy in workflow, -prompt, and Vontology artefacts with telemetry-visible evidence. - -## 4. Use Case - -### 4.1 Scenario - -A SAIL researcher uploads a draft paper or asks Von to analyse an arXiv paper -that the lab is considering for a project. The researcher wants to know which -SAIL-relevant research topics the publication is centrally about, which topics -are only secondary, and why Von thinks so. - -Example request: - -```text -Analyse this SAIL paper and describe the research topics it is salient to. -Rank the topics, map them to Vontology where possible, and explain the -evidence. -``` - -### 4.2 Actors - -- `Researcher`: requests the analysis and may correct topic assignments. -- `Von`: represents the paper, retrieves evidence, maps topics to Vontology, - and produces a topic-salience report. -- `Reviewer or project lead`: uses the report to triage papers, update project - knowledge, or seed recommendation profiles. - -### 4.3 Preconditions - -- The paper is available as a file copy, arXiv source, metadata record, or - represented scholarly-paper concept. -- The caller has access to the paper under the active namespace and - organisation scope. -- The relevant paper-representation workflow can produce or reuse a - `paper_concept_id`. -- Existing candidate topic concepts are resolved from Vontology before any new - topic concept is proposed. - -### 4.4 Happy Path - -1. Von invokes or reuses `#V#scholarly_paper_representation_workflow` or - `#V#arxiv_paper_representation_workflow`. -2. The topic-salience workflow gathers paper evidence: - title, abstract, metadata topics, represented sections or chunks, citation - context if available, author-supplied keywords, and existing `#V#about` - relations. -3. Candidate topics are assembled from Vontology concepts, metadata labels, - paper content, and optional corpus-level discovery support. -4. The LLM salience step evaluates each candidate topic against the paper - evidence and assigns: - `primary`, `secondary`, `method`, `application`, `background`, or - `rejected`. -5. The workflow resolves topics to existing Vontology concepts where possible. - Proposed new concepts are emitted as review candidates rather than silently - becoming durable ontology authority. -6. The workflow materialises a topic-salience report as a Vontology text - relation on the paper concept and, where approved, relation assertions - linking the paper to salient topics. -7. Von answers the user with the ranked topics, evidence-backed rationale, - uncertainty, and any review-needed topic candidates. - -### 4.5 User-Visible Output - -The answer should be topic-first, not execution-bookkeeping-first: - -- a ranked topic list with salience roles and concise explanations; -- evidence snippets or section references for each accepted topic; -- mapped Vontology concept IDs where available; -- uncertainty notes where the paper text is thin or the mapping is ambiguous; -- review prompts only for genuinely blocking ontology choices. - -### 4.6 Failure and Review Paths - -- If no paper concept can be represented, the workflow fails closed and asks for - a usable paper source. -- If the paper content is unavailable and only metadata exists, the report is - marked metadata-only and cannot claim full-paper salience. -- If candidate topics cannot be mapped confidently to Vontology, they remain - proposed topic labels with provenance rather than being silently asserted. -- If the topic list is generated by a corpus baseline only, the report must - mark those topics as baseline candidates, not authoritative salience. - -## 5. Authoritative Artefacts - -Before implementation, resolve existing Vontology concepts and predicates. Do -not create these blindly. - -Existing artefacts to reuse: - -- workflow: `#V#scholarly_paper_representation_workflow` -- workflow: `#V#arxiv_paper_representation_workflow` -- type: `#V#research_topic` -- predicate: `#V#about` -- text predicate: `#V#has_topic_labels` -- recommendation workflow: `#V#paper_recommendation_evaluation_workflow` -- recommendation prompt: `#V#paper_recommendation_reranker_prompt` - -Candidate artefacts for the future salience implementation: - -- workflow: `#V#publication_topic_salience_analysis_workflow` -- prompt: `#V#publication_topic_salience_analysis_prompt` -- prompt: `#V#topic_salience_explanation_prompt` -- text predicate: `#V#has_topic_salience_report_json` -- relation predicate: `#V#has_salient_topic` -- type: `#V#topic_salience_assessment` - -Authority split: - -- Workflows decide when salience analysis runs and which evidence is required. -- Prompt/Vontology artefacts carry the judgement and explanation policy. -- Vontology carries durable topic concepts, salience reports, provenance, and - review state. -- Python supplies support-only surfaces: text extraction, chunk lookup, - candidate retrieval, JSON schema validation, telemetry, and safe - materialisation. - -## 6. Proposed Workflow Shape - -Workflow ID: - -- `#V#publication_topic_salience_analysis_workflow` - -Inputs: - -- `paper_concept_id` -- `file_copy_concept_id` or `source_uri` -- `user_concept_id` -- `organisation_concept_id` -- optional `candidate_topic_concept_ids` -- optional `analysis_profile` - -Outputs: - -- `topic_salience_report_json` -- `salient_topic_concept_ids` -- `proposed_topic_labels` -- `unsupported_topic_candidates` -- `evidence_span_ids` -- `confidence_summary` -- `review_required` - -Proposed step graph: - -1. `ensure_paper_representation` - - invokes the existing scholarly or arXiv paper workflow when needed. -2. `gather_paper_evidence` - - retrieves paper text, metadata, represented chunks, and existing topic - assertions under namespace controls. -3. `assemble_candidate_topics` - - gathers candidate concepts from Vontology, metadata labels, corpus support - methods, and optional user-supplied topics. -4. `judge_topic_salience` - - LLM-mode step using the authoritative salience prompt and the gathered - evidence. It must cite evidence spans for every accepted topic. -5. `resolve_topic_concepts` - - deterministic support step that maps labels to existing Vontology topics - and emits unresolved labels for review. -6. `validate_salience_report` - - schema, provenance, confidence, and unsupported-claim checks. -7. `materialise_salience_report` - - writes `#V#has_topic_salience_report_json` and approved topic relations. -8. `compose_user_report` - - answer-first rendering of ranked topics and evidence. -9. `completed` or `failed`. - -## 7. Report Contract - -The salience report should use a versioned JSON contract. - -```json -{ - "schema_version": "paper_topic_salience_report.v1", - "paper_concept_id": "#V#paper_example", - "analysis_basis": "full_text", - "topics": [ - { - "rank": 1, - "topic_label": "Neuro-symbolic workflow control", - "topic_concept_id": "#V#research_topic_neuro_symbolic_workflow_control", - "salience_score": 0.92, - "salience_role": "primary", - "confidence": "high", - "evidence": [ - { - "span_id": "abstract:1", - "quote": "The paper proposes a Vontology-authored workflow layer..." - } - ], - "rationale": "The problem statement, method, and evaluation all centre on represented workflow control." - } - ], - "proposed_topics": [], - "unsupported_candidates": [], - "review_required": false -} -``` - -Scoring guidance: - -- `salience_score` is a calibrated judgement output, not a frequency score. -- The score must be paired with a categorical `salience_role`, confidence, and - evidence. -- A topic is `primary` only when it is central to the paper's problem, method, - contribution, or evaluated claim. -- Broad adjacency such as "AI" or "machine learning" should be down-ranked - unless the paper itself makes that broad topic central. - -## 8. Sample Data - -The synthetic fixture in -`docs/engineering/jvnautosci_150_topic_salience_sample.json` provides a -complete sample request and expected report. It deliberately uses a synthetic -SAIL-style paper so the fixture does not invent claims about a real publication. - -Expected top topics in the sample: - -- neuro-symbolic workflow control; -- Vontology and knowledge representation; -- agent evaluation and telemetry; -- research-team assistance; -- paper recommendation is rejected as unsupported because it appears only as a - downstream use case, not as a central contribution. - -## 9. Evaluation Criteria - -### 9.1 Benchmark Set - -Create a small expert-reviewed benchmark first: - -- 20 to 40 SAIL or SAIL-adjacent papers with permitted text access. -- Each paper has expert-labelled primary and secondary topics. -- Each label is mapped to an existing Vontology topic where possible. -- Ambiguous or missing topics are recorded as review candidates. -- At least five examples should have sparse metadata so the system must use - content evidence rather than arXiv categories alone. - -### 9.2 Baselines - -Compare against: - -- current metadata-only topic extraction from arXiv categories and - `#V#has_topic_labels`; -- embedding or BERTopic-style clustering over abstracts/full text; -- LLM-only topic generation without Vontology candidate grounding; -- the workflow-authoritative salience design in this document. - -Baselines are diagnostic. They must not become hidden production authority. - -### 9.3 Metrics - -Use a mixed evaluation: - -- top-k topic recall against expert primary and secondary labels; -- nDCG or MAP over ranked topics; -- ontology mapping accuracy; -- unsupported topic rate; -- explanation grounding rate, measured by whether each accepted topic has a - valid evidence span; -- reviewer edit rate for topic labels, ranks, and concept mappings; -- cost and latency per paper; -- failure visibility, including correct metadata-only caveats. - -### 9.4 Real-Path Acceptance - -Future implementation should include: - -- gateway-backed tests for the materialisation tool path, not only direct - handler tests; -- a workflow execution test through `WorkflowExecutor` with a mock paper - fixture; -- an env-gated live acceptance lane against a small public arXiv sample, similar - to the existing arXiv paper representation lane; -- telemetry assertions that record source basis, prompt version, model, - candidate-topic provenance, evidence spans, and materialised report IDs. - -### 9.5 Human Review - -Topic salience is not complete without human correction loops. Review UI or -MCP surfaces should allow: - -- accept mapped topic; -- remap to existing topic; -- propose a new topic concept; -- demote or reject a topic; -- edit rationale; -- mark an evidence span as insufficient. - -Those corrections should be preserved as evaluation data and, where justified, -promoted into durable Vontology knowledge. - -## 10. Security and Access Notes - -- Respect namespace and organisation scope when retrieving paper text, - metadata, and prior recommendations. -- Do not expose private uploaded-paper text in telemetry, Jira, logs, or sample - artefacts. -- Treat external paper text as untrusted content. It may provide evidence for - the salience judgement, but it must not control tool authority or workflow - policy. -- Destructive ontology changes are out of scope. Proposed new topics require - review before durable mutation. - -## 11. Closure Criteria for JVNAUTOSCI-150 - -This task is complete when the repository contains: - -- a use-case walkthrough; -- a technical design that fits the current paper workflow family; -- a sample input/output artefact; -- evaluation criteria and future acceptance guidance; -- explicit authority boundaries showing that durable salience policy belongs - in workflow, prompt, KB, and Vontology artefacts rather than Python ranking - code. - diff --git a/docs/engineering/jvnautosci_1554_paper_recommender_rescope.md b/docs/engineering/jvnautosci_1554_paper_recommender_rescope.md deleted file mode 100644 index 6a88d520f..000000000 --- a/docs/engineering/jvnautosci_1554_paper_recommender_rescope.md +++ /dev/null @@ -1,317 +0,0 @@ -# JVNAUTOSCI-1554 Paper Recommender Re-scope - -Status: closure note and backlog redirect -Last updated: 2026-04-04 (Pacific/Auckland) -Task: `JVNAUTOSCI-1554` - -## 1. Purpose - -`JVNAUTOSCI-1554` exists to reinterpret the old paper recommender work as a -modern workflow-first Von capability rather than a restoration of the earlier -Slack prototype. - -This note records: - -- what currently exists in code and Vontology, -- which legacy `JVNAUTOSCI-1` subtasks are obsolete, historical, or still - conceptually relevant, -- and how the active implementation path under `JVNAUTOSCI-1553` was - re-scoped. - -It now serves primarily as closure evidence for `JVNAUTOSCI-1554`. The live -follow-on planning for the next open `JVNAUTOSCI-1553` step should be read from: - -- `docs/engineering/jvnautosci_1654_open_epic_review.md` -- `docs/engineering/jvnautosci_1676_recommendation_tool_vs_workflow.md` - -## 2. Audited current state - -### 2.1 What exists now - -The current Von codebase already has a real scholarly-paper representation -substrate: - -- canonical workflow concepts for: - - `#V#scholarly_paper_representation_workflow` - - `#V#arxiv_paper_representation_workflow` -- paper acquisition and representation tools: - - `download_paper` - - `get_paper_metadata` - - `materialise_scholarly_representation_for_file_copy` -- paper/arXiv workflow publication/bootstrap support in - `paper_representation_workflow_vontology_service.py` -- targeted paper/arXiv workflow tests and gateway tests across the internal MCP - path - -`JVNAUTOSCI-1415` is already `Done`, and its closure evidence states that the -explicit paper and arXiv workflows were materialised and published to -Vontology. - -### 2.2 What does not exist now - -No current repo evidence supports the idea that Von still has an active Slack -paper recommender implementation as a first-class product surface. There are -small historical or unrelated Slack references, but no active paper -recommendation path built around Slack app delivery. - -So the right interpretation is: - -- the old Slack recommender is historical, -- the current substrate is scholarly-paper ingestion and representation, -- and the missing capability is workflow-first recommendation on top of that - substrate. - -### 2.3 Important phase-gating constraint - -The broader `JVNAUTOSCI-1657` Phase 2 review concluded that the vertical slice -should not expand as if the context-bundle, dossier, provenance, and event-model -substrate were already complete. - -That means the paper recommender should be advanced as a bounded modernisation -plan, but not treated as ready for broad end-to-end rollout until the relevant -substrate tasks land, especially: - -- `JVNAUTOSCI-254` -- `JVNAUTOSCI-1563` -- `JVNAUTOSCI-1564` -- `JVNAUTOSCI-531` -- `JVNAUTOSCI-969` -- `JVNAUTOSCI-970` -- `JVNAUTOSCI-971` -- `JVNAUTOSCI-972` - -## 3. Legacy task interpretation - -### 3.1 Parent task `JVNAUTOSCI-1` - -`JVNAUTOSCI-1` is still useful as history, but its description is explicitly -about a Slack-integrated matcher. It should therefore be treated as a legacy -prototype umbrella, not as the literal implementation brief for the modern Von -capability. - -### 3.2 Legacy subtasks: mapping - -The following mapping is the recommended interpretation of the historical -`JVNAUTOSCI-1` subtasks. - -#### Historical input only - -These are useful as research or prototype history, but should not drive current -implementation literally: - -- `JVNAUTOSCI-2` survey models/tools for paper matching -- `JVNAUTOSCI-5` survey models/tools for paper matching - Alex -- `JVNAUTOSCI-8` curate an initial dataset of papers posted on Slack and project descriptions -- `JVNAUTOSCI-9` collect information on matching accuracy and preferences -- `JVNAUTOSCI-10` first prototype using OpenAI API and Slack app - -#### Conceptually relevant, but superseded by modern workflow-first tasks - -These describe needs that still matter, but their old wording should not be -implemented literally: - -- `JVNAUTOSCI-3` explanation/highlighting methods -- `JVNAUTOSCI-4` survey models/tools for paper matching - Yang -- `JVNAUTOSCI-7` survey explanation methods - Yang -- `JVNAUTOSCI-20` recommend papers after project-description update -- `JVNAUTOSCI-44` extract paper information from PDF files shared in a channel -- `JVNAUTOSCI-45` modify prompts for performance -- `JVNAUTOSCI-46` allow multiple projects per user and modify UI -- `JVNAUTOSCI-47` separate feedback for decision and explanation -- `JVNAUTOSCI-51` make it visually easier to tell which paper is recommended - -These should now be covered by the active `JVNAUTOSCI-1553` follow-on tasks, -not by reviving the original prototype structure. - -#### Obsolete or prototype-specific - -These are tied to the old operational surface and should be treated as obsolete -for the modern implementation path: - -- `JVNAUTOSCI-52` Slack-native URL extraction -- `JVNAUTOSCI-56` `/R2D2-help` private-channel command -- `JVNAUTOSCI-62` duplicate deployment/development copy of the whole system - -#### Generic prototype engineering work already absorbed historically - -Several other old subtasks were generic prototype engineering or packaging work -and are best treated as absorbed historical scaffolding rather than live -backlog: - -- database/persistence tidy-up tasks -- logging/style/docstring tasks -- package/readme/deployment-structure tasks -- prompt wording and dev-only notification tweaks - -## 4. Modern target capability - -The modern paper recommender should be treated as a workflow-first knowledge and -recommendation capability with these layers. - -### 4.1 Paper intake and representation - -This layer already has meaningful coverage: - -- ingest or fetch candidate papers, -- represent them as scholarly-paper concepts, -- persist provenance and file-copy/source linkage, -- expose inspectable workflow and tool telemetry. - -This is the substrate created by the current paper/arXiv workflow family. - -### 4.2 Researcher recommendation profiles - -Von needs a modern way to represent what makes a paper relevant to a person, -team, or project. That should not be a single opaque text box glued to a Slack -profile. It should become a Vontology-native recommendation profile that can be: - -- user-visible and editable, -- maintained over time by Von, -- grounded in project descriptions, uploaded artefacts, and already-represented - papers where appropriate, -- and explained back to the user. - -### 4.3 Candidate discovery - -Candidate papers can come from several sources: - -- explicit paper/arXiv ingestion -- future feed or source integrations -- optional external signals such as Papers with Code or SciSciNet - -External sources are enrichments, not the core recommender definition. - -### 4.4 Ranking and explanation - -This is the core missing capability: - -- compare represented papers against represented researcher/project context, -- score or rank candidates, -- emit concise explanation grounded in KB evidence and provenance, -- and fail visibly when the substrate is too incomplete to justify a strong - recommendation. - -### 4.5 Delivery and review surfaces - -Recommendation delivery should be separated from the ranking core. Slack may be -one future surface, but it should not again become the authoritative shape of -the capability. - -Preferred first surfaces are: - -- a Von/web review surface for recommendations, -- triggered recommendations after profile updates or candidate-paper ingestion, -- inspectable recommendation provenance and rationale. - -### 4.6 Feedback and evaluation - -The system should capture graded user feedback on: - -- whether the recommendation itself was useful, -- whether the explanation was useful, -- and any free-text correction that helps future ranking/evaluation. - -## 5. Post-rescope backlog snapshot under `JVNAUTOSCI-1553` - -### 5.1 What landed from the re-scope - -#### `JVNAUTOSCI-117` - -This is now `Done` and established the modern researcher recommendation profile -surface: - -- editable project/research-interest representation, -- stronger specificity than the old free-text description, -- eventual Von-assisted maintenance with conservative confirmation rules. - -#### `JVNAUTOSCI-120` - -This is now `Done` and should be read as the first workflow-first ranking and -explanation step on the modern substrate rather than as a request to restore the -old Slack prototype. - -The more explicit semantic-evaluation replacement task is now -`JVNAUTOSCI-1679`, which replaced the lexical ranker with workflow-materialised -semantic evaluation. - -If Jira still shows open child work beneath `JVNAUTOSCI-1679`, treat that as -the remaining detailed semantic-evaluation tranche rather than reopening -`JVNAUTOSCI-120` as the live brief. - -#### `JVNAUTOSCI-1675` - -This is now `Done` and established the first authoritative Von-native review -surface: - -- a settings-tab recommendation review path, -- inspectable rationale and provenance, -- and bounded trigger points for recommendation inspection. - -### 5.2 Current open and later follow-on work - -#### `JVNAUTOSCI-1676` - -This is the next open architecture/planning step under `JVNAUTOSCI-1553`. It -should clarify: - -- when recommendation remains a bounded reusable primitive, -- when it should be expressed as a workflow family, -- and which research-group recommendation use cases should call the primitive - versus becoming explicit workflows. - -#### `JVNAUTOSCI-116` - -This remains open (`Backlog`) and should now be read as post-review feedback -capture work: - -- implement graded recommendation feedback, -- capture explanation feedback separately from relevance feedback, -- and preserve enough signal for later evaluation, learning, or workflow - invalidation. - -#### `JVNAUTOSCI-270` - -Keep as an optional source-enrichment evaluation task for Papers with Code / -Hugging Face Papers. - -#### `JVNAUTOSCI-776` - -Keep as an optional source-enrichment evaluation task for SciSciNet. - -### 5.3 What should no longer be treated as the live backlog brief - -Do not continue to read this note as if `JVNAUTOSCI-120` were still the single -active implementation brief for the recommendation core. The April 2026 follow-on -split is now: - -- `JVNAUTOSCI-1679` for the semantic evaluation/materialisation replacement, -- `JVNAUTOSCI-1675` for delivery and review surfaces, -- `JVNAUTOSCI-1676` for the next tool-versus-workflow clarification. - -## 6. Recommended next interpretation order - -1. Keep paper/arXiv representation as the authoritative substrate and repair any - residual workflow-family issues there if they block downstream work. -2. Treat the landed profile, semantic evaluation, and review surfaces - (`JVNAUTOSCI-117`, `JVNAUTOSCI-1679`, `JVNAUTOSCI-1675`) as the current base - capability. -3. Clarify the bounded recommendation primitive versus workflow-family split in - `JVNAUTOSCI-1676` before adding broader research-group recommendation flows. -4. Implement feedback capture and evaluation (`JVNAUTOSCI-116`) against that - clarified workflow family. -5. Use `JVNAUTOSCI-270` and `JVNAUTOSCI-776` as optional source-enrichment - decisions rather than hard blockers. - -## 7. Closure interpretation for `JVNAUTOSCI-1554` - -`JVNAUTOSCI-1554` should count as complete when: - -- the legacy Slack-centred work is explicitly reinterpreted, -- the current paper/arXiv capability is recognised as the real substrate, -- the old subtasks are mapped to obsolete, historical, or superseded status, -- and the active `JVNAUTOSCI-1553` backlog is rewritten to express the modern - workflow-first implementation path. - -That closure condition is now satisfied. This note should therefore remain as a -historical re-scope record, while live `JVNAUTOSCI-1553` follow-on planning -shifts to the later April 2026 task split and linked notes. diff --git a/docs/engineering/jvnautosci_1587_workflow_studio_literature_review.md b/docs/engineering/jvnautosci_1587_workflow_studio_literature_review.md deleted file mode 100644 index c680e5f17..000000000 --- a/docs/engineering/jvnautosci_1587_workflow_studio_literature_review.md +++ /dev/null @@ -1,523 +0,0 @@ -# JVNAUTOSCI-1587 Workflow Studio Literature Review and Visual Design Brief - -Date: 2026-03-27 -Task: `JVNAUTOSCI-1587` -Parent epic: `JVNAUTOSCI-1586` -Status: Finalised design brief for implementation planning - -## 1. Purpose - -This note records a short literature-informed design brief for a new Workflow -Studio inside Von: an independent workflow-focused program/surface for -discovering, listing, inspecting, comparing, tracing, and eventually editing -authoritative VWL/Vontology workflows. - -This is not a repeat of the completed renderer-aligned workflow visualisation -work (`JVNAUTOSCI-1148`). That earlier work added workflow-related rendering in -existing task-centric UI paths. The Workflow Studio is a separate workflow-first -surface with its own browse/search/navigation model and its own requirements -around overview, drill-down, operational overlays, and mixed-initiative editing. - -## 2. Current Von Substrate - -The current implementation already provides a strong read-side substrate: - -- workflow listing and definition APIs in `src/backend/server/routes/workflows_routes.py` -- workflow summary assembly in `src/backend/workflows/workflow_listing_service.py` -- workflow management/introspection tool surfacing in - `docs/engineering/workflow_mcp_capability_matrix.md` -- a lightweight existing workflow monitor in - `src/frontend/web/von_interface/templates/chat_tab.html` -- a reusable diagram contract baseline in Appendix A of - `docs/engineering/von_workflow_language_manual.md` - -Session observations taken on 2026-03-27: - -- `workflow_list_definitions(limit=5)` succeeded and reported `48` workflows. -- The same payload reported: - - `48` registry workflows - - `48` Vontology-discovered workflows - - `48` graph-complete workflows - - `0` authority-missing workflows - - `30` `retrieval_ready` workflow descriptions - - `18` `stub` descriptions - - `16` under-specified workflow descriptions -- `workflow_mcp_health_check(include_introspection=true)` succeeded for: - - `workflow_list_definitions` - - `workflow_list_instances` - - `workflow_list_event_bindings` - - `workflow_list_schedules` - - `settings_get_public` - - `chat_introspect` - -Recent session history also matters. On 2026-03-26, the same health check -showed a `workflow_list_instances` timeout while definition/binding/schedule -reads still worked. So the correct design conclusion is not "the operational -surface is broken", but "operational surface health varies enough that the -Workflow Studio must treat degraded and partial states as first-class UX". - -This current state supports a staged plan: - -1. build the studio on top of existing authoritative workflow data, -2. keep diagrams and operational summaries derived, -3. make partial availability explicit, -4. postpone write-capable editing until preview, validation, and authority - contracts are designed. - -## 3. Literature Review - -### 3.1 Process-model size and decomposition - -The strongest repeated result in the BPM literature is that size and control-flow -complexity harm comprehension. Mendling, Reijers, and van der Aalst's -seven-process-model-guidelines work argues for keeping models as small as -possible, using structured modelling, and decomposing once models become too -large or too branch-heavy. A useful working threshold from this line of work is -that models above roughly fifty elements should not be presented as the default -single view. - -The earlier "What makes process models understandable?" study reinforces the -same direction: size, density, label quality, structuredness, and reader -experience all materially affect comprehension. The practical implication for Von -is that the Workflow Studio should not optimise for "show me the whole graph"; -it should optimise for "help me answer my current workflow question with the -minimum necessary visual load". - -### 3.2 Layout, attention, and flow consistency - -Recent work strengthens the case that layout is not cosmetic. Burattin et al. -showed that flow consistency can be measured and treated as a real model-quality -signal. Figl et al. (2024) pushed this further with eye-tracking evidence: -consistent flow direction and visible control-flow patterns help users know where -to look next and make it easier to detect sequential, parallel, and exclusive -blocks. - -This matters directly for the Workflow Studio. The diagram engine should not -allow arbitrary free-form layouts as the primary view. The default layout should -be stable, directional, and optimised for control-flow interpretation. Layout -determinism is part of comprehension and diffability. - -### 3.3 Hierarchy helps, but hidden detail can hurt - -There is no simple "collapsed sub-processes are always good" conclusion. The -literature is more nuanced. - -- Hierarchy and modularisation help when the alternative is an overwhelming flat - model. -- But collapsed sub-processes can hide information that is necessary for the - user's task, forcing mentally expensive context-switching. -- Turetken et al. (2020) found that modularisation changes understandability in - task-dependent ways, and that users often benefit when expanded detail can be - consulted directly rather than inferred from collapsed placeholders alone. - -The correct design move for Von is therefore not "always collapse" or "always -flatten". The studio should use overview-plus-detail and semantic zoom: - -- default to a manageable overview, -- allow drill-in, -- preserve breadcrumbs and local context, -- provide lightweight inline previews for collapsed structures, -- avoid sending users to a completely separate context just to inspect one hidden - branch. - -### 3.4 Visual notation quality matters independently of workflow semantics - -Moody's Physics of Notations remains the best concise design theory for the -visual side of the problem. The most relevant principles here are: - -- semantic transparency -- perceptual discriminability -- complexity management -- cognitive integration -- dual coding -- graphic economy - -For the Workflow Studio this means: - -- different node/edge types should look meaningfully different -- colour should not be the only channel -- labels must carry semantic weight, not just IDs -- users should be able to integrate views without memorising unstable visual - conventions -- the notation should remain visually economical instead of encoding too many - categories at once - -### 3.5 Interaction is part of comprehension - -The literature on process visualisation and broader visual analytics points in -the same direction: for long, branching, or operationally rich workflows, a -static single canvas is not enough. Brath et al. (2024) describe practical -lessons from real-world journey analytics deployments that map well onto -workflow inspection: - -- aggregation and grouping are necessary -- filtering and focus are necessary -- multiple coordinated views are better than one overloaded view -- operational overlays and narrative summaries help users interpret complex - event-driven structures - -This aligns with the existing Von substrate, where the real workflow story is -already split across topology, schedules, bindings, traces, and health signals. -The Workflow Studio should treat multi-view coordination as the baseline model, -not as a later enhancement. - -### 3.6 LLM assistance should be mixed-initiative, not opaque - -The recent LLM + visual analytics literature is useful here, but it does not -justify direct AI-owned editing. - -LEVA (2024) and Li et al. (2024) suggest a better pattern: - -- let the LLM help with explanation, onboarding, summarisation, search intent, - and candidate action formulation -- keep the visual state explicit and inspectable -- keep the user's selection/context explicit -- make proposed operations reviewable before execution - -In other words, the LLM should operate as interpreter, coordinator, and -suggestion engine around the workflow view, not as a silent direct mutator of -the authoritative workflow graph. - -## 4. What the Literature Supports Strongly - -These conclusions are strongly supported by multiple sources and should be -treated as design constraints rather than optional style choices. - -### 4.1 The studio needs multiple coordinated views - -One workflow diagram cannot carry all of these simultaneously without becoming -hard to read: - -- topology -- control-flow logic -- dataflow/context mapping -- schedules -- event bindings -- durable runtime state -- execution traces -- task/Jira relationships -- health/parity/authority diagnostics - -The Workflow Studio should therefore expose a coordinated view family rather than -one "master" diagram. - -### 4.2 The default view should be cognitively light - -The default view should answer "what is this workflow?" quickly. It should not -start by showing every context key, every trace event, and every operational -overlay. - -Recommended default: - -- workflow catalogue on the left -- topology view as the primary central view -- metadata/authority panel on the right -- optional operational overlays layered in rather than baked into the default - -### 4.3 Layout must be stable and directional - -For the default topology view: - -- use a stable left-to-right or top-to-bottom flow direction -- minimise unnecessary directional reversals -- expose parallel/exclusive patterns clearly -- keep deterministic layout ordering for diffability and repeatability - -### 4.4 Hierarchy must preserve context - -The studio should support hierarchy, but not by hiding everything behind opaque -"plus" boxes. Users should be able to move between overview and detail while -preserving: - -- location in the parent workflow -- breadcrumb context -- inbound/outbound transition meaning -- key metadata about the collapsed region - -### 4.5 Search, filtering, and explanation are first-class - -Users will ask different questions: - -- Which workflows exist? -- Which ones are healthy? -- Which ones are background-triggered? -- Why did this workflow run? -- What data keys does this step write? -- What changed between two workflows? - -Those are not all diagram problems. The studio must combine search, filtering, -and explanatory summaries with diagrammatic views. - -## 5. What Remains Design Judgement - -The literature does not settle these questions fully; they require local design -judgement. - -### 5.1 Whether the studio lives inside the main Von shell or as a more isolated app - -The literature supports independence of task context, but it does not dictate -whether the program should be: - -- a separate route in the main web app, -- a more isolated expert-mode page, -- or a dedicated workspace shell. - -My current recommendation is a separate route inside the existing web app so it -can reuse auth, navigation, and styling infrastructure without being trapped -inside chat-specific UI assumptions. - -### 5.2 Which diagram library to use - -The literature supports stable directional layouts and semantic differentiation, -not a specific library. The decision between SVG-based custom rendering, -Graphviz-derived layout, ELK, or a React flow canvas should be made in -`JVNAUTOSCI-1588` and `JVNAUTOSCI-1590` against Von-specific needs: - -- deterministic layout -- large-graph performance -- semantic zoom -- overlays -- diffability -- edit preview support - -### 5.3 Whether topology or task-oriented narrative should dominate the first render - -The evidence favours cognitively light defaults, but the exact first render -could be: - -- workflow summary card + simplified topology -- topology first -- or task-oriented "why/when/how" summary first - -My current recommendation is topology-first with a short generated/curated -summary panel beside it. - -## 6. Workflow Studio Design Brief for Von - -### 6.1 Authoritative vs derived artefacts - -The Workflow Studio must keep the following distinction rigid: - -- authoritative: - - Vontology workflow concepts and relations - - workflow prompt/routing/publication metadata - - generic workflow-authoring/runtime surfaces -- derived: - - diagram payloads - - layout coordinates - - overlay summaries - - filtered catalogue projections - - comparison summaries - -The studio must never make repo-side workflow files the effective source of -truth. - -### 6.2 Recommended initial task taxonomy - -The studio should explicitly support these user tasks: - -- discover: find workflows by name, role, executability, quality, trigger type -- inspect: understand one workflow's purpose and structure -- compare: compare two workflows or two variants -- trace: connect a workflow definition to schedules, bindings, runs, and traces -- debug: find why a workflow is missing, unhealthy, incomplete, or stale -- edit-preview: formulate and preview a change before any mutation happens - -### 6.3 Recommended initial view family - -Phase-1 read-only view family: - -- Catalogue view - - searchable workflow list - - badges for description quality, authority, background launch, health -- Topology view - - steps and transitions only - - stable directional layout -- Metadata view - - workflow description, source, definition identity, authority/parity signals -- Decision view - - conditional transitions, priority order, failure branches -- Dataflow view - - context keys, tool param mappings, output-field writes -- Operational view - - schedules, event bindings, recent runs, health diagnostics - -Phase-2 additions: - -- comparison view -- execution trace playback -- route-map / typed-subworkflow view -- task/Jira linkage view - -### 6.4 Recommended default page layout - -- left rail: catalogue, search, filters -- centre: selected diagram view -- right rail: details, summary, diagnostics, related artefacts -- top bar: view selector, scope filters, compare mode, assistant entrypoint - -This layout matches the literature's support for overview+detail and coordinated -views, while keeping the diagram central. - -### 6.5 Recommended treatment of hierarchy - -- default to one workflow level at a time -- show collapsed sub-workflows as drillable nodes -- expose a lightweight preview on hover/select -- keep breadcrumbs and parent-context strips visible -- offer an "expand locally" mode for small subgraphs rather than forcing full - navigation away - -### 6.6 Recommended treatment of operational overlays - -Operational overlays should be optional layers on top of a stable structural -view, not replacements for it. - -Examples: - -- schedule badges and cadence hints -- event-binding counts and warnings -- recent run status strip -- trace availability -- health-check or parity warnings - -If an overlay is unavailable: - -- keep the structural view usable -- show the missing overlay state explicitly -- preserve the last-known timestamp if available -- avoid empty panels that look like "no data exists" - -### 6.7 Recommended role of the assistant - -The assistant should help users: - -- find the right workflow -- explain what a workflow or step does -- summarise a large workflow in plain language -- answer questions about branches, dependencies, and written context keys -- propose edits in explicit previewable form - -The assistant should not: - -- silently edit the workflow graph -- become the primary source of truth about workflow structure -- hide the concrete proposed mutation behind prose only -- bypass validation or authority checks - -## 7. Do / Do Not List - -### Do - -- keep the first screen simple -- keep diagrams derived from authoritative workflow data -- use stable directional layouts -- make hierarchy drillable but contextual -- separate structure from operational overlays -- show authority/health/degraded status explicitly -- use the assistant for explanation and edit proposal generation -- require preview and validation before any write - -### Do not - -- start with one giant "everything" graph -- rely on colour alone for semantics -- collapse important detail without local preview or breadcrumb context -- mix layout artefacts with source-of-truth workflow data -- make operational overlay failure look like workflow absence -- let the assistant mutate authoritative workflow state opaquely -- encode workflow-authority logic in repo-side files for the editor path - -## 8. Initial Evaluation Rubric - -Later implementation tasks should measure at least: - -- discovery efficiency - - can users find the intended workflow quickly? -- comprehension - - can users identify purpose, start step, major branches, and key outputs? -- navigational stability - - can users move between overview and detail without losing context? -- operational debuggability - - can users tell whether a problem is structural, operational, or authority-related? -- trust and provenance - - can users see what is authoritative vs derived? -- editing safety - - can users understand proposed edits before they are applied? - -## 9. Immediate Implications for Follow-on Tasks - -### `JVNAUTOSCI-1588` - -Must define: - -- the independent route/program boundary -- the authority contract -- the diagram-family contract -- degraded overlay semantics - -### `JVNAUTOSCI-1589` - -Should implement generic read-model transforms for: - -- topology -- decision logic -- dataflow -- operational overlays - -It should not hard-code one studio-only special-case payload if the same read -surfaces are reusable elsewhere. - -### `JVNAUTOSCI-1590` - -Should build the catalogue + topology + metadata baseline first, not jump -straight to editing or full trace playback. - -### `JVNAUTOSCI-1591` - -Should treat operational overlays as optional but well-signposted layers. -Recent session variability in health checks is enough to justify this even when -the current check is green. - -### `JVNAUTOSCI-1592` and `JVNAUTOSCI-1593` - -Should use a mixed-initiative pattern: - -- user selects context -- assistant proposes bounded change -- studio shows explicit patch/preview -- validation runs -- authoritative apply happens only after explicit confirmation - -## 10. References - -1. Jan Mendling, Hajo A. Reijers, Wil M. P. van der Aalst. "Seven process - modeling guidelines (7PMG)." 2010. - URL: https://www.vdaalst.com/publications/p574.pdf -2. Jan Mendling, Hajo A. Reijers, Jorge Cardoso. "What makes process models - understandable?" 2007. - URL: https://jorge-cardoso.github.io/rd/Papers/CP-2007-040-BPM-What-makes-process-models-understandable.pdf -3. Daniel L. Moody. "The Physics of Notations: Toward a Scientific Basis for - Constructing Visual Notations in Software Engineering." 2009. - Overview/tutorial URL: https://business.uq.edu.au/sites/default/files/events/files/daniel_moody_paper.pdf -4. Andrea Burattin et al. "Detection and Quantification of Flow Consistency in - Business Process Models." 2016. - URL: https://arxiv.org/abs/1602.02992 -5. Kathrin Figl et al. "Guiding attention in flow-based conceptual models - through consistent flow and pattern visibility." 2024. - URL: https://www.sciencedirect.com/science/article/pii/S0167923624001258 -6. Oktay Turetken et al. "The Influence of Using Collapsed Sub-processes and - Groups on the Understandability of Business Process Models." 2020. - URL: https://link.springer.com/article/10.1007/s12599-019-00577-4 -7. Hajo A. Reijers et al. "Assessing the Impact of Hierarchy on Model - Understandability." 2011. - URL: https://hreijers.win.tue.nl/H.A.%20Reijers%20Bestanden/eessmod_2011.pdf -8. Richard Brath et al. "Visual Journey Analytics: lessons learned from - real-world implementations." 2024. - URL: https://diglib.eg.org/bitstream/handle/10.2312/vipra20241105/05_vipra20241105.pdf -9. Jian Zhao et al. "LEVA: Using Large Language Models to Enhance Visual - Analytics." 2024 workshop paper. - URL: https://arxiv.org/abs/2403.05816 -10. Tong Li et al. "A Preliminary Roadmap for LLMs as Assistants in Exploring, - Analyzing, and Visualizing Knowledge Graphs." 2024. - URL: https://ieeevis.b-cdn.net/vis_2024/pdfs/w-nlviz-1021.pdf diff --git a/docs/engineering/jvnautosci_1654_open_epic_review.md b/docs/engineering/jvnautosci_1654_open_epic_review.md deleted file mode 100644 index fe1278b7d..000000000 --- a/docs/engineering/jvnautosci_1654_open_epic_review.md +++ /dev/null @@ -1,470 +0,0 @@ -# JVNAUTOSCI-1654 Open Epic Review - -Initial review: 2026-04-03 -Refreshed: 2026-04-07 -Actions executed (JVNAUTOSCI-1758): 2026-04-07 - -## Objective - -Review the currently open epics through one practical lens: - -- Which work is most likely to improve whether Von correctly understands and successfully carries out user intent end-to-end? - -This note deliberately prioritises execution correctness, knowledge-grounded action, and visible completion over local subsystem elegance. - -## Evaluation lens - -An epic scores highly if it materially improves one or more of these: - -1. Von chooses the right workflow/tool path for the user's intent. -2. Von completes the intended action rather than reporting a false success. -3. Von can retrieve, aggregate, and present the right knowledge without brittle code-side fallbacks. -4. We can measure failures directly and regress them reliably. -5. Behaviour remains KB-authoritative and workflow-first rather than drifting back into Python-side special cases. - -## Portfolio status as of 2026-04-07 - -### Completed epics (closed since initial review) - -| Key | Summary | Completed | -|-----|---------|-----------| -| JVNAUTOSCI-833 | Agentic Behaviours in Von | 2026-04-07 | -| JVNAUTOSCI-964 | Von Experimental Evaluation and Measurement | 2026-04-06 | -| JVNAUTOSCI-1553 | Paper Recommender | 2026-04-05 | -| JVNAUTOSCI-1586 | Workflow Studio | 2026-04-06 | - -These completions shift the portfolio balance. The agentic-behaviour umbrella and the evaluation infrastructure are now closed. Paper recommendation has a completed vertical slice. Workflow Studio delivered independent workflow listing, visualisation, and AI-assisted editing. The question is now: what should follow? - -### New epics (added since initial review) - -| Key | Summary | Status | Impact on intent execution | -|-----|---------|--------|---------------------------| -| JVNAUTOSCI-1667 | Von Coding Agent VS Code Extension Plugin | To Do | Low — new interaction surface, not an intent-execution bottleneck | -| JVNAUTOSCI-1714 | Paper recommendation workflow extensions | To Do | Medium — extends the completed paper-recommender vertical slice | -| JVNAUTOSCI-1752 | Type-specific instance renderers for concept pages | To Do | High — KB-driven presentation that directly improves how users recognise, orient, and act on represented knowledge | - -### Open epic inventory (22 epics, after 1758 cleanup) - -**In Progress (4):** -- JVNAUTOSCI-144 — Document Representation and Handling -- JVNAUTOSCI-167 — Continuously Running, Self-Updating Von Prototype -- JVNAUTOSCI-537 — Von Frontend UI/UX & Interaction Improvements -- JVNAUTOSCI-932 — Memory and knowledge retention -- JVNAUTOSCI-936 — Representationally rich Vontology + inference engine - -**To Do (15):** -- JVNAUTOSCI-535 — Data Model & Normalisation -- JVNAUTOSCI-536 — Observability & Performance -- JVNAUTOSCI-538 — Knowledge Discovery & Recommendation -- JVNAUTOSCI-539 — Access & Governance -- JVNAUTOSCI-541 — Onboarding & Lab Operations -- JVNAUTOSCI-857 — Production hardening -- JVNAUTOSCI-866 — Beautiful, flexible UX -- JVNAUTOSCI-885 — Multimodal/sensory I/O enablement -- JVNAUTOSCI-934 — Project admin/coordination -- JVNAUTOSCI-977 — External communications channels (Gmail, Slack) -- JVNAUTOSCI-1116 — Architectural De-bloating & Modularisation -- JVNAUTOSCI-1119 — System Reliability & Type Safety -- JVNAUTOSCI-1460 — External interaction surfaces (channels, voice, devices) -- JVNAUTOSCI-1667 — Von Coding Agent VS Code Extension -- JVNAUTOSCI-1714 — Paper recommendation workflow extensions -- JVNAUTOSCI-1752 — Type-specific instance renderers for concept pages - -**Backlog (1):** -- JVNAUTOSCI-540 — Developer Platform & Infrastructure - -**Retired/Superseded (removed from active portfolio by 1758):** -- JVNAUTOSCI-710 — Project Codebase Refactoring for Public Release → SUPERSEDED (child 725→540) -- JVNAUTOSCI-739 — Codebase transition (private to Von) → SUPERSEDED (children 740→857, 749→535) -- JVNAUTOSCI-766 — Vontology-based Tool Usage Heuristics → SUPERSEDED, 9 children reparented to 936 -- JVNAUTOSCI-1117 — UI/UX Modernisation & Onboarding → SUPERSEDED, 4 children reparented to 537 - -## Recommended next tranche (refreshed) - -With 833 and 964 now closed, the evaluation and agentic-behaviour foundations are in place. The priority shifts from "build the ability to measure" to "use measurement to drive KB substrate, recommendation quality, and hardening". - -### 1. Finish the KB substrate Von needs to act intelligently - -Primary epics: - -- `JVNAUTOSCI-932` Memory and knowledge retention *(In Progress)* -- `JVNAUTOSCI-936` Representationally rich Vontology + inference engine *(In Progress)* - -Why this is now first: - -- With evaluation infrastructure delivered (964 Done), the main bottleneck is the quality and completeness of the context available to workflows at execution time. -- Context bundles/dossiers (JVNAUTOSCI-254) remain the most directly useful in-progress work. Finishing them means workflows operate on bounded, provenance-bearing representations instead of ad hoc lookups. -- Episodic memory and artefact tasks (972, 969, 970, 971, 973) directly improve whether Von retains the right information across multi-step interactions. - -Recommended output: - -- Context bundle/dossier substrate completed and integrated into workflow execution paths. -- Episodic/artefact memory tasks landed with real call-path validation. -- Older low-leverage tasks under these epics reinterpreted or deferred. - -### 2. Extend the paper recommendation vertical slice into a full KB-aggregation proving ground - -Primary epics: - -- `JVNAUTOSCI-1714` Paper recommendation workflow extensions *(new, To Do)* -- `JVNAUTOSCI-538` Knowledge Discovery & Recommendation -- `JVNAUTOSCI-144` Document Representation and Handling *(In Progress)* - -Why this moves up: - -- Paper recommendation (1553) is now Done as a vertical slice. The follow-on epic (1714) extends it into researcher reading flows, explanation quality, and feedback loops. -- This is a natural proving ground for the KB substrate work above — it exercises retrieval, aggregation, explanation, provenance, and user feedback end-to-end. -- 538 provides the broader capability umbrella; 144 provides the document-representation backing. - -Recommended output: - -- Paper recommendation extended with workflow-first researcher reading flows. -- Explanation and provenance affordances surfaced alongside recommendations. -- Feedback loops that improve recommendation quality over time. -- Lessons from this slice generalised to other knowledge-discovery domains. - -### 3. Observability and performance as a continuous discipline - -Primary epic: - -- `JVNAUTOSCI-536` Observability & Performance - -Why this stays high: - -- With 964 (evaluation) closed, 536 inherits the operational measurement responsibility. Intent-execution improvement requires ongoing measurement of where Von succeeds and fails. -- JVNAUTOSCI-1429 (dispatch latency/visibility) should be pulled forward as a concrete deliverable. -- Selector evaluation benchmarks should be established as an ongoing regression tool, not a one-off. - -Recommended output: - -- Dashboards and SLIs for intent-completion success/failure/false-success rates. -- Dispatch-latency and visibility fixes (1429). -- Selector evaluation benchmark running as repeatable regression. - -### 4. Expose KB-driven presentation, not generic UI polish - -Primary epics: - -- `JVNAUTOSCI-1752` Type-specific instance renderers for concept pages *(new, To Do)* -- `JVNAUTOSCI-866` Beautiful, flexible UX for a knowledge-using and -creating AI system -- `JVNAUTOSCI-537` Von Frontend UI/UX & Interaction Improvements *(In Progress)* - -Why this is fourth: - -- UI work matters, but most current open UI tasks are polish/convenience oriented rather than directly improving whether intent is carried out correctly. -- The high-value UI work is the part that makes KB-driven, provenance-bearing, workflow-produced results understandable and actionable. -- Users need inspectable views of what Von knows, why it believes it, and what provenance supports the current output. -- `JVNAUTOSCI-1752` is the most concrete and immediately product-facing epic in this area. It defines type-specific summary renderers (papers, people, tasks, memories, rooms, events) driven by Vontology applicability metadata — not scattered frontend hard-coding. This is directly aligned with the intent-execution lens: users recognise, orient on, and act from what they see. - -Recommended output: - -- Type-specific instance renderers for at least several core concept families (papers, people, tasks, events, episodic memories) with furled/unfurled views. -- Renderer selection driven by Vontology type/applicability, not frontend special-cases. -- KB-defined views for high-value concept/workflow/document contexts. -- Explanation/provenance affordances for aggregated results and recommendations. -- Minimal but strong interaction loops around accepting, editing, rejecting, and refining generated/aggregated outputs. - -### 5. Harden the pathways that must fail closed - -Primary epics: - -- `JVNAUTOSCI-857` Production hardening -- `JVNAUTOSCI-539` Access & Governance - -Why this should run in parallel where feasible: - -- As Von becomes better at carrying out intent, the cost of unsafe or non-deterministic behaviour rises. -- Auth, namespace, rate-limit, and other fail-closed protections should advance in parallel with stronger execution. -- Triage 857 into reliability-critical fail-closed work versus later hardening, so it does not remain a large undifferentiated backlog. -- Key tasks to prioritise now: - - JVNAUTOSCI-1438 — Add authentication to admin endpoints - - JVNAUTOSCI-1437 — Implement rate limiting on Flask API endpoints - - Permission/visibility work: JVNAUTOSCI-871, JVNAUTOSCI-638, JVNAUTOSCI-627 - -## Epic-by-epic disposition - -### Advance now (highest priority) - -- `JVNAUTOSCI-932` Memory and knowledge retention *(In Progress)* - - High leverage because context continuity and artefact retention directly affect whether Von can complete multi-step intent reliably. - -- `JVNAUTOSCI-936` Representationally rich Vontology + inference engine *(In Progress)* - - High leverage because better query/aggregation/dossier surfaces feed both routing and execution. - -- `JVNAUTOSCI-536` Observability & Performance - - Necessary to see failure modes and latency bottlenecks in real call paths. - - Now also inherits ongoing evaluation/measurement responsibility from the closed 964 epic. - -- `JVNAUTOSCI-857` Production hardening - - Advance the fail-closed and admin/auth safety tasks that protect current behaviour. - -### Advance as a vertical slice (second priority) - -- `JVNAUTOSCI-1714` Paper recommendation workflow extensions *(new)* - - Follow-on from the completed paper-recommender slice. Extends into researcher reading flows and feedback loops. - -- `JVNAUTOSCI-538` Knowledge Discovery & Recommendation - - Broad capability umbrella. Paper recommendation is the first proving ground; generalise from there. - -- `JVNAUTOSCI-144` Document Representation and Handling *(In Progress)* - - Important as the document/paper representation substrate. - - Many open tasks are old and need sceptical reinterpretation. - -### Supporting epics; advance only where they unblock work above - -- `JVNAUTOSCI-535` Data Model & Normalisation - - Useful only insofar as it removes mixed/legacy representation pain for 932 and 936. - -- `JVNAUTOSCI-539` Access & Governance - - Important supporting safety epic; coordinate with 857 rather than driving an independent strategy. - -- `JVNAUTOSCI-1116` Architectural De-bloating & Modularisation - - Worth doing when a hotspot blocks progress, but not a primary priority axis. - -- `JVNAUTOSCI-1119` System Reliability & Type Safety - - Supports robustness. Advance as it intersects with active work. - -- `JVNAUTOSCI-167` Continuously Running, Self-Updating Von Prototype *(In Progress)* - - Background enrichment improves data quality over time. Advance where it intersects with 932/936. - -### Advance as KB-driven presentation (third–fourth priority) - -- `JVNAUTOSCI-1752` Type-specific instance renderers for concept pages *(new)* - - The most concrete KB-driven presentation epic. Defines type-specific summary renderers backed by Vontology applicability metadata. - - Directly improves user recognition, orientation, and actionability on concept pages. - - Coordinates with 866 (vision) and 537 (execution) but is more focused and deliverable. - - Subsumes the intent of 1750 and 1751, which are early local manifestations of the same idea. - -### Defer for now - -- `JVNAUTOSCI-537` Von Frontend UI/UX & Interaction Improvements - - Defer the polish-heavy items; keep only the tasks that directly help users understand or steer KB/workflow outcomes. - -- `JVNAUTOSCI-866` Beautiful, flexible UX - - Keep as the long-term design north star, but do not treat generic beauty/flexibility as the next bottleneck. - -- `JVNAUTOSCI-1117` UI/UX Modernisation & Onboarding - - Broad overlap with 537 and 866. - -- `JVNAUTOSCI-885` Multimodal/sensory I/O enablement - - Broadens channels without fixing core intent completion. - -- `JVNAUTOSCI-1460` External interaction surfaces (channels, voice, devices) - - Same reason as 885; defer until core execution is stronger. - -- `JVNAUTOSCI-977` External communications channels (Gmail, Slack) - - New interaction surfaces, not an intent-execution bottleneck. - -- `JVNAUTOSCI-541` Onboarding & Lab Operations - - Useful, but not on the critical path. - -- `JVNAUTOSCI-1667` Von Coding Agent VS Code Extension *(new)* - - Interesting new direction, but a separate interaction surface. Does not address the core intent-execution problem. - -- `JVNAUTOSCI-540` Developer Platform & Infrastructure *(Backlog)* - - Enables faster development but not user-facing. - -### Consolidate / reinterpret - -- `JVNAUTOSCI-766` Vontology-based Tool Usage Heuristics System - - The goal is valid, but the framing is now too separate from workflow-first execution. - - With 833 closed, the routing/selector work that 766 describes should be reinterpreted under the active KB substrate epics (932, 936) or a successor agentic-behaviour epic. - - Recommend: supersede or fold into 936 with an explicit KB-routing sub-scope. - -- `JVNAUTOSCI-1117` UI/UX Modernisation & Onboarding - - Likely better folded into 537 and 866. - -### Retire or explicitly supersede after quick confirmation - -- `JVNAUTOSCI-710` Project Codebase Refactoring for Public Release - - Description says the epic is no longer used and points to 739. - - Should not remain open. - -- `JVNAUTOSCI-739` Codebase transition (private to Von) - - The repo IS "Von" now. Review whether any open children still matter. - - If not, close or supersede so it stops competing for attention. - -- `JVNAUTOSCI-933` Project admin/coordination *(already SUPERSEDED)* - - Confirm fully closed in Jira. - -### Coordination only - -- `JVNAUTOSCI-934` Project admin/coordination - - Keep as the umbrella for planning/review/admin tasks like this one, not as a feature-priority destination. - -## Most important overlaps to clean up - -### 1. Routing / tool-selection overlap (766 → now orphaned from closed 833) - -- `JVNAUTOSCI-766` heavily overlapped with the now-closed `JVNAUTOSCI-833`. -- The present architecture direction should fold this into KB-authored routing metadata under `JVNAUTOSCI-936` or create a focused successor. -- **Action:** Supersede 766 by explicitly capturing its useful scope under 936. - -### 2. Recommendation overlap (538, 144, 1714) - -- `JVNAUTOSCI-538`, `JVNAUTOSCI-144`, and `JVNAUTOSCI-1714` overlap. -- With 1553 (Paper Recommender) now Done, 1714 is the natural follow-on. -- Suggested framing: - - 538 = the broad capability - - 144 = document/paper representation substrate - - 1714 = active vertical-slice extension (reading flows, feedback, explanation) - -### 3. UX overlap (537, 866, 1117, 1752) - -- Four UX/UI epics, now including 1752 (type-specific instance renderers). -- 1752 is the most concrete and KB-aligned of the four. It defines what needs to exist on concept pages — type-specific summary renderers backed by Vontology applicability. -- Suggested framing: - - 1752 = the actionable KB-driven presentation epic (advance now) - - 866 = long-term KB-defined UI vision (keep as north star) - - 537 = concrete frontend execution work (advance where it serves 1752 or other active work) - - 1117 = likely unnecessary as a separate open epic; fold into 537 - -### 4. Safety / governance overlap (857, 539) - -- Closely related. Keep both only if 857 stays focused on operational hardening and 539 on permission/governance semantics. Otherwise, use links and comments to prevent duplicate work. - -### 5. Interaction surface overlap (885, 1460, 977, 1667) - -- Four epics that expand how users interact with Von (multimodal, external channels, VS Code, devices). -- None address the core intent-execution problem. All should be deferred until the execution substrate is strong. -- Caution against fragmenting effort across multiple new surfaces before the core is reliable. - -## KB-authoritative surfaces to prioritise - -If the goal is better user-intent execution, the following should be treated as first-class authored surfaces: - -- workflow concepts, launch contracts, and executable-completeness metadata -- routing/selection metadata and repair/recovery workflow definitions -- prompt concepts for workflow generation, workflow repair, and explanation/provenance shaping -- turn execution records, critic bundles, experiment specs, and experiment runs -- context bundle / dossier representations for concepts, workflows, documents, and artefacts -- view definitions and renderer applicability metadata for KB-driven presentation -- document/paper concepts plus file-copy and provenance relations for recommendation/explanation workflows - -Supporting code surfaces should remain limited to: - -- runtime enforcement -- validation -- execution plumbing -- telemetry collection -- deterministic rendering / safety envelopes - -## Concrete suggested next tasks (refreshed) - -These are the work items to tackle next, in order, whether by reinterpreting existing tasks or by creating fresh ones. - -1. **Context bundles/dossiers plus episodic/artefact completion** - - Epic home: `JVNAUTOSCI-936` and `JVNAUTOSCI-932` - - Finish the bounded, provenance-rich context substrate Von needs before acting. - - This was tranche 2 in the initial review; promoted to first now that 833/964 are closed. - -2. **Paper recommendation extension and researcher reading flows** - - Epic home: `JVNAUTOSCI-1714` / `JVNAUTOSCI-538` / `JVNAUTOSCI-144` - - Build on the completed 1553 vertical slice with workflow-first reading flows, explanation, and feedback. - -3. **Observability SLIs and selector regression benchmarks** - - Epic home: `JVNAUTOSCI-536` - - Establish intent-completion dashboards, dispatch-latency fixes (1429), and repeatable selector benchmarks. - -4. **KB-driven presentation and type-specific instance renderers** - - Epic home: `JVNAUTOSCI-1752` / `JVNAUTOSCI-866` / `JVNAUTOSCI-537` - - Build type-specific summary renderers for concept pages (papers, people, tasks, events, rooms) driven by Vontology applicability. - - Build inspectable views of what Von knows and why it believes it. - -5. **Admin/auth/namespace hardening** - - Epic home: `JVNAUTOSCI-857` with `JVNAUTOSCI-539` - - Prioritise fail-closed pathways and remove silent privileged gaps. - -## Merge guidance from JVNAUTOSCI-1655 - -JVNAUTOSCI-1655 surfaced several useful concrete refinements. The strongest ideas have been integrated into this refreshed review: - -### Integrated into this review - -- Selector evaluation benchmark — captured as an explicit deliverable in tranche 3. -- JVNAUTOSCI-1429 dispatch-latency — pulled forward in tranche 3. -- KB aggregation and presentation gap — stated directly in tranche 4. -- Workflow generation from intent — noted as a gap; 833 is now closed so a successor home is needed. -- JVNAUTOSCI-857 triage — reflected in the hardening recommendations under tranche 5. - -### Points rejected or reframed - -- Do not keep JVNAUTOSCI-766 as a separate top-priority track. Reinterpret under 936. -- Do not over-separate measurement (536) from evaluation (now 964 is closed, 536 holds both). -- Do not pull memory-injection tasks ahead of dossier/context substrate work. -- Do not treat production hardening as mostly late work. Auth and fail-closed protections advance in parallel. - -## Short conclusion - -With the agentic-behaviour (833), evaluation (964), paper-recommender (1553), and workflow-studio (1586) epics now closed, Von has its foundational execution, measurement, and authoring infrastructure in place. - -The next portfolio centre of gravity should be: - -1. **KB substrate completion** — context bundles, episodic memory, dossiers (932, 936) -2. **Recommendation/document vertical extension** — build on paper-recommender with reading flows, explanation, feedback (1714, 538, 144) -3. **Observability as continuous discipline** — intent-completion SLIs, selector benchmarks, dispatch latency (536) -4. **KB-driven presentation** — type-specific instance renderers and inspectable, provenance-bearing views (1752, 866, 537) -5. **Fail-closed hardening** — auth, namespace, rate-limit protections (857, 539) - -The biggest planning mistake to avoid is fragmenting effort across new interaction surfaces (VS Code extension, multimodal, external channels) or generic UX modernisation before Von can reliably complete, verify, and explain the intents it already supports. - -## Actions taken (JVNAUTOSCI-1758, 2026-04-07) - -The following actions were executed per the 1758 task plan. All changes are recorded in Jira comments with rationale. - -### Step 1 — Retired stale epics - -- **710** (Codebase Refactoring for Public Release) → SUPERSEDED. Open child 725 reparented to 540. -- **739** (Codebase transition private→Von) → SUPERSEDED. Open children: 740→857, 749→535. -- **933** — confirmed already SUPERSEDED in Jira ✓ - -### Step 2 — Consolidated overlapping epics - -- **766** (Vontology-based Tool Usage Heuristics) → SUPERSEDED. 9 open children (767–774, 845) reparented to 936. -- **1117** (UI/UX Modernisation & Onboarding) → SUPERSEDED. 4 open children (1124–1127) reparented to 537. -- Issue links created: 538↔144, 538↔1714, 144↔1714 (all "Relates") - -### Step 3 — Established tranche 1 (KB substrate) - -- **254** (context bundles/dossiers): confirmed already Done ✓ -- Core memory/retention tasks confirmed under 932 with comments: 820, 968–973, 995, 1035, 1222 -- 9 misparented tasks reparented out of 932 with comments: - - 312→538, 366→936, 522→537, 527→537, 529→535, 572→535, 603→540, 624→540, 683→535 -- 936 open children confirmed correctly scoped - -### Step 4 — Established tranche 2 (paper recommendation extension) - -- All 4 existing 1714 tasks confirmed: 1, 34, 270, 776 -- **Created JVNAUTOSCI-1759** — Paper recommendation feedback capture and learning loop -- **Created JVNAUTOSCI-1760** — Evaluate and improve paper recommendation explanation quality -- 1714 now covers the full recommendation lifecycle (6 tasks): discovery → ranking → explanation → presentation → feedback → learning - -### Step 5 — Established tranche 3 (observability) - -- **1429** (dispatch latency): confirmed already Done under 536 ✓ -- **775** (MCP call logging): confirmed correctly scoped under 536 -- **Created JVNAUTOSCI-1761** — Selector evaluation benchmark for tool/action selection accuracy -- 536 now has 2 open actionable tasks: 775, 1761 - -### Step 6 — Established tranche 4 (KB-driven presentation) - -- **1752** links confirmed/added: already linked to 866, 1750, 1751; newly linked to 537 -- **1750** and **1751** subordinated as children of 1752 (were unparented) - -### Step 7 — Triaged tranche 5 (fail-closed hardening) - -- 857 has ~57 open children; triaged into two tiers: -- **RELIABILITY-CRITICAL** (5 tasks, tagged with comments): - - 1438 — Add authentication to admin endpoints - - 1437 — Implement rate limiting on Flask API endpoints - - 871 — Visibility / Permission Semantics Are Broken - - 638 — Secure user/org-specific concept access control - - 627 — Harden concept scoping relation toggles -- **Security-adjacent** (next tier): 858, 859, 835, 861 -- **Later hardening**: remaining ~48 tasks (CI, UI polish, caching, docs, infrastructure) - -### Remaining gaps - -- 857 has many children that could benefit from further reparenting to more specific epics (CI tasks to 540, UI tasks to 537, etc.) — deferred as a future cleanup pass. -- JVNAUTOSCI-1655 and JVNAUTOSCI-1657 remain open under 934; consider superseding now that 1758 has executed the concrete plan. -- Some older tasks under 144 and 538 may need sceptical reinterpretation in a future review. diff --git a/docs/engineering/jvnautosci_1657_parallel_triage.md b/docs/engineering/jvnautosci_1657_parallel_triage.md deleted file mode 100644 index 13954ea03..000000000 --- a/docs/engineering/jvnautosci_1657_parallel_triage.md +++ /dev/null @@ -1,177 +0,0 @@ -# JVNAUTOSCI-1657 Parallel Triage - -This note records the outcome of `JVNAUTOSCI-1663`: the parallel portfolio-clarity work needed so `JVNAUTOSCI-1657` can proceed without ambiguity about hardening ownership or stale-epic consolidation. - -## Outcome - -Two conclusions are now explicit: - -1. `JVNAUTOSCI-857` and `JVNAUTOSCI-539` should not be blurred together. -2. The main consolidation candidates (`766`, `1117`, `710`, `739`, `977`, `1116`, `1119`) are mostly review-and-mapping cases, not safe closure cases, because they still retain open child tasks. - -## Hardening Split - -### `JVNAUTOSCI-539` remains the governance epic - -`JVNAUTOSCI-539` is the right home for durable access/governance semantics: - -- `JVNAUTOSCI-631` Add role and aspect to user and org -- `JVNAUTOSCI-704` Implement role-based UI visibility with Vontology-driven inheritance -- `JVNAUTOSCI-901` Replace namespace-based access with explicit organisation user role binding concepts -- `JVNAUTOSCI-1036` Implement Von Super-User Administration System with MCP Tool Gating -- `JVNAUTOSCI-1168` Define authoritative effective-namespace contract (already `Done`) - -This epic should stay distinct. It is not just generic production hardening. It is the long-lived access-control and governance line. - -### `JVNAUTOSCI-857` should be operationally split - -`JVNAUTOSCI-857` is too broad to act as a single execution bucket. For planning purposes it should be treated as two tracks. - -Fail-closed / privileged-safety work to pull forward: - -- `JVNAUTOSCI-1438` Add authentication to admin endpoints -- `JVNAUTOSCI-1437` Implement rate limiting on Flask API endpoints -- `JVNAUTOSCI-871` Visibility / Permission Semantics Are Broken -- `JVNAUTOSCI-638` Secure user-specific and org-specific concept access control modifications -- `JVNAUTOSCI-627` Harden concept scoping relation toggles -- `JVNAUTOSCI-835` Prevent tests or tools from wiping prod `von_db` -- `JVNAUTOSCI-858` admin endpoints and authorisation review -- `JVNAUTOSCI-859` secrets and configuration hygiene -- `JVNAUTOSCI-860` backup policy safety -- `JVNAUTOSCI-861` audit logging for privileged operations - -Robustness / performance / operational resilience work that can remain a separate queue: - -- `JVNAUTOSCI-811` Handle OpenAI 429 TPM errors gracefully -- `JVNAUTOSCI-659` Handle large numbers of concepts -- `JVNAUTOSCI-357` low-latency caching for `von_db` -- similar non-governance operational tasks still sitting under `857` - -The planning rule is: - -- use `539` for authoritative governance model work -- use the fail-closed subset of `857` when privileged safety must advance in parallel with execution improvements -- treat the rest of `857` as robustness/performance backlog rather than as a competing governance epic - -## Consolidation Mapping - -### `JVNAUTOSCI-766` versus `JVNAUTOSCI-833` - -`766` should be reinterpreted as an early formulation of KB-authored routing/selector behaviour now better expressed under `JVNAUTOSCI-833`. - -However, `766` is not ready for closure. It still has open child tasks including: - -- `767` rule schema -- `768` Vontology concept structure for heuristics -- `769` rule loader service -- `770` orchestrator integration -- `771` admin UI -- `772` MCP tools -- `773` testing and validation -- `774` documentation and migration -- `845` persisted artefacts / virtual Vontology view - -So the correct action is: - -- keep `833` as the active home for selector/routing capability -- review each `766` child task individually for re-parenting, supersession, or closure as obsolete -- do not close `766` on the basis of a broad “already absorbed” statement - -### `JVNAUTOSCI-1117` into `JVNAUTOSCI-537` and `JVNAUTOSCI-866` - -`1117` is directionally aligned with the active UI epics, especially: - -- `JVNAUTOSCI-537` Von Frontend UI / UX & Interaction Improvements -- `JVNAUTOSCI-866` Beautiful, flexible UX for a knowledge-using and -creating AI system - -But `1117` still has open child tasks: - -- `1124` Empty State & Onboarding -- `1125` Chat Progressive Disclosure -- `1126` Visual Polish & Design Tokens -- `1127` Settings Page Simplification - -So the correct action is: - -- treat `537` and `866` as the active execution homes -- explicitly map `1124` to `1127` into those epics or close them individually as obsolete -- do not close `1117` until that child-task mapping is recorded - -### `JVNAUTOSCI-710` - -`710` is not ready for closure. It still has at least one open child task: - -- `725` Repository Cleanup - Phase 5: Re-documentation - -This means `710` should be treated as review-for-closure only. It can be closed later only after its remaining child work is either completed or explicitly superseded. - -### `JVNAUTOSCI-739` - -`739` is also not ready for closure. It still has open child tasks including: - -- `740` Login with Google feature does not work -- `749` cascading deletion cleanup - -These are substantive residual tasks, not historical bookkeeping. `739` should remain a review/mapping case until those items are explicitly moved, superseded, or completed. - -### `JVNAUTOSCI-977` versus `JVNAUTOSCI-1460` - -`1460` is the broader and more current epic for external interaction surfaces and shared gateway/runtime work. - -Future new channel/device/browser surface work should default there, not under `977`. - -But `977` is not yet safely superseded, because it still retains open child tasks including: - -- `121` congratulatory messages/posts -- `248` Google Tasks MCP integration -- `793` harden Gmail/GSuite/Slack MCP access -- `802` Gmail sending capability -- `805` calendar access -- `979` role-based access to Slack teams - -So the correct action is: - -- treat `1460` as the strategic successor epic -- review `977` child tasks one by one and either move them under `1460` / `539`, or close them as obsolete -- avoid blanket closure language until that work is done - -### `JVNAUTOSCI-1116` and `JVNAUTOSCI-1119` - -These were previously easy to overstate as “probably absorbed”, but they are not close-ready. - -`1116` still has open child tasks: - -- `235` Generalize LLM Integration Architecture with ModelManager Pattern -- `1120` Deconstruct Frontend Monoliths -- `1121` Refactor Backend Route Layers -- `1122` Consolidate Project Utilities -- `1123` Repository Housekeeping - -`1119` still has open child tasks: - -- `1132` Strict Error Handling Reform -- `1133` Type Safety Initiative -- `1134` CSS Monolith Splitting -- `1135` Global State Refactoring - -So these should be treated as review-for-consolidation umbrellas, not as epics that can already be declared finished or absorbed. - -## Portfolio Rule Going Forward - -When `JVNAUTOSCI-1657` or later planning work makes a consolidation claim, that claim should name: - -- the source epic/task -- the active destination epic/task -- whether the source item is being re-parented, superseded, narrowed, or closed as obsolete -- whether unresolved child tasks still remain - -If unresolved child tasks remain, the source epic should usually stay open until those children are explicitly dealt with. - -## Immediate Effect on `JVNAUTOSCI-1657` - -`JVNAUTOSCI-1657` can now proceed with a clearer portfolio model: - -- Phase 2 / later execution should treat `539` as governance-authority work. -- Parallel hardening should pull forward the fail-closed subset of `857`. -- Broader robustness/performance tasks under `857` can remain queued separately. -- Consolidation candidates should be handled as explicit mapping exercises, not as optimistic closures. diff --git a/docs/engineering/jvnautosci_1657_phase1_review.md b/docs/engineering/jvnautosci_1657_phase1_review.md deleted file mode 100644 index b004afe4a..000000000 --- a/docs/engineering/jvnautosci_1657_phase1_review.md +++ /dev/null @@ -1,196 +0,0 @@ -# JVNAUTOSCI-1657 Phase 1 Review - -**Status**: Review note -**Primary Jira tasks**: `JVNAUTOSCI-1660`, `JVNAUTOSCI-1657` -**Date**: 2026-04-03 - -## 1. Scope - -This note reviews the completed Phase 1 execution-correctness tranche under `JVNAUTOSCI-1657`. - -Reviewed implementation tasks: - -- `JVNAUTOSCI-965` -- `JVNAUTOSCI-1664` -- `JVNAUTOSCI-1429` -- `JVNAUTOSCI-966` -- `JVNAUTOSCI-967` - -Reviewed evidence surfaces: - -- `turn_execution_build_benchmark` -- `turn_execution_build_selector_benchmark` -- `docs/engineering/execution_correctness_benchmark_protocol.md` - -## 2. Evidence Snapshot - -### 2.1 Selector-routing benchmark - -Current selector benchmark result from the Phase 1 seed corpus: - -- `scanned_count=5` -- `selector_accuracy_pct=80.0` -- `baseline_accuracy_pct=20.0` -- `misrouting_count=1` -- one explicit abstain/no-safe-route case is present and routes correctly - -Interpretation: - -- the selector layer is real and reviewable -- it is already good enough to expose routing regressions and safe-abstention behaviour -- it is not yet broad enough to act as a universal hard gate for all workflow-selection changes - -### 2.2 Turn-execution benchmark - -Representative benchmark run over the latest 100 records in the active namespace: - -- `likely_failure_rate_pct=60.0` -- `false_success_rate_pct=0.0` -- `unresolved_follow_up_rate_pct=60.0` -- `tool_or_workflow_misrouting_rate_pct=6.0` -- `successful_completion_rate_pct=40.0` -- dominant failure mode: `mutation_not_executed` - -Important supporting signals: - -- the false-success guard passes -- retry/stop telemetry signals are absent on follow-up turns -- `selected_workflow_id` is missing on part of the corpus - -Interpretation: - -- Phase 1 has materially improved truthfulness and inspectability -- the system is now more often failing closed or surfacing unresolved work rather than silently claiming success -- the dominant remaining problem is not hidden completion fraud but incomplete execution of intended write or representation work - -That is a meaningful success for Phase 1, even though the absolute failure burden is still high. - -### 2.3 Practical meaning of the current metrics - -The current evidence says: - -1. The taxonomy is doing useful work. -2. The benchmark surfaces are exposing genuine operational bottlenecks. -3. The completion guard appears stronger than the execution substrate beneath it. - -In other words, Phase 1 has made the system easier to trust diagnostically, but not yet easy to trust operationally. - -## 3. Review Findings - -### 3.1 Taxonomy sufficiency - -The current Phase 1 label set is sufficient: - -- `successful_completion` -- `false_success` -- `unresolved_follow_up_needed` -- `tool_or_workflow_misrouting` -- `abstain_escalate_no_safe_route` - -No blocking taxonomy expansion is needed before Phase 2. - -However, Phase 2 may later justify finer sub-classification inside the existing structure, especially around: - -- KB/context insufficiency versus execution failure -- provenance or representation-contract failure versus tool failure - -Those should be treated as future refinements, not reasons to delay Phase 2. - -### 3.2 Dashboard and protocol sufficiency - -The Phase 1 dashboard and protocol stack is sufficient for: - -- engineering review -- replay-based triage -- shadow-gating of substantial routing, completion-path, and observability changes - -It is not yet sufficient for: - -- a universal hard merge gate across all relevant changes -- broad release-governance decisions without reviewer judgement - -The main reasons are corpus size, missing routing metadata in part of the measured corpus, and incomplete retry/stop telemetry. - -### 3.3 Main remaining bottleneck - -The dominant observed failure is unresolved intended work, especially `mutation_not_executed`, not false success. - -That shifts the next-tranche emphasis in a useful way: - -- Phase 1 should not continue as a pure measurement tranche -- Phase 2 should proceed, because better KB/context/provenance substrate is now likely to improve the newly visible failure class - -## 4. Decision - -### 4.1 Can Phase 2 proceed? - -Yes. - -Phase 2 should proceed now via `JVNAUTOSCI-1661`. - -The current evidence is strong enough to justify moving on from Phase 1 implementation to Phase 2 substrate work, with one important constraint: - -- Phase 1 benchmarking should remain active in shadow-gate mode while Phase 2 work proceeds - -### 4.2 What “shadow-gate mode” means here - -For substantial changes that affect routing, completion semantics, mutation execution, or latency attribution: - -- run the relevant Phase 1 benchmark surfaces -- review the replay cases -- record regressions explicitly in Jira - -But do not require the current measurement stack to act as a strict universal hard gate yet. - -## 5. Sequencing Update for JVNAUTOSCI-1657 - -The updated sequencing should be: - -1. Start `JVNAUTOSCI-1661` next as the main execution track. -2. Start `JVNAUTOSCI-1663` in parallel rather than leaving it as background paperwork. -3. Keep Phase 1 benchmark use active during Phase 2 in shadow-gate mode. -4. Use `JVNAUTOSCI-1662` as the next formal review gate before vertical-slice expansion. - -This is a clarification rather than a major reversal. - -The important update is that `1663` should now be treated as a parallel requirement for keeping the plan executable and the portfolio clean, not as optional later cleanup. - -## 6. Phase 2 Entry Criteria - -Phase 2 entry criteria should now be stated explicitly as: - -- the Phase 1 taxonomy, benchmark, dashboard, and protocol stack are in place -- false-success protection is strong enough to trust the measurements directionally -- the dominant remaining failure class is unresolved execution rather than hidden success-claiming -- Phase 2 work will preserve use of the Phase 1 benchmark surfaces as a shadow gate - -Phase 2 should not be blocked on: - -- universal hard-gate readiness -- large selector-corpus expansion -- perfect workflow-selection metadata coverage - -Those are useful follow-ons, but they are not good reasons to delay the KB-substrate tranche. - -## 7. Non-Blocking Follow-On Work Still Needed - -The review identifies four concrete non-blocking follow-ons that should remain visible during Phase 2: - -1. Expand the selector-routing corpus from live incidents and convert the highest-value misroutes into stronger regression cases. -2. Improve `turn_execution_record` completeness where `selected_workflow_id` is still missing. -3. Add retry/stop telemetry signals to follow-up turns so unresolved-work burden is more diagnosable. -4. Improve replay-case to Jira linkage so benchmark evidence lands in issue triage more directly. - -These should be treated as continuing measurement hardening, not as a reason to reopen Phase 1 as the main tranche. - -## 8. Conclusion - -Phase 1 should be considered complete for implementation purposes and successful for planning purposes. - -It did not solve user-intent execution by itself, but it did something strategically necessary: - -- it made routing, completion, and unresolved-work failures measurable on one stack -- it appears to have reduced or at least exposed false-success risk -- it clarified that the next limiting factor is substrate quality and execution follow-through - -That is enough to move `JVNAUTOSCI-1657` forward into Phase 2, with the updated sequencing and shadow-gate expectations above. diff --git a/docs/engineering/jvnautosci_1657_phase2_operationalisation.md b/docs/engineering/jvnautosci_1657_phase2_operationalisation.md deleted file mode 100644 index b915b24c5..000000000 --- a/docs/engineering/jvnautosci_1657_phase2_operationalisation.md +++ /dev/null @@ -1,196 +0,0 @@ -# JVNAUTOSCI-1657 Phase 2 Operationalisation - -**Status**: Operationalisation note -**Primary Jira tasks**: `JVNAUTOSCI-1661`, `JVNAUTOSCI-1657` -**Date**: 2026-04-03 - -## 1. Purpose - -This note turns Phase 2 of `JVNAUTOSCI-1657` into a concrete, non-duplicative KB-substrate task set. - -The Phase 1 review concluded that the dominant remaining failure class is unresolved execution rather than hidden false success. The next tranche should therefore improve the substrate Von uses to carry context, memory, artefact provenance, and structured representation, while keeping the Phase 1 benchmark stack active in shadow-gate mode. - -This is not a new backlog. It is a clarification of which existing tasks are the active Phase 2 spine, what each epic owns, and which work must remain gated. - -## 2. Phase 2 Backbone - -Phase 2 should be treated as two coordinated workstreams. - -### 2.1 `JVNAUTOSCI-936` owns KB context/dossier substrate and representational normalisation - -This epic should own the graph-first representational substrate that lets Von reconstruct bounded working context rather than stuffing prompts with heuristic long-range text. - -Active Phase 2 tasks in this line: - -- `JVNAUTOSCI-254` as the canonical context-bundles and reconstructed-dossiers task -- `JVNAUTOSCI-1563` for hybrid dossier/theory representation -- `JVNAUTOSCI-1564` for bounded workspace reconstruction contract and telemetry -- `JVNAUTOSCI-1565` for ablation and retained-case evaluation -- `JVNAUTOSCI-531` for unified relation/text-link normalisation - -Role of this line: - -- define the authoritative context-bundle and dossier artefact model -- make bounded reconstructed workspaces reusable across workflow families -- keep representation graph-first and promotion-aware -- reduce drift between concept/text relations, assertion state, theory-local state, and workflow-local report state - -### 2.2 `JVNAUTOSCI-932` owns episodic memory, artefact provenance/privacy, and durable retention - -This epic should own the memory and artefact substrate that stores durable evidence safely and makes it retrievable without breaking namespace, provenance, or privacy boundaries. - -Active Phase 2 tasks in this line: - -- `JVNAUTOSCI-972` as the episodic-memory event-model anchor -- `JVNAUTOSCI-969` for provenance and privacy/namespace gating -- `JVNAUTOSCI-970` for durable artefact storage abstraction and ingestion pipeline -- `JVNAUTOSCI-971` for artefact metadata extraction and ontology linking - -Already-landed anchors that should be treated as part of the substrate, not separate directions: - -- `JVNAUTOSCI-1607` for `#V#episode_critique_memory` artefacts and queryable projections -- `JVNAUTOSCI-1395` for generic remote URL artefact ingestion and `#V#computer_file_copy` registration - -Role of this line: - -- give episodic memory a first-class event structure -- make stored artefacts and memories provenance-aware and namespace-safe -- preserve durable file-copy and memory records as authoritative KB artefacts -- provide retrieval eligibility and auditability before memory injection becomes a default behaviour - -## 3. Authoritative KB Artefacts for This Tranche - -The key Phase 2 artefacts should be named explicitly and treated as authoritative KB surfaces rather than repo-local pseudo-structures. - -### 3.1 Context bundles and dossiers - -The context/dossier line should centre on the artefacts named in `JVNAUTOSCI-254`: - -- `#V#context_bundle` -- `#V#workflow_context_bundle` -- `#V#concept_context_bundle` -- `#V#context_facet` -- `#V#context_dossier` -- `#V#workflow_report_revision` -- `#V#evidence_receipt` - -These should carry: - -- inheritable graph-first context -- bounded textual synthesis where text is genuinely primary -- explicit evidence receipts and revision history -- report-state and reconstruction semantics that can survive long-horizon work without collapsing into one large prompt transcript - -### 3.2 Hybrid dossier and theory-local state - -Phase 2 should treat hybrid dossier reconstruction as mixed textual and KR state, not as text-only memory. - -The relevant authoritative artefacts and control surfaces are: - -- `#V#testing_theory` -- theory-local assertions and promotion/rollback surfaces -- mixed dossier/report state described in `JVNAUTOSCI-1563` -- bounded workspace reconstruction and telemetry described in `JVNAUTOSCI-1564` - -This is the right home for counterfactual branches, local hypotheses, promotable assertions, open questions, and bounded reconstructed workspace state. - -### 3.3 Episodic memory and event structure - -The episodic-memory line should use: - -- Davidsonian event representations under `JVNAUTOSCI-972` -- `#V#episode_critique_memory` as the already-landed exemplar durable memory artefact -- event-linked projections that remain namespace-safe and provenance-aware - -The event model should be the common substrate for interaction episodes, memory-worthy mutations, and later durable memory retrieval. - -### 3.4 Artefact retention and provenance - -The artefact line should use: - -- `#V#computer_file_copy` as the canonical durable file-copy artefact -- provenance/privacy/namespace controls from `JVNAUTOSCI-969` -- durable artefact storage and ingestion from `JVNAUTOSCI-970` -- metadata extraction and ontology linkage from `JVNAUTOSCI-971` - -This line already has a partial operational seam through `JVNAUTOSCI-1395`, but that should be treated as a landed foundation rather than a substitute for the remaining provenance and event-model work. - -### 3.5 Representational normalisation - -`JVNAUTOSCI-531` is the clearest Phase 2 normalisation task. - -It should define the canonical direction for: - -- unified relation/text-link representation -- graph-first handling of concept-to-concept and concept-to-text assertions -- future n-ary or reified relation patterns needed for event, provenance, and qualifier-rich memory structures - -Phase 2 work should align with that direction rather than create new ad hoc storage forms. - -## 4. Explicit Gating for Retrieval-Time Memory Injection - -`JVNAUTOSCI-968` and `JVNAUTOSCI-995` should remain gated. - -They should not be treated as the leading edge of Phase 2. They should start only once the following substrate conditions are materially in place: - -1. `JVNAUTOSCI-254`, `JVNAUTOSCI-1563`, and `JVNAUTOSCI-1564` have made bounded context-bundle and dossier reconstruction explicit and reusable. -2. `JVNAUTOSCI-969` has made provenance, privacy, and namespace gating explicit for stored memories and artefacts. -3. `JVNAUTOSCI-972` has established the episodic-memory event-model anchor. -4. `JVNAUTOSCI-531` has at least fixed the canonical representational direction for relation/text-link normalisation so retrieval reads do not target a drifting storage model. - -Interpretation of the gate: - -- retrieval-time context injection is not blocked on the entire Phase 2 tranche being complete -- it is blocked on the dossier/provenance/event-model substrate being trustworthy enough that injected memory has a stable representation, safe retrieval boundary, and clear audit trail - -## 5. Clean Task Set and Ownership Map - -The clean Phase 2 task set should be read as follows. - -### 5.1 Primary active tasks - -- `JVNAUTOSCI-254`, `JVNAUTOSCI-1563`, `JVNAUTOSCI-1564`, `JVNAUTOSCI-1565` -- `JVNAUTOSCI-531` -- `JVNAUTOSCI-972` -- `JVNAUTOSCI-969` -- `JVNAUTOSCI-970` -- `JVNAUTOSCI-971` - -### 5.2 Supporting already-landed anchors - -- `JVNAUTOSCI-1607` -- `JVNAUTOSCI-1395` - -### 5.3 Explicitly gated later tasks - -- `JVNAUTOSCI-968` -- `JVNAUTOSCI-995` - -This is sufficient to represent Phase 2 as a clean task set in Jira without creating duplicate planning tickets. - -## 6. Recommended Sequencing Inside Phase 2 - -The recommended execution order is: - -1. Treat `JVNAUTOSCI-254` plus `JVNAUTOSCI-1563` and `JVNAUTOSCI-1564` as the main context/dossier substrate line. -2. Run `JVNAUTOSCI-969` and `JVNAUTOSCI-972` in parallel as the memory-safety and event-model anchors. -3. Keep `JVNAUTOSCI-531` active as the representation-normalisation direction-setting task, but do not block all Phase 2 work on full migration completion. -4. Use `JVNAUTOSCI-970` and `JVNAUTOSCI-971` to generalise the artefact substrate beyond the currently landed file-copy seam. -5. Run `JVNAUTOSCI-1565` once enough substrate exists to compare simpler context strategies against reconstructed-dossier approaches. -6. Start `JVNAUTOSCI-968` and `JVNAUTOSCI-995` only after the gate in Section 4 is judged satisfied. - -## 7. What This Clarifies - -This operationalisation clarifies five things. - -1. Phase 2 is not “do memory injection next”. It is “finish the substrate that makes memory injection safe and useful”. -2. `JVNAUTOSCI-254` and its subtasks are not side work. They are the core context-bundle/dossier spine for this tranche. -3. `JVNAUTOSCI-1607` and `JVNAUTOSCI-1395` are meaningful landed foundations, but they do not remove the need for `JVNAUTOSCI-969`, `JVNAUTOSCI-970`, `JVNAUTOSCI-971`, and `JVNAUTOSCI-972`. -4. `JVNAUTOSCI-531` matters because the substrate should not stabilise on top of fragmented relation/text-link semantics. -5. The Phase 1 benchmark stack remains active during all of this in shadow-gate mode. - -## 8. Conclusion - -`JVNAUTOSCI-1661` should be considered fully implemented once Jira reflects this task set, this ownership split, and this gating rule. - -Phase 2 can now proceed as a coherent KB-substrate programme rather than as a mix of memory ideas, context-window hacks, and partially overlapping storage tasks. diff --git a/docs/engineering/jvnautosci_1657_phase2_review.md b/docs/engineering/jvnautosci_1657_phase2_review.md deleted file mode 100644 index 3790940c4..000000000 --- a/docs/engineering/jvnautosci_1657_phase2_review.md +++ /dev/null @@ -1,211 +0,0 @@ -# JVNAUTOSCI-1657 Phase 2 Review - -**Status**: Review/update note -**Primary Jira tasks**: `JVNAUTOSCI-1662`, `JVNAUTOSCI-1657` -**Date**: 2026-04-03 - -## 1. Purpose - -This note reviews the actual implementation state of the Phase 2 KB-substrate tranche under `JVNAUTOSCI-1657`. - -`JVNAUTOSCI-1661` operationalised the tranche well. The remaining question for `JVNAUTOSCI-1662` is different: whether enough of that substrate has actually landed to justify moving into the document/recommendation vertical slice and the KB-driven presentation tranche. - -The short answer is: - -- meaningful substrate anchors exist -- the Phase 2 task set is still only partly implemented -- broad vertical-slice expansion should remain gated - -## 2. Evidence Snapshot - -### 2.1 Phase 2 task state - -The active Phase 2 spine is still largely unimplemented: - -- `JVNAUTOSCI-254`: `In Progress` -- `JVNAUTOSCI-1563`: `To Do` -- `JVNAUTOSCI-1564`: `To Do` -- `JVNAUTOSCI-1565`: `To Do` -- `JVNAUTOSCI-531`: `To Do` -- `JVNAUTOSCI-972`: `To Do` -- `JVNAUTOSCI-969`: `To Do` -- `JVNAUTOSCI-970`: `To Do` -- `JVNAUTOSCI-971`: `To Do` - -Later-gated tasks remain unstarted, as expected: - -- `JVNAUTOSCI-968`: `To Do` -- `JVNAUTOSCI-995`: `To Do` - -Two anchor tasks are already landed: - -- `JVNAUTOSCI-1607`: `Done` -- `JVNAUTOSCI-1395`: `Done` - -### 2.2 Authoritative KB artefact state - -The intended dossier/context artefacts named in the Phase 2 operationalisation note are not yet present as canonical concepts: - -- `#V#context_bundle`: absent -- `#V#workflow_context_bundle`: absent -- `#V#concept_context_bundle`: absent -- `#V#context_dossier`: absent -- `#V#workflow_report_revision`: absent -- `#V#evidence_receipt`: absent - -Some important substrate artefacts do exist: - -- `#V#testing_theory`: present -- `#V#episode_critique_memory`: present -- `#V#computer_file_copy`: present - -## 3. What Has Actually Landed - -### 3.1 Durable artefact/file-copy seam - -`JVNAUTOSCI-1395` has already given Von a meaningful durable artefact seam around `#V#computer_file_copy`. - -That means: - -- remote URL artefacts can already be persisted -- file-copy registration is already a real KB path -- some document-oriented flows can build on a genuine substrate rather than a hypothetical one - -### 3.2 Episodic-memory exemplar - -`JVNAUTOSCI-1607` has already given Von a meaningful memory artefact seam around `#V#episode_critique_memory`. - -That means: - -- the system already has one real pattern for durable episodic-memory-style artefacts -- Phase 2 is not starting from zero on memory representation -- later event-model work has a concrete exemplar to generalise from - -### 3.3 Testing-theory substrate - -`#V#testing_theory` already exists, and the repo contains testing-workflow contract surfaces for theory-local state. - -That means: - -- hybrid dossier/theory work is not wholly speculative -- there is already a canonical place for some local, promotable, bounded KR state - -## 4. What Has Not Yet Landed - -### 4.1 Dossier/context-bundle substrate is not materially available yet - -The central Phase 2 context-bundle and reconstructed-dossier line is not yet landed in a way that later phases can safely rely on. - -The most important evidence is: - -- `JVNAUTOSCI-254` is still only `In Progress` -- `JVNAUTOSCI-1563` and `JVNAUTOSCI-1564` are still `To Do` -- the named bundle/dossier concepts do not yet exist in Vontology - -So the current state is: the intended design direction is clear, but the authoritative dossier substrate is not yet real enough to underwrite later-phase implementation. - -### 4.2 Provenance/privacy and event-model substrate are not landed yet - -The Phase 2 note correctly treated `JVNAUTOSCI-969` and `JVNAUTOSCI-972` as central anchors. Both remain `To Do`. - -That means the tranche still lacks: - -- explicit provenance/privacy/namespace gating for stored memories and artefacts -- the general episodic-memory event model intended to support broader retrieval and reasoning - -This is a substantive blocker for treating retrieval-time memory use as trustworthy by default. - -### 4.3 Representational direction is not yet fixed enough - -`JVNAUTOSCI-531` remains `To Do`. - -That matters because the later phases should not build broad new behaviour on top of relation/text-link semantics that are still expected to move. - -### 4.4 Retrieval-time memory injection remains correctly gated - -Because the dossier, provenance, and event-model substrate has not yet landed, the earlier gating decision remains correct: - -- `JVNAUTOSCI-968` should stay gated -- `JVNAUTOSCI-995` should stay gated - -This review does not weaken that gate. - -## 5. Readiness Verdicts - -### 5.1 Broad document/recommendation vertical-slice implementation - -**Verdict: not ready** - -The vertical slice should not yet be expanded as if Phase 2 were complete. The missing dossier/context-bundle, provenance/privacy, and event-model substrate would make the slice too likely to grow around provisional or partial representations. - -### 5.2 Narrow vertical-slice planning and re-scoping work - -**Verdict: ready now** - -Planning and re-scoping work for the vertical slice can proceed now, especially: - -- `JVNAUTOSCI-1554` to re-scope the paper recommender as a workflow-first capability -- bounded planning inside `JVNAUTOSCI-538`, `JVNAUTOSCI-144`, and `JVNAUTOSCI-1553` - -This kind of work does not require Phase 2 to be complete, and it will reduce later ambiguity. - -### 5.3 Narrow implementation that reuses already-landed artefact seams - -**Verdict: only with care** - -A narrow implementation that reuses already-landed seams such as `#V#computer_file_copy` may be justified in limited cases, but it should not be treated as evidence that the broader Phase 2 gate is satisfied. - -In practice, this means: - -- existing document/file-copy flows can continue to be hardened -- they should not be misread as a substitute for `254 / 1563 / 1564 / 969 / 972 / 531` - -### 5.4 KB-driven presentation tranche - -**Verdict: not ready** - -KB-driven presentation depends heavily on stable dossier/context/provenance structures. Because those are not yet landed, the presentation tranche should remain behind the same substrate gate rather than advancing as if the KB views were already authoritative. - -## 6. Updated Entry Criteria for Later Phases - -`JVNAUTOSCI-1657` should be read with the following explicit entry criteria. - -### 6.1 Before broad Phase 3 implementation - -The following should be materially true before broad document/recommendation implementation starts: - -1. `JVNAUTOSCI-254` has landed a real context-bundle/dossier substrate rather than just a task description and intended artefact list. -2. `JVNAUTOSCI-1563` and `JVNAUTOSCI-1564` have landed, or their essential hybrid-dossier and bounded-reconstruction capabilities have been delivered through a justified narrower path. -3. `JVNAUTOSCI-969` has landed explicit provenance/privacy/namespace gating for stored artefacts and memories. -4. `JVNAUTOSCI-972` has landed the event-model anchor, or Phase 2 has otherwise established an equally clear general event representation. -5. `JVNAUTOSCI-531` has at least fixed the canonical representational direction strongly enough that later work is not building on a drifting storage model. -6. `JVNAUTOSCI-1554` has re-scoped the paper/document vertical slice in workflow-first terms. - -### 6.2 What is not required before planning can continue - -The following should not be treated as blockers for planning/re-scoping work: - -- `JVNAUTOSCI-1565` -- `JVNAUTOSCI-973` -- `JVNAUTOSCI-1276` - -These remain valuable, but they are not the minimum gate for saying whether Phase 3 planning may continue. - -## 7. Recommended Update to the Unified Plan - -After this review, `JVNAUTOSCI-1657` should be interpreted as follows: - -1. Phase 2 remains the current active implementation phase. -2. The document/recommendation tranche is still the right next proving ground, but only after the Phase 2 substrate gate is materially satisfied. -3. The immediate later-phase work that can proceed now is planning/re-scoping work, especially `JVNAUTOSCI-1554`, rather than broad implementation. -4. The memory-injection tasks `968` and `995` remain later tasks, not readiness indicators. -5. The KB-driven presentation tranche should remain gated on the same dossier/provenance stability that gates the broader vertical slice. - -## 8. Conclusion - -`JVNAUTOSCI-1662` should be considered complete once `JVNAUTOSCI-1657` explicitly reflects this review outcome: - -- Phase 2 has good foundations but is not yet complete enough for broad vertical-slice expansion -- vertical-slice planning may proceed now -- broad vertical-slice implementation and KB-driven presentation should remain gated -- the gating conditions are explicit and evidence-backed rather than implicit diff --git a/docs/engineering/jvnautosci_1676_recommendation_tool_vs_workflow.md b/docs/engineering/jvnautosci_1676_recommendation_tool_vs_workflow.md deleted file mode 100644 index 7a58f8904..000000000 --- a/docs/engineering/jvnautosci_1676_recommendation_tool_vs_workflow.md +++ /dev/null @@ -1,226 +0,0 @@ -# JVNAUTOSCI-1676 Recommendation Tool vs Workflow - -Status: working architecture note -Last updated: 2026-04-04 (Pacific/Auckland) -Task: `JVNAUTOSCI-1676` - -## 1. Purpose - -`JVNAUTOSCI-1676` exists to clarify when paper recommendation in Von should -remain a bounded reusable primitive and when it should be expressed as a -workflow family, especially for research-group use cases. - -This note is intentionally grounded in the current implementation rather than a -fresh greenfield design. - -## 2. Grounded current state - -The current paper recommendation vertical slice already has four distinct -surfaces. - -### 2.1 Subject profile and context surface - -`JVNAUTOSCI-117` established a Vontology-backed recommendation profile for a -subject concept, currently exposed through the settings routes and stored via -the generic paper-matching profile predicate. - -Relevant current anchors: - -- `src/backend/server/routes/settings_routes.py` -- `src/backend/services/paper_recommendation_profile_vontology_service.py` -- `#V#has_paper_matching_profile_json` - -### 2.2 Recommendation evaluation/materialisation primitive - -The current recommendation core is a bounded semantic evaluation path that: - -- assembles a subject bundle, -- recalls candidate papers, -- applies embedding scoring plus LLM reranking, -- and materialises recommendation assertions plus evaluation JSON into - Vontology. - -Relevant current anchors: - -- `src/backend/services/paper_recommendation_materialisation_service.py` -- `src/backend/services/paper_recommendation_vontology_service.py` -- `#V#paper_recommendation_assertion` -- `#V#has_paper_recommendation_assertion` -- `#V#has_paper_recommendation_evaluation_json` - -### 2.3 Workflow authority and event-triggered refresh - -The current recommendation logic is already tied to a canonical workflow and -prompt authority surface: - -- workflow: `#V#paper_recommendation_evaluation_workflow` -- prompt concept: `#V#paper_recommendation_reranker_prompt` -- event type: `paper_recommendation.requested` - -The workflow bootstrap and event binding support live in: - -- `src/backend/services/paper_recommendation_workflow_vontology_service.py` - -The repo seed bundle remains only a transitional publication asset: - -- `src/backend/workflows/repo_seed_bundles/paper_recommendation_workflow_seed_bundle.json` - -### 2.4 Von-native review surface - -`JVNAUTOSCI-1675` established the first authoritative Von-native review path in -the settings tab. It is a bounded inspection surface layered on top of the -recommendation primitive, not the authority for the recommendation logic -itself. - -Relevant current anchors: - -- `src/backend/services/paper_recommendation_review_service.py` -- `src/backend/server/routes/settings_routes.py` -- `src/frontend/web/von_interface/templates/settings_tab.html` - -## 3. Decision - -Recommendation in Von should remain both: - -- a bounded reusable primitive for explicit ranking/evaluation requests; and -- a workflow family for triggered, stateful, or multi-stage recommendation - behaviour. - -This is not a contradiction. The primitive is the shared capability; workflows -are the authoritative orchestration and policy surfaces that call it when the -task is larger than one bounded evaluation. - -## 4. When recommendation is a bounded primitive - -Recommendation should stay a tool-like primitive when all or most of the -following are true: - -- the caller has an explicit subject concept to evaluate for -- the candidate pool is explicit or can be bounded tightly -- the goal is "rank or inspect these candidate papers now" -- the result can be consumed immediately by a user, route, MCP tool, or another - workflow step -- there is no multi-actor review/approval chain -- there is no need for scheduling, digests, or downstream follow-on actions -- failure can be reported locally without needing recovery orchestration - -Examples: - -- "Rank these ten represented papers for this researcher profile." -- "Re-run the current bounded review after a profile save." -- "Show rationale and provenance for the current shortlist." -- "Evaluate whether newly ingested paper X should be considered for subject Y." - -Current primitive surfaces: - -- `build_paper_recommendations` -- `materialise_paper_recommendations` -- `materialise_paper_recommendations_for_subject(...)` - -These should remain reusable support surfaces rather than being replaced by one -bespoke workflow per small ranking request. - -## 5. When recommendation should be a workflow family - -Recommendation should be expressed as a workflow when one bounded ranking call -is not enough to represent the intended behaviour. - -That is the case when any of the following apply: - -- recommendation is triggered by events or schedules rather than a direct user - request -- candidate discovery, evaluation, review, feedback, and downstream action need - to be chained together -- recommendation must be refreshed or invalidated after KB mutations -- multiple subjects, teams, or projects must be evaluated in one coordinated run -- the result should drive later actions such as notifications, meeting - preparation, assignment, or follow-up collection -- provenance, staleness, or approval checkpoints matter across steps - -Examples: - -- new paper ingestion -> identify affected subjects -> refresh recommendations - -> surface review opportunities -- weekly research-group digest of new papers matched to projects or people -- project-level literature triage for a team rather than one person -- "who in this group should review this paper?" as a multi-subject matching and - comparison workflow -- meeting-preparation workflows that nominate papers worth discussing and attach - rationale/provenance - -These behaviours belong in workflow authority, not in Python-side orchestration -or in an oversized single MCP tool. - -## 6. Recommended authoritative split - -### 6.1 Shared primitive authority - -The reusable recommendation evaluation capability should stay grounded in: - -- subject profile overlay JSON -- materialised recommendation assertion concepts -- evaluation JSON with score, rationale, decision mode, and provenance -- the canonical reranker prompt concept - -### 6.2 Workflow authority - -Workflow-level behaviour should carry: - -- when refreshes happen -- which subject sets are in scope -- how candidate discovery is widened or narrowed -- how recommendations are surfaced for review -- when feedback is requested -- what downstream actions happen after review -- how staleness, invalidation, and retry are handled - -In short: - -- the primitive decides "how well do these represented papers fit this subject?" -- workflows decide "when, why, for whom, and what happens next?" - -## 7. Research-group interpretation - -For research-group use cases, the right default is not "replace the primitive -with a single big workflow" and not "keep everything as one tool call". - -The better stack is: - -- primitive for bounded subject-paper semantic evaluation -- workflow family for group-level orchestration -- review surfaces for human inspection and correction -- later feedback capture for learning/evaluation loops - -Recommended early research-group workflow families: - -- shared literature triage for one group or project -- role-targeted paper digests -- matching papers to projects or teams rather than only to one user profile -- reviewer suggestion workflows for a new paper -- meeting agenda suggestion workflows that nominate candidate papers for - discussion - -## 8. Immediate next work implied by this note - -The next practical step under `JVNAUTOSCI-1553` should be: - -1. Keep the current bounded recommendation primitive as the shared evaluation - surface. -2. Treat the canonical recommendation workflow as the authority for refresh and - materialisation. -3. Implement `JVNAUTOSCI-116` so feedback is captured against this split rather - than against a Slack-like one-shot interface. -4. Add later workflow tasks for group-level recommendation families only where a - bounded primitive call is clearly insufficient. - -## 9. Bottom line - -Recommendation in Von should not collapse entirely into "just a tool" or "just -a workflow". - -The durable architecture is: - -- primitive for bounded semantic recommendation evaluation -- workflow family for triggered and multi-stage recommendation behaviour -- Vontology assertions and prompt concepts as the authoritative substrate -- Von-native review and later feedback surfaces on top diff --git a/docs/engineering/jvnautosci_1781_workflow_discovery_architecture_review_2026-04-19.md b/docs/engineering/jvnautosci_1781_workflow_discovery_architecture_review_2026-04-19.md deleted file mode 100644 index d33ed652c..000000000 --- a/docs/engineering/jvnautosci_1781_workflow_discovery_architecture_review_2026-04-19.md +++ /dev/null @@ -1,382 +0,0 @@ -## JVNAUTOSCI-1781 Workflow Discovery Architecture Review - -Status: Final review for Jira closure -Date: 19 April 2026 -Author: Codex - -### 1. Purpose - -`JVNAUTOSCI-1781` asked whether workflow discovery should remain on the current -capability index, move to a scalable RAG-backed architecture, or adopt a hybrid -design. - -As of 19 April 2026, that framing is materially stale. Von already has a -hybrid discovery pipeline, but its primary dedicated workflow tier is still a -Python-authored lexical BM25 index. The live architectural question is now: - -- should Von keep a dedicated workflow retrieval plane; -- should that plane remain lexical/BM25 based; -- or should it evolve into a scalable authority-aligned hybrid retrieval - substrate while preserving the current discovery API boundary. - -This document closes `1781` as the architectural review/design task and defines -the target future runtime state that subsequent implementation tasks should -deliver. - -### 2. Current Live Architecture - -#### 2.1 Discovery pipeline - -The current discovery path is layered rather than singular: - -- `src/backend/services/workflow_discovery_service.py:61` sets a hard - `SEARCH_TIMEOUT_SECONDS = 0.5`. -- `src/backend/services/workflow_discovery_service.py:1137` begins the primary - dedicated workflow search via `_search_workflow_capabilities(...)`. -- `src/backend/services/workflow_discovery_service.py:335` falls back to - `_search_workflows_semantic(...)`, which delegates to general concept semantic - search. -- `src/backend/services/workflow_discovery_service.py:383` then falls back to - `_search_workflows_vontology(...)`. -- `src/backend/services/workflow_discovery_service.py:1182` composes these into - `discover_workflows(...)`. -- `src/backend/services/workflow_discovery_service.py:1416` wraps this for turn - use in `discover_workflows_for_turn(...)`. - -#### 2.2 Current dedicated workflow tier - -The dedicated workflow tier is still transitional: - -- `src/backend/services/workflow_capability_service.py:57` defines - `_TOKENISE_RE`. -- `src/backend/services/workflow_capability_service.py:58` defines - `_STOP_WORDS`. -- `src/backend/services/workflow_capability_service.py:150` defines - `WorkflowCapabilityIndex`. -- `src/backend/services/workflow_capability_service.py:191` builds the index - from authoritative workflow registry material via `index_from_registry(...)`. -- `src/backend/services/workflow_capability_service.py:321` performs ranked - search over that in-memory BM25 index. -- `src/backend/services/workflow_capability_service.py:629` exposes runtime - state for readiness/build/error diagnostics. -- `src/backend/services/workflow_capability_service.py:812` exposes the public - `search_workflow_capabilities(...)` API. - -This is already better than hard-coded workflow-family heuristics because the -indexed text is sourced from authoritative Vontology workflow descriptions and -routing metadata. But the retrieval core remains Python-authored lexical policy. - -#### 2.3 General semantic fallback - -The semantic fallback is not a dedicated workflow retrieval surface: - -- `src/backend/services/concept_search_service.py:544` defines - `_semantic_search(...)`. -- `src/backend/services/concept_search_service.py:643` defines - `search_concepts(...)`. -- `src/backend/services/concept_search_service.py:721` includes `"semantic"` in - the supported `match_type` values. -- `src/backend/services/concept_search_service.py:913` routes semantic requests - through `_semantic_search(...)`. -- `src/backend/services/concept_embedding_service.py:42` fixes the concept - embedding namespace as `"concepts"`. -- `src/backend/services/rag_backends/llamaindex_backend.py:120` defines - `LlamaIndexRAGService`. -- `src/backend/services/rag_backends/llamaindex_backend.py:220`, - `:239`, and `:393` show that this backend already provides a namespace-aware - persisted retrieval surface with incremental query support. - -This means Von already has a scalable general retrieval substrate, but workflow -discovery is not yet using a dedicated hybrid workflow namespace on top of it. - -#### 2.4 Selector/routing context - -Workflow discovery is already part of a richer turn pipeline: - -- `src/backend/workflows/conversation_turn_stage_model.py:282` declares the - `expected_outcome_inference` stage. -- `src/backend/workflows/conversation_turn_stage_model.py:292` declares the - `selector_preparation` stage. -- `src/backend/workflows/conversation_turn_stage_model.py:302` declares the - `selector_decision` stage. -- `src/backend/integrations/internal_mcp/orchestrator.py:24948` defines - `_prepare_turn_selector_context_outputs(...)`. -- `src/backend/integrations/internal_mcp/orchestrator.py:24962` derives the - `expected_outcome_contract`. -- `src/backend/integrations/internal_mcp/orchestrator.py:24999` calls - `discover_workflows_for_turn(...)`. -- `src/backend/integrations/internal_mcp/orchestrator.py:25378` begins the - route/action phase that consumes the prepared discovery output. -- `src/backend/workflows/workflow_selector.py:515` and `:790` show that the - selector already consumes candidate workflows as a distinct decision surface. - -This matters because the future retrieval substrate should improve recall and -candidate quality without moving selector or answer policy back into Python. - -#### 2.5 Present test limitations - -The current unit surface does not strongly exercise real dedicated recall: - -- `tests/backend/test_workflow_discovery_service.py:53` autostubs - `_search_workflow_capabilities` to return `[]`. -- `tests/backend/test_workflow_discovery_service.py:60` also stubs capability - runtime state to a ready/no-error shape. - -That is acceptable for narrow unit tests, but it means `1781` should not infer -architectural adequacy from those tests alone. - -### 3. Recent Failure Evidence - -`1781` should be interpreted in light of the current replay evidence, not only -the older task wording. - -#### 3.1 Broad fallback instead of clean workflow retrieval - -`logs/replay_1925_result.json` captured an earlier replay for: - -`What are key predicates or represented relationships for SAIL students?` - -Observed evidence: - -- `conversation.request_id = live-kb-prompt-fc50bb79-d4bb-4908-b513-440645fb0af9` -- `evaluation.selected_workflow_id = #V#tool_calling_workflow` -- `evaluation.observed_tools = [search_knowledge_base, search_concepts, jira_search, search_web, search_arxiv]` - -That is not a clean workflow retrieval success. It is broad tool fan-out from -the generic tool-calling path. - -#### 3.2 Null workflow and zero tool path - -`logs/replay_1925_salient_variant_result.json` captured the variant: - -`What predicates are salient to SAIL students?` - -Observed evidence: - -- `request_id = live-kb-prompt-62731277-00c4-41bc-b759-ab23e926a015` -- `selected_workflow = null` -- `observed_tools = []` -- response began: `I could not find any direct information...` - -This is earlier-stage failure: no grounded workflow or tool path was selected -at all. - -#### 3.3 Post-1914 reduction in broad heuristic fan-out - -`logs/replay_1914_1925_result.json` shows the same original prompt after the -`1914` lexical-heuristic removal: - -- `evaluation.selected_workflow_id = #V#tool_calling_workflow` -- `evaluation.observed_tools = [find_relations_with_argument, search_knowledge_base]` - -This is better than the older five-tool fan-out, but it is still not a robust -authority-aligned workflow retrieval outcome. - -### 4. Architectural Options - -#### Option A: Keep the current BM25 capability index as the long-term substrate - -Advantages: - -- very simple cold start and rebuild behaviour; -- fast in-memory lookup; -- good observability because the index is small and explicit. - -Problems: - -- lexical tokenisation, stopwording, and normalisation remain Python-authored - semantic policy; -- multilingual and paraphrase robustness remains poor; -- the current replay failures are consistent with missed semantic recall and - selector candidate starvation; -- this does not align with the broader `1913` anti-hack direction. - -Decision: reject as the long-term architecture. - -#### Option B: Collapse workflow discovery into the general concept-search plane - -Advantages: - -- reuse the existing concept vector infrastructure; -- remove one dedicated workflow retrieval subsystem. - -Problems: - -- workflows are not ordinary concepts for retrieval purposes; they need a - different index unit and metadata contract; -- workflow discovery needs executability, routing-profile, publication, and - discovery-exemplar awareness, not just concept similarity; -- mixing workflows into the general concept retrieval plane would weaken - observability and make workflow-specific ranking harder to diagnose; -- the current concept semantic path is a fallback convenience, not an explicit - workflow retrieval authority surface. - -Decision: reject as the primary architecture. - -#### Option C: Keep a dedicated workflow retrieval plane, but replace the -transitional BM25 core with a scalable hybrid retrieval substrate - -Advantages: - -- preserves the correct architectural boundary: workflow retrieval remains its - own support surface; -- keeps authority in Vontology workflow text, routing metadata, and discovery - exemplars rather than in Python heuristics; -- supports a future runtime state with dense, sparse, or multi-vector recall - without changing the discovery API; -- fits both today’s implementation constraints and the expected future runtime - state. - -Decision: adopt. - -### 5. Recommended Future Runtime State - -The future runtime state should look like this: - -1. A dedicated workflow retrieval namespace remains the primary recall surface. -2. Indexed workflow documents are materialised only from authoritative - Vontology workflow text and metadata: - - workflow description/purpose text - - routing profile metadata - - discovery exemplars - - publication/executability metadata needed for later filtering -3. The retrieval engine behind `search_workflow_capabilities(...)` becomes a - scalable hybrid substrate rather than a Python BM25 implementation. -4. Hybrid means: - - dense semantic recall for paraphrase and multilingual robustness; - - sparse or lexical-compatible recall for exact terminology and rare tokens; - - optional later reranking or late interaction where justified by runtime - budgets. -5. Executability, routing eligibility, and safety filtering remain explicit - support-surface checks after retrieval, not hidden lexical proxies before it. -6. Discovery keeps a hard bounded latency budget, but the budget is spent on a - dedicated workflow retrieval plane rather than on a Python-authored lexical - first pass plus fallback retries. -7. Telemetry records: - - retrieval source and namespace - - candidate counts before and after filtering - - build readiness / cold-start state - - whether dense, sparse, or hybrid recall was used - - the effective discovery query used after expected-outcome enrichment -8. Failure semantics remain fail-closed: - - no fallback to guessed Python-authored workflow descriptions; - - no return to English stopword lists or token overlap steering. - -### 6. Authority Model - -The correct authority split is: - -- Authoritative: - - workflow descriptions and purpose text in Vontology text relations - - workflow routing profiles - - workflow discovery exemplars - - workflow publication/executability metadata -- Support-only code: - - indexing/materialisation pipeline - - retrieval backend adapter - - bounded latency/timeouts - - post-retrieval validation and filtering - - telemetry and diagnostics - -The retrieval model, index structure, and storage engine may evolve. The -authored workflow discovery policy must not drift back into Python token tables, -stopword sets, or hard-coded semantic cues. - -### 7. Cold Start, Rebuild, and Incremental Update - -#### 7.1 Current state - -The current capability index is simple to rebuild in-process and easy to expose -through runtime state. By contrast, the concept retrieval plane already has a -more scalable lifecycle: - -- `src/backend/utilities/concept_index_worker.py:27`, `:169`, and `:182` show a - worker-driven indexing path for concept embeddings. -- `src/backend/services/concept_embedding_service.py:36`, `:38`, `:631`, and - `:718` show explicit stale/pending/indexed status handling and aggregate - stats. - -#### 7.2 Required future behaviour - -The dedicated workflow retrieval plane should adopt equivalent lifecycle -discipline: - -- authoritative workflow text/metadata changes mark workflow retrieval - documents stale; -- incremental reindex should be the default; -- full rebuild remains available for cold start or drift repair; -- discovery telemetry should expose whether misses occurred because the index - was stale, empty, or unavailable. - -### 8. Short Retrieval Literature Review - -This review is intentionally short and targeted toward the architectural choice. - -- Lewis et al., *Retrieval-Augmented Generation for Knowledge-Intensive NLP - Tasks* (NeurIPS 2020), showed the value of combining parametric generation - with explicit non-parametric retrieval for knowledge-intensive tasks, with - better factuality and updateability than parametric-only generation. - Source: -- Santhanam et al., *ColBERTv2* (NAACL 2022), showed that late interaction can - materially improve retrieval quality while using compression to keep the space - cost manageable. - Source: -- Formal et al., *SPLADE v2* (2021), showed that learned sparse retrieval can - preserve exact-term and inverted-index strengths while substantially - outperforming plain lexical baselines on retrieval benchmarks. - Source: -- Chen et al., *M3-Embedding* (2024/2025 revisions), is directly relevant to - Von’s expected future runtime state because it unifies dense, sparse, and - multi-vector retrieval in one multilingual model family. - Source: - -Implication for Von: the long-term choice should not be “BM25 or dense vectors”. -The right design is a hybrid workflow retrieval plane with a stable API and -authoritative Vontology inputs, leaving room for dense+sparse or dense+sparse -+ multi-vector implementations as runtime maturity increases. - -### 9. Decision - -`JVNAUTOSCI-1781` is resolved with this recommendation: - -- keep a dedicated workflow retrieval plane; -- retire the current Python BM25/stopword capability index as the long-term - retrieval core; -- replace it with an authority-aligned scalable hybrid workflow retrieval - substrate; -- keep the existing discovery API boundary where practical so that the change - is mostly internal to the retrieval support surface. - -### 10. Follow-On Work - -#### 10.1 Immediate implementation follow-on - -`JVNAUTOSCI-1915` is the direct implementation task for this architectural -decision. It should: - -- replace the current BM25/English stopword discovery surface; -- preserve authoritative Vontology workflow text/metadata as the indexed input; -- add proper lifecycle and telemetry for the new retrieval substrate. - -#### 10.2 Replay-driven failure follow-on - -`JVNAUTOSCI-1943` remains the targeted follow-on for the salience/paraphrase -failure family. It should not redefine the retrieval architecture itself. It -should implement and validate the discovery/selector path against this chosen -architecture. - -### 11. User Impact Summary - -This task does not directly change a production answer path by itself. Its -effect is to remove architectural ambiguity before more code is written in the -wrong direction. - -If the follow-on work is implemented correctly, users should see: - -- fewer missed workflow candidates for paraphrased or multilingual requests; -- less fallback to generic tool-calling behaviour; -- better diagnosability when workflow recall fails; -- better long-term scaling as the workflow catalogue grows. - -Without this decision, Von is likely to keep accumulating small lexical fixes on -top of a transitional retrieval tier. diff --git a/docs/engineering/jvnautosci_1962_enduring_memory_architecture_2026-04-23.md b/docs/engineering/jvnautosci_1962_enduring_memory_architecture_2026-04-23.md deleted file mode 100644 index 5f06458b8..000000000 --- a/docs/engineering/jvnautosci_1962_enduring_memory_architecture_2026-04-23.md +++ /dev/null @@ -1,259 +0,0 @@ -# JVNAUTOSCI-1962 Enduring Memory Architecture - 23 April 2026 - -> **Document status: Long-horizon design with a dated implementation snapshot.** -> The design may remain useful. Claims using “current”, “already”, “still”, -> source anchors, and task priority describe the April 2026 evidence base and -> require live revalidation. This is not release-status authority. - -## Purpose - -This note records the concrete enduring-memory architecture for Von after the -main-turn memory landing in `JVNAUTOSCI-1988` and the self-improvement/profile -landings through `JVNAUTOSCI-1994` and `JVNAUTOSCI-1993`. - -`JVNAUTOSCI-1962` is not a task to "add memory" in the abstract. The live code -already has multiple memory-capable substrates. The actual gap is that Von -still lacks one integrated architecture across semantic, episodic, procedural, -and policy memory with explicit: - -- represented-vs-virtual boundaries; -- promotion and revision rules; -- provenance and evidence handling; and -- namespace and visibility rules. - -This note therefore does two narrower things: - -1. states the memory architecture that now matches the live code and the design - intent in `docs/engineering/Von_for_AgenticAI.md`; and -2. turns that architecture into the first concrete implementation slices rather - than leaving it as a prose umbrella. - -## Current Live Substrate - -The current Von code already contains the following durable or memory-adjacent -surfaces: - -- Main-turn memory context resolution and rendering in - `src/backend/services/conversation_turn_memory_context_service.py:547`. -- Context bundles, context dossiers, report revisions, reconstructed - workspaces, and evidence receipts in - `src/backend/services/context_bundle_service.py:622`, - `:998`, `:1296`, and `:1541`. -- Durable critique-memory persistence and projection in - `src/backend/services/episode_critique_memory_service.py:1723`. -- Workflow-use episode logging and aggregate recomputation in - `src/backend/services/workflow_episode_service.py:395` and `:472`. -- Proposal-addressable workflow authoring and promotion-evaluation storage in - `src/backend/workflows/workflow_studio_service.py:624`, `:772`, and `:2000`. -- Represented self-improvement profile loading in - `src/backend/services/episode_self_improvement_profile_vontology_service.py:286`. - -The problem is not absence. The problem is fragmentation. These surfaces still -behave more like adjacent subsystems than one governed enduring-memory -architecture. - -## Architecture Decision - -### 1. Memory Strata Stay Distinct - -Von should keep four explicit enduring-memory strata: - -| Stratum | Canonical durable authority | What should be represented | What may remain virtual or raw-backed | -| --- | --- | --- | --- | -| Semantic | Vontology concepts, relations, texts, and other KB assertions | Durable facts, stable entity links, promoted hypotheses, revision records | Joined retrieval views, transient search expansions, query-specific synthesis | -| Episodic | Turn records, workflow-use episodes, critique memories, dossiers, report revisions, evidence receipts | Critique memories, dossiers, report revisions, evidence receipts, explicit durable episode projections | High-volume raw turn logs, reconstructed workspaces, aggregate counters, query manifests | -| Procedural | Workflows, prompts, validators, launch/routing profiles, publication lifecycle, authoring proposals | Published artefacts, proposal records, promotion evaluations, procedural revision lineage | Preview graphs, diffs, local experiment scaffolds, other bounded pre-publication views | -| Policy | Profiles, guidelines, routing or context policy artefacts, evaluator-derived guidance objects | Represented profile revisions and durable guidance objects | Per-query ranking views, transient scoring outputs, local diagnostic summaries | - -The engineering consequence is simple: Von should stop treating "memory" as one -flat retrieval pool. Query surfaces must preserve stratum, provenance, and -scope. - -### 2. Not Everything Should Be Materialised the Same Way - -Some artefacts should be durable represented individuals or relations in -Vontology. Others should remain virtual or raw-store-backed and only surface as -typed references. - -Represent by default when the artefact is: - -- evidence-backed; -- reusable across sessions or workflows; -- worth revising later; -- needed as an auditable promotion or publication record; or -- needed as a stable reference target for later evaluators or workflows. - -Allow raw-backed or virtual views when the artefact is: - -- high-volume episode telemetry whose primary role is replay or audit; -- a derived aggregation or join; -- a bounded workspace reconstruction view; -- a query-time manifest assembled from authoritative durable sources; or -- an ephemeral experiment-local intermediate that has not yet passed promotion. - -This means Von should not mirror every raw episode into Vontology just to claim -"represented memory". It should instead expose raw-backed stores through typed, -governed references and promote only the durable reusable slices. - -### 3. Every Queryable Memory Reference Needs a Common Contract - -Across all strata, every durable artefact or queryable memory reference should -carry, directly or by support-surface projection: - -- `memory_stratum` -- `artefact_kind` -- `represented` vs `virtual` -- `subject_concept_ids` and/or `workflow_ids` -- `namespace` -- `user_concept_id` -- `organisation_concept_id` -- `visibility_scope` -- `source_memory_ids` or episode anchors -- `evidence_receipt_ids` -- `revision_lineage` or predecessor/supersession pointers -- `created_at_utc` and `updated_at_utc` - -This is the contract the current code lacks. Turn-memory, evaluator, and -self-improvement paths still use separate local knowledge about where memory -lives. - -### 4. Promotion Must Become an Explicit Cross-Strata Workflow - -The durable promotion path should be: - -- `episodic -> semantic` - Promote evidence-backed facts, assertions, or corrected beliefs into - Vontology/KB, preserving provenance and retraction pathways. -- `episodic -> procedural` - Promote repeated successful or repeatedly failing traces into workflow, - prompt, validator, or other procedural revision proposals with evaluation and - publication gates. -- `episodic -> policy` - Promote repeated critique/evaluator signals into represented profile or - guidance revisions rather than leaving them only as local suggestions. - -For all three: - -- promotion candidates should be explicit durable objects or equivalent - represented workflow state; -- acceptance and rejection should both preserve provenance; -- missing evidence, missing namespace clarity, or missing authoritative publish - surfaces should fail closed; and -- revision should create lineage, not silent overwrite. - -### 5. Namespace and Visibility Are Part of the Memory Model - -The authoritative namespace contract in -`docs/engineering/effective_namespace_contract.md` applies to memory artefacts -as well, not only to RAG or request routing. - -Durable memory should therefore: - -- preserve both the namespace string and its user/org components; -- support both `#V#user` and `#V#user@org` scopes; -- avoid silently degrading org-scoped memory to user-only scope; -- make shared/global visibility explicit rather than implicit; and -- fail closed when a user-scoped memory operation lacks required scope data. - -This matters because enduring memory is not just "what is stored". It is also -who may reuse it and under what scope. - -### 6. Query Surfaces Should Become First-Class Support Surfaces - -The intended support-surface stack is: - -1. a canonical enduring-memory manifest/query layer across the existing live - memory stores and represented artefacts; -2. turn-context construction that can consume that manifest instead of bespoke - point lookups; -3. promotion workflows that reference the same stable memory IDs or manifests; - and -4. evaluators and benchmark worlds that score memory use against those stable - references. - -This preserves represented authority while keeping Python in its correct role: -federation, validation, telemetry, persistence, and other reusable execution -support. - -## Evaluation Requirements - -The enduring-memory architecture should be evaluated on tasks that actually -require long-horizon memory, not just explicit recall. - -Minimum acceptance dimensions: - -- buried evidence retrieval -- task-state reconstruction after interruption -- preference or policy grounding across sessions -- knowledge update and retraction handling -- abstention or graceful recovery when memory is absent -- procedural reuse or proposal quality from prior episodes - -Acceptance should combine: - -- repeatable benchmark-style cases for comparison against simple baselines; and -- a smaller replay or nearest-real-path acceptance set on the real Von path. - -## Short Literature Note - -The architecture direction here was checked against recent primary sources: - -- [Memory for Autonomous LLM Agents: Mechanisms, Evaluation, and Emerging Frontiers](https://arxiv.org/abs/2603.07670) - reinforced treating memory as a write-manage-read architecture over temporal - scope, representational substrate, and control policy. This directly supports - a cross-strata architecture task rather than another local retrieval feature. -- [E-mem: Multi-agent based Episodic Context Reconstruction for LLM Agent Memory](https://arxiv.org/abs/2601.21714) - reinforced that some memory value is lost when everything is flattened into - decontextualised retrieval artefacts. This supports preserving raw or - reconstructable episodic context instead of materialising everything into one - flat store. -- [MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks](https://arxiv.org/abs/2602.16313) - reinforced that memory and action must be evaluated together. This is why the - evaluation slice here includes task-state reconstruction and multi-session use - rather than only recall. -- [LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://arxiv.org/abs/2410.10813) - reinforced the need to evaluate knowledge updates, temporal reasoning, and - abstention, not just retrieval hit rate. -- [ProcMEM: Learning Reusable Procedural Memory from Experience via Non-Parametric PPO for LLM Agents](https://arxiv.org/abs/2602.01869) - reinforced the direction of promoting episodic traces into reusable procedural - artefacts with explicit executability and gating. - -These papers did not overturn Von's design direction. They sharpened the -represented-vs-virtual distinction, the promotion architecture, and the need -for stronger long-horizon evaluation. - -## First Implementation Slices - -`JVNAUTOSCI-1962` now decomposes into the following concrete tasks: - -1. `JVNAUTOSCI-1995` - Implement a canonical enduring-memory manifest and query surface across - dossiers, workflow-use episodes, critique memories, and durable artefacts. -2. `JVNAUTOSCI-1996` - Implement represented memory-promotion and revision workflows from episodic - evidence into semantic, procedural, and policy memory. -3. `JVNAUTOSCI-1997` - Add a long-horizon enduring-memory evaluation and task-state reconstruction - harness on the real Von path. - -These tasks are intentionally not replacements for `JVNAUTOSCI-1964` or -`JVNAUTOSCI-1963`. They are the concrete continuation of the enduring-memory -line that those broader evaluator and self-improvement umbrellas should now -consume. - -## Open Questions Deferred - -This note does not settle everything. The following remain intentionally open -for later implementation work: - -- whether some episodic raw stores should gain richer represented projections - over time, rather than only a federated manifest layer; -- how much of semantic promotion should be handled through existing assertion - pathways versus new represented promotion records; -- how far policy memory should unify profile, guideline, and evaluator-derived - guidance objects; and -- whether certain benchmark or experiment-local memory artefacts should expire - automatically rather than entering durable lineage. - -Those are implementation questions for the follow-on tasks above, not reasons to -leave the architecture implicit. diff --git a/docs/engineering/jvnautosci_1963_self_improvement_worlds_architecture_2026-04-23.md b/docs/engineering/jvnautosci_1963_self_improvement_worlds_architecture_2026-04-23.md deleted file mode 100644 index 4b0970423..000000000 --- a/docs/engineering/jvnautosci_1963_self_improvement_worlds_architecture_2026-04-23.md +++ /dev/null @@ -1,305 +0,0 @@ -# JVNAUTOSCI-1963 Isolated Self-Improvement Worlds Architecture - 23 April 2026 - -> **Document status: Long-horizon design with a dated implementation snapshot.** -> The design may remain useful. Claims using “current”, “already”, “still”, -> source anchors, and task priority describe the April 2026 evidence base and -> require live revalidation. This is not release-status authority. - -## Purpose - -This note records the concrete self-improvement-world architecture for Von after -the represented self-improvement and proposal-addressability landings through -`JVNAUTOSCI-1987`, `JVNAUTOSCI-1993`, `JVNAUTOSCI-1994`, `JVNAUTOSCI-1962`, -and `JVNAUTOSCI-1964`. - -`JVNAUTOSCI-1963` is not a task to "add self-improvement" in the abstract. The -live code already has a real represented self-improvement loop and a separate -testing or experiment substrate. The actual gap is narrower and more important: -those two lines are still only loosely connected. - -Von can already: - -- generate represented workflow-improvement proposals from critique memory; -- store proposal-addressable procedural candidates and promotion evaluations; -- run experiment specs and experiment runs with verdict computation; -- use testing theories for local assertions, diffing, promotion, and rollback. - -What it still lacks is one coherent release-engineering architecture that turns -those pieces into isolated candidate worlds with explicit: - -- baseline-vs-candidate identity; -- evaluator- and memory-backed comparison; -- shadow or bounded pre-activation validation; -- rollback-safe promotion gates; and -- a path beyond workflow-only candidates into prompt, policy, and retrieval - candidates. - -This note therefore does two things: - -1. states the self-improvement-world architecture that now matches the live code - and the design intent in `docs/engineering/Von_for_AgenticAI.md`; and -2. turns that architecture into the first concrete implementation slices rather - than leaving `1963` as a broad umbrella. - -## Current Live Substrate - -The current Von code already contains the following self-improvement-capable -surfaces: - -- Represented workflow-improvement context building, proposal submission, - promotion-context construction, and promotion-evaluation persistence in - `src/backend/services/episode_self_improvement_service.py:440`, `:555`, - `:638`, and `:706`. -- Critique-triggered launch of the represented self-improvement loop in - `src/backend/workflows/durable/episode_evaluation_workflow.py:318`. -- Proposal-addressable workflow candidate validation, promotion evaluation, - approval review, and rollback in - `src/backend/workflows/workflow_studio_service.py:1684`, `:2000`, `:2093`, - and `:2251`. -- Represented experiment spec creation, experiment-run execution, and - experiment verdict computation in - `src/backend/services/experiment_run_service.py:1046`, `:1181`, and `:1688`. -- Testing-theory import, local assertion, diff, rollback, and validated-claim - promotion in `src/backend/services/testing_theory_service.py:363`, `:420`, - `:478`, `:573`, and `:625`. -- Benchmark reporting over critique memories in - `src/backend/services/episode_critique_benchmark_service.py:1393`. - -The problem is not absence. The problem is separation: - -- the represented self-improvement loop still builds promotion context mostly - from critique memory plus a benchmark summary rather than from explicit - candidate-world evidence; -- the experiment-run substrate already has `baseline_workflow_id`, - `candidate_workflow_ids`, `benchmark_world_id`, degradation assessment, and - promotion recommendation, but the main self-improvement loop does not yet use - it as its default control surface; -- testing theories still sit more in the testing line than in the main - self-improvement release path; and -- the live self-improvement path is still mostly workflow-only even though the - design note calls for prompt, retrieval-policy, and policy-memory revision as - well. - -## Architecture Decision - -### 1. Self-Improvement Should Be Treated as Release Engineering - -Von should not model self-improvement as opaque in-agent recursion or as a set -of candidate artefacts that immediately compete for canonical status. - -The intended model is: - -- one canonical published line for each durable artefact family; -- one or more isolated candidate worlds that may evaluate revisions against the - canonical line; -- explicit regression-aware evidence before promotion; and -- explicit rollback or demotion support after publication if later evidence - invalidates the promotion. - -This means `1963` should consume the already represented proposal and experiment -surfaces, not bypass them with prompt-local or benchmark-summary-only -promotion decisions. - -### 2. Candidate Worlds Must Become First-Class Represented Objects - -Von should stop treating the phrase "benchmark world" as a loose idea spanning -proposal records, theory sandboxes, experiment runs, and benchmark outputs. - -Every isolated candidate world should carry, directly or by support-surface -projection: - -- `candidate_world_id` -- `candidate_surface` -- `candidate_artefact_ids` -- `baseline_artefact_ids` -- `proposal_id` -- `experiment_spec_id` -- `experiment_run_id` -- `benchmark_world_id` -- `testing_theory_id` -- `namespace` -- `user_id` -- `org_id` -- `memory_manifest_fingerprint` -- `evaluator_axis_versions` -- `promotion_gate_state` -- `rollback_plan` -- `created_at_utc` -- `updated_at_utc` - -This is the contract the current code lacks. The constituent objects exist, but -the self-improvement path still does not expose one machine-usable identity for -"the candidate world being evaluated". - -### 3. Reuse the Existing Experiment and Testing-Theory Substrate - -`JVNAUTOSCI-1963` should not create a second experiment system parallel to -`experiment_run_service.py`. - -The intended division of labour is: - -- workflow, prompt, or policy candidate records remain the represented artefact - candidates; -- experiment specs and experiment runs remain the canonical execution and - verdict container for isolated evaluation; -- testing theories remain the bounded local-write surface for ephemeral claims, - assumptions, and candidate-only semantic or policy assertions; and -- workflow publication lifecycle and proposal review remain the canonical - publication and rollback surface. - -The missing work is integration and contract-shaping, not a fresh storage model. - -### 4. World Types Should Be Explicit - -Von should distinguish at least four world types: - -| World type | Main role | Durability | Authority | -| --- | --- | --- | --- | -| Canonical world | The currently published line used for real execution | Durable | Canonical authority | -| Candidate world | A represented revision bundle under experiment | Durable manifest, isolated mutable state | Non-canonical | -| Benchmark world | Replay, fixture, or simulated environment used to score the candidate world | Usually durable manifest plus bounded run state | Non-canonical evidence surface | -| Shadow world | Bounded post-sandbox, pre-activation observation against live-like traffic or live traces without canonical replacement | Durable summary plus bounded run state | Non-canonical evidence surface | - -Not every candidate must traverse every world type, but the architecture should -make the phase explicit. A promotion gate should know whether it is looking at -sandbox-only evidence, benchmark-world evidence, or shadow-world evidence. - -### 5. Promotion Gates Must Consume Evaluator and Memory Evidence - -The future promotion gate for self-improvement should not be driven mainly by a -generic benchmark summary, scalar verdict, or free-text justification. - -It should consume: - -- the multi-axis evaluator contract from `JVNAUTOSCI-1998`; -- grounded helpfulness, calibration, recovery, and long-horizon evaluator - families from `JVNAUTOSCI-1999`, `JVNAUTOSCI-2000`, and `JVNAUTOSCI-2001`; -- stable memory refs or manifests from `JVNAUTOSCI-1995`; and -- explicit flip or regression accounting between baseline and candidate worlds. - -The important engineering consequence is that candidate promotion should be -reasoned over machine-usable axis evidence and stable memory bindings, not only -over one aggregate score or one benchmark prose summary. - -### 6. Rollback and Activation Need Explicit Lifecycle Stages - -Von already has approval review, publication lifecycle, and rollback surfaces -for workflow candidates. `1963` should extend that into an explicit -candidate-world lifecycle rather than treating rollback as a rare exceptional -afterthought. - -The intended high-level lifecycle is: - -1. proposal created; -2. candidate validated; -3. isolated experiment running; -4. experiment verdict computed; -5. shadow or bounded pre-activation evaluation, when required; -6. review-ready and approval-gated; -7. canonical promotion or rejection; -8. post-promotion monitoring and possible rollback or demotion. - -This does not mean every stage must be fully automatic. It means the stage -boundary should be represented and auditable. - -### 7. Workflow-First Now, Broader Candidate Surfaces Next - -The current live self-improvement loop is still workflow-first, and that is the -right immediate implementation order because workflow proposal, validation, -review, and rollback surfaces already exist. - -But the architecture should not stop there. The same candidate-world contract -should eventually support: - -- prompt-family revisions; -- retrieval or routing profile revisions; -- policy-memory or guideline revisions; and -- other represented procedural or policy artefacts. - -The rule is the same across all of them: represented candidate artefact, bound -to an isolated candidate world, promoted only after evaluable evidence. - -## Evaluation Requirements - -The self-improvement-world architecture should be accepted on real or near-real -Von paths, not only by creating isolated data structures. - -Minimum acceptance dimensions: - -- proposal-to-world identity and traceability -- explicit baseline-vs-candidate comparison -- forbidden side-effect and policy-violation handling -- evaluator-axis evidence preservation across experiment and promotion -- rollback or demotion readiness after adverse evidence -- correct fail-closed behaviour when required memory refs, evaluator outputs, or - candidate artefact bindings are missing - -Acceptance should combine: - -- targeted replay or nearest-real-path experiments over the actual candidate - workflow line; and -- smaller benchmark-world or simulator-backed cases that include fault - injection, missing evidence, or degraded inputs. - -## Short Literature Note - -The architecture direction here was checked against recent primary sources: - -- [AgentDevel: Reframing Self-Evolving LLM Agents as Release Engineering](https://arxiv.org/abs/2601.04620) - reinforced treating self-improvement as an auditable release pipeline with a - single canonical version line and regression-aware promotion evidence rather - than opaque in-agent recursion. -- [Governed Capability Evolution for Embodied Agents: Safe Upgrade, Compatibility Checking, and Runtime Rollback for Embodied Capability Modules](https://arxiv.org/abs/2604.08059) - reinforced staged validation, sandbox evaluation, shadow deployment, gated - activation, and rollback as first-class system stages. That directly supports - making candidate-world phase explicit in Von. -- [Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluation](https://arxiv.org/abs/2402.11443) - reinforced that static benchmark suites understate failure modes and that - evaluation worlds should be able to evolve or harden over time. -- [OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation](https://arxiv.org/abs/2604.10866) - reinforced the value of environment simulation plus controlled fault injection - when measuring agent robustness, and therefore supports explicit - benchmark-world modelling rather than only replaying happy-path traces. - -These papers did not overturn Von's design direction. They sharpened the need -for represented candidate-world identity, staged promotion, and evaluator-backed -regression evidence. - -## First Implementation Slices - -`JVNAUTOSCI-1963` now decomposes into the following concrete tasks: - -1. `JVNAUTOSCI-2002` - Implement represented candidate-world manifests and proposal-to-experiment-run - linkage for workflow self-improvement candidates. -2. `JVNAUTOSCI-2003` - Implement evaluator- and memory-backed baseline-vs-candidate comparison for - self-improvement experiment runs. -3. `JVNAUTOSCI-2004` - Implement experiment-backed promotion, bounded shadow evaluation, and - rollback-safe publication gating for workflow self-improvement candidates. -4. `JVNAUTOSCI-2005` - Extend isolated self-improvement worlds beyond workflows to prompt, policy, - and retrieval or memory candidate surfaces. - -These tasks are intentionally not replacements for `JVNAUTOSCI-1998`, -`JVNAUTOSCI-1999`, `JVNAUTOSCI-2000`, `JVNAUTOSCI-2001`, `JVNAUTOSCI-1995`, -or `JVNAUTOSCI-1996`. They are the concrete continuation of the -self-improvement-world line that those evaluator and memory tasks should now -feed. - -## Open Questions Deferred - -This note does not settle everything. The following remain intentionally open -for later implementation work: - -- how much simulator-backed benchmark-world generation should be centralised - versus domain-specific; -- when shadow-world evidence is mandatory versus optional before activation; -- how prompt and policy candidate packaging should look once workflow-first - slices land; and -- which failed candidate worlds should be retained durably for audit versus - garbage-collected after a bounded period. - -Those are follow-on implementation questions, not reasons to leave the -architecture implicit. diff --git a/docs/engineering/jvnautosci_1964_evaluator_architecture_2026-04-23.md b/docs/engineering/jvnautosci_1964_evaluator_architecture_2026-04-23.md deleted file mode 100644 index 2f0aee7e1..000000000 --- a/docs/engineering/jvnautosci_1964_evaluator_architecture_2026-04-23.md +++ /dev/null @@ -1,281 +0,0 @@ -# JVNAUTOSCI-1964 Evaluator Architecture - 23 April 2026 - -> **Document status: Long-horizon design with a dated implementation snapshot.** -> The design may remain useful. Claims using “current”, “already”, “still”, -> source anchors, and task priority describe the April 2026 evidence base and -> require live revalidation. This is not release-status authority. - -## Purpose - -This note records the concrete evaluator architecture for Von after the -postcondition-critic, critique-memory, self-improvement, and enduring-memory -landings through `JVNAUTOSCI-1987`, `JVNAUTOSCI-1993`, `JVNAUTOSCI-1994`, and -`JVNAUTOSCI-1962`. - -`JVNAUTOSCI-1964` is not a task to "add more critics" in the abstract. The live -code already has a real evaluator substrate: - -- deterministic postcondition and completion-gate support in - `src/backend/services/turn_execution_record_service.py`; -- bounded episode evidence bundles in - `src/backend/services/episode_critic_evidence_service.py`; -- prompt-backed episode evaluation workflow support in - `src/backend/workflows/durable/episode_evaluation_workflow.py`; -- durable critique-memory persistence in - `src/backend/services/episode_critique_memory_service.py`; and -- replay or sampled benchmark reporting in - `src/backend/services/episode_critique_benchmark_service.py`. - -The actual gap is narrower and more important: evaluator coverage is still too -scalar and too completion-centric for grounded helpfulness, calibration, -abstention, recovery quality, and long-horizon task-state integrity to serve as -first-class machine-usable evidence. - -This note therefore does two things: - -1. states the evaluator architecture that now matches the live code and the - design intent in `docs/engineering/Von_for_AgenticAI.md`; and -2. turns that architecture into the first concrete implementation slices rather - than leaving `1964` as a broad umbrella. - -## Current Live Substrate - -The current Von code already contains the following evaluator-capable surfaces: - -- Postcondition checks, completion-gate derivation, execution correctness, and - critic summary construction in - `src/backend/services/turn_execution_record_service.py:1300`, `:5506`, - `:5658`, and `:6399`. -- Episode evidence bundle construction, reconstructable turn-record recovery, - bounded receipts, and format-over-content diagnostics in - `src/backend/services/episode_critic_evidence_service.py:1395`. -- Prompt-backed episode evaluation workflow execution and critique-memory - persistence in - `src/backend/workflows/durable/episode_evaluation_workflow.py:107`, - `:228`, and `:293`. -- Durable verdict/confidence persistence and critique-memory projection in - `src/backend/services/episode_critique_memory_service.py:716` and `:1723`. -- Sampled meta-audit and recurrence reporting in - `src/backend/services/episode_critique_benchmark_service.py:1393`. -- Canonical workflow and prompt-family bootstrap in - `src/backend/services/episode_evaluation_workflow_vontology_service.py:79`. - -The problem is not absence. The problem is evaluator narrowness: - -- the core persistent shape is still mostly `verdict`, `confidence`, - `unresolved_check_count`, and remediation links; -- the benchmark layer still reasons mostly over verdict proxies, recurrence, - remediation presence, and sampled rebuilds rather than explicit evaluator - axes; -- the workflow bootstrap currently seeds one main episode-critic prompt family - rather than distinct groundedness, calibration, recovery, and long-horizon - evaluator families; and -- the current evidence bundle only partially exposes answer-grounding, - abstention, and long-horizon state-reconstruction evidence as first-class - reusable receipts. - -## Architecture Decision - -### 1. Evaluation Must Become Multi-Axis - -Von should stop treating episode evaluation as one scalar verdict with a small -set of follow-up hints. The durable evaluator contract should carry multiple -axes that are independently measurable and independently failable. - -The minimum first-class axes should be: - -| Axis | Main question | Primary live evidence | Why it matters | -| --- | --- | --- | --- | -| Execution correctness | Did the required effects and declared completion state actually match reality? | Postcondition checks, completion gate, tool or workflow receipts | Already partially live; remains the foundation | -| Grounded helpfulness | Did the answer or action stay faithful to retrieved, observed, or memory-backed evidence while still being useful? | Shared turn context, search or tool receipts, answer artefacts, evidence bundle | Prevents "correct sounding" but ungrounded or weakly supported output | -| Calibration and abstention | Did the agent commit, abstain, escalate, or ask follow-up in a way that matched uncertainty and evidence strength? | Completion gate, answer text, evidence gaps, selected workflow or route, explicit abstention path | Needed for trustworthy non-bluffing behaviour | -| Recovery quality | When evidence or execution failed, did the system choose a bounded, appropriate recovery path? | Critique memory, remediation routing, later turn outcomes, recovery actions | Needed for safe self-improvement and practical deployability | -| Long-horizon task-state integrity | Did the agent reconstruct the right task state over time, use memory appropriately, and preserve continuity across interruptions or updates? | Durable memory refs, dossiers, workflow episodes, critique memory, later replay evidence | Needed for long-horizon collaboration rather than turn-local correctness | - -The engineering consequence is simple: a later promotion, benchmark, or -roll-back gate should not have to infer all of this from one generic verdict. - -### 2. Every Axis Needs a Common Machine-Usable Contract - -Across all axes, persisted evaluator outputs should carry a common shape with -axis-specific evidence inside it. - -Each axis result should carry, directly or by support-surface projection: - -- `axis_id` -- `axis_version` -- `status` -- `score` or `utility_estimate` when the axis supports one -- `confidence` -- `reason_codes` -- `summary` -- `evidence_receipt_ids` -- `source_memory_ids` or other durable artefact refs -- `subject_workflow_ids`, `subject_tool_names`, and `subject_concept_ids` -- `counterfactual_recommended_action` when the axis is about abstention, - escalation, or recovery -- `measured_at_utc` - -This contract should sit above individual workflows or prompt variants. -`JVNAUTOSCI-1964` is therefore not mainly a prompt-writing task. It is a -support-surface and represented-workflow-family task. - -### 3. Evaluator Families Should Be Workflow-First and Composable - -Von should not respond to the missing evaluator coverage by turning one larger -Python coordinator into the new policy home. - -The intended structure is: - -1. one shared bounded evidence bundle and locator contract; -2. axis-specific critic or evaluator workflows that consume the same evidence - substrate and shared turn context; -3. durable persistence of axis results into critique memory and benchmark - projections; and -4. promotion, rollback, or remediation consumers that operate over those - machine-usable axis results. - -That means the current postcondition critic should remain one evaluator family, -not the universal critic. - -### 4. The Evidence Bundle Must Grow Stable Answer and Memory Receipts - -The shared evidence bundle should become the canonical input substrate for the -new evaluator families. To do that, it needs stronger first-class receipts for: - -- final answer or user-visible response artefacts; -- answer-support alignment evidence, not only tool or search receipts; -- durable memory refs and task-state refs once `JVNAUTOSCI-1995` lands; -- explicit abstention, refusal, escalation, and follow-up prompts or outputs; -- recovery-route evidence and later-turn continuation evidence; and -- context-lineage or reconstruction receipts showing what the evaluator and the - actor actually saw. - -This is support-surface work, not a justification for new answer-repair policy -in Python. - -### 5. Critique Memory Should Persist Evaluator State, Not Only Verdict State - -`episode_critique_memory` should remain the durable episodic evaluator artefact, -but its persistent contract should expand from scalar verdict state to -multi-axis evaluator state. - -The durable object should preserve: - -- the top-level episode summary and actionable follow-up state; -- axis-level results; -- durable evidence and memory refs; -- revision lineage when a later evaluator run supersedes an earlier one; and -- links to remediation, proposal, and promotion decisions driven by those - results. - -This is the missing bridge between current critic runs and later self-improving -or long-horizon memory work. - -### 6. Benchmark and Audit Surfaces Should Score by Axis - -The current benchmark surface is useful, but it still acts mostly as a verdict -and recurrence audit layer. - -The intended acceptance surface should score and sample by evaluator axis: - -- grounded helpfulness and evidence-answer consistency; -- calibration and abstention quality; -- recovery quality; -- long-horizon task-state reconstruction and continuity; and -- interaction with execution correctness rather than replacement of it. - -Sampled rebuilds should remain, but the reporting layer should stop treating -aggregate success or a single verdict as the main unit of truth. - -## Evaluation Requirements - -The evaluator architecture should be accepted on real or near-real Von paths, -not only on synthetic unit-shaped examples. - -Minimum acceptance dimensions: - -- answer-evidence consistency on direct-response and workflow-answer paths -- correct abstention or follow-up under incomplete evidence -- recovery quality after grounded failure or incomplete execution -- long-horizon task-state reconstruction after interruption or memory update -- explicit disagreement detection between actor claims and evaluator evidence -- durable axis persistence suitable for later proposal or promotion decisions - -Acceptance should combine: - -- targeted replay or nearest-real-path evaluator tests on the live workflow - family; and -- smaller benchmark or sampled-audit sets that score the new axes directly. - -## Short Literature Note - -The architecture direction here was checked against recent primary sources: - -- [Groundedness in Retrieval-augmented Long-form Generation: An Empirical Study](https://arxiv.org/abs/2404.07060) - reinforced that answer correctness alone is too weak when generated sentences - are not actually grounded in retrieved evidence. This supports a first-class - grounded helpfulness axis rather than relying on aggregate answer success. -- [Calibrating the Confidence of Large Language Models by Eliciting Fidelity](https://arxiv.org/abs/2404.02655) - reinforced that confidence should be decomposed and measured, not inferred - from one generic score. -- [BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidence](https://arxiv.org/abs/2604.03216) - reinforced that abstention-aware utility should be treated as a distinct - evaluation problem, especially when overconfident errors are costlier than - underconfidence. -- [Agentic Confidence Calibration](https://arxiv.org/abs/2601.15778) - reinforced that calibration for agents should be process-centric and - trajectory-aware rather than borrowed unchanged from single-turn settings. -- [Process Reward Models for LLM Agents: Practical Framework and Directions](https://arxiv.org/abs/2502.10325) - reinforced that agent improvement needs machine-usable process signals, not - only outcome labels. -- [OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflows](https://arxiv.org/abs/2508.09124) - reinforced that long-horizon evaluation must include history-sensitive, - multi-step tasks rather than only atomic end states. - -These papers did not overturn Von's design direction. They sharpened the need -for explicit evaluator axes, process-level evidence, and long-horizon -acceptance. - -## First Implementation Slices - -`JVNAUTOSCI-1964` now decomposes into the following concrete tasks: - -1. `JVNAUTOSCI-1998` - Implement a canonical multi-axis episode evaluator contract and critique - memory projection expansion across execution correctness, grounded - helpfulness, calibration, recovery, and long-horizon quality. -2. `JVNAUTOSCI-1999` - Implement an authority-backed grounded helpfulness and evidence-answer - consistency critic workflow using the shared evidence bundle and durable - answer receipts. -3. `JVNAUTOSCI-2000` - Implement an authority-backed calibration, abstention, and recovery-quality - evaluator workflow and benchmark surface. -4. `JVNAUTOSCI-2001` - Implement long-horizon task-state reconstruction and memory-conditioned - evaluator support so later memory and benchmark work can score continuity - and interruption handling on stable references. - -These tasks are intentionally not replacements for `JVNAUTOSCI-1963`, -`JVNAUTOSCI-1995`, `JVNAUTOSCI-1996`, or `JVNAUTOSCI-1997`. They are the -concrete continuation of the evaluator line that those broader memory and -self-improvement tasks should now consume. - -## Open Questions Deferred - -This note does not settle everything. The following remain intentionally open -for later implementation work: - -- whether some evaluator axes should support both scalar utility and categorical - pass/fail state, rather than one or the other; -- how much actor-side confidence or abstention should be made explicitly - user-visible versus remaining internal evaluator evidence; -- whether some long-horizon evaluator evidence should be persisted only as - sampled benchmark artefacts rather than every-day critique memory; and -- how quickly current lightweight diagnostics inside - `episode_critic_evidence_service.py` should be replaced by represented - workflow-family outputs. - -Those are follow-on implementation questions, not reasons to keep the -evaluator architecture implicit. diff --git a/docs/engineering/jvnautosci_2011_role_learning_review_2026-04-24.md b/docs/engineering/jvnautosci_2011_role_learning_review_2026-04-24.md deleted file mode 100644 index af0184bd0..000000000 --- a/docs/engineering/jvnautosci_2011_role_learning_review_2026-04-24.md +++ /dev/null @@ -1,265 +0,0 @@ -# JVNAUTOSCI-2011 Role-Learning Design Review - 24 April 2026 - -> **Planning superseded on 4 September 2026:** Use the -> [role-learning convergence guide](role_learning_convergence.md) and live -> `JVNAUTOSCI-2011` for current direction and delivery. The April observations -> below remain historical; they do not require a universal role schema or -> promotion framework before bounded role learning can begin. - -> **Document status: Long-horizon design with a dated implementation snapshot.** -> The role-learning framing may remain useful. Claims about the current -> substrate, gaps, source anchors, or task ordering describe the 24 April 2026 -> evidence base and require live revalidation. - -## Purpose - -This note records the design review that led to `JVNAUTOSCI-2011`, the epic -for minimal-imposition role learning. - -The review question was whether Von's current design documents are sufficient -for a system that can take on otherwise hard-to-fill roles in an organisation -or team, learn how to perform them increasingly well, and improve by building -durable knowledge about workflows, interactions, people, organisations, -processes, work products, notes, artefacts, models, prompts, and skills. - -The short answer is: - -- the architecture direction is right; -- the substrate work is increasingly concrete; -- but the role-learning object was not first-class enough; -- and the near-term SAIL research-operations proving ground needed to be named - more explicitly. - -## Review Scope - -Primary local documents reviewed: - -- `docs/engineering/Von_for_AgenticAI.md` -- `docs/engineering/minimal_imposition_design_principle.md` -- `docs/engineering/minimal_imposition_benchmark_model.md` -- `docs/engineering/agent_memory_and_enduring_knowledge.md` -- `docs/engineering/prompt_programs_and_model_routing_playbook.md` -- `docs/engineering/agent_evaluation_and_research_uptake.md` -- `docs/engineering/von_workflow_language_manual.md` -- `docs/engineering/jvnautosci_1962_enduring_memory_architecture_2026-04-23.md` -- `docs/engineering/jvnautosci_1963_self_improvement_worlds_architecture_2026-04-23.md` -- `docs/engineering/jvnautosci_1964_evaluator_architecture_2026-04-23.md` -- `docs/engineering/agentic_architecture_status_2026-04-23.md` -- `docs/engineering/context_bundle_workspace_contract.md` - -External research and context checked included: - -- Strong AI Lab overview: - -- Cyc platform and publications: - and -- Gendron, Bao, Witbrock, and Dobbie, *Large Language Models Are Not Strong - Abstract Reasoners*: - -- Matuszek, Witbrock, Cabral, and DeOliveira, *Searching for Common Sense: - Populating Cyc from the Web*: - -- Recent agent-memory, self-improvement, and workplace-task evaluation work, - including OccuBench, GDPval, tau-bench, TheAgentCompany, SkillWeaver, and - long-horizon memory surveys or benchmarks. - -The external scan did not overturn Von's direction. It reinforced three -points: explicit knowledge matters, memory and self-improvement need evaluable -promotion loops, and professional-role capability must be evaluated on -realistic work products and processes rather than only on chat answers. - -## Main Finding - -Von's documents are strong as a **neuro-symbolic agent substrate** design, but -weaker as a **role-learning programme** design. - -They already say the right things about: - -- Vontology authority; -- workflows and prompt programmes; -- memory strata; -- evaluator axes; -- candidate worlds and rollback-safe self-improvement; -- model portfolios; -- provenance, telemetry, and namespace discipline; -- avoiding Python-owned semantic policy. - -What was missing was a clear represented object for the thing Von should learn -to do in the near term: perform useful roles for real teams with low burden. - -## Near-Term Focus - -The immediate target should be process improvement and operational -effectiveness for AI research in the Strong AI Lab. - -That focus should not narrow the AGI goal. It should act as the first real -environment in which the AGI-shaped substrate becomes operational: - -1. SAIL AI research operations. -2. Science, discovery, and R&D more broadly. -3. Auxiliary organisational roles. -4. Roles in general. - -This staged path gives the programme a practical spine while preserving the -larger aim: a deployable system that can learn, represent, evaluate, and -improve role performance across settings. - -## Shortfalls - -### 1. Role Performance Was Not First-Class - -The design needs a canonical representation for: - -- role; -- responsibilities and obligations; -- permissions and authority boundaries; -- workflows and process states; -- handoffs and interaction patterns; -- work products and quality standards; -- notes, meetings, artefacts, and decisions; -- evidence receipts and provenance; -- skills, prompts, tools, model choices, and workflow variants; -- success metrics and evaluator outputs. - -Without this, Von risks becoming a set of useful capabilities rather than a -system that can learn an under-filled role and improve at it. - -### 2. SAIL Research Operations Were Too Implicit - -Earlier docs gesture at research-team support but do not make the near-term -operational targets concrete enough. - -Candidate SAIL pilot roles should be treated as examples, not hard-coded -policy. Useful initial examples include: - -- research-project continuity aide; -- literature and paper-representation steward; -- evaluation and replay coordinator; -- meeting, action-note, and follow-up maintainer; -- artefact and knowledge-base curator; -- new-lab-member onboarding assistant; -- grant, reporting, or administrative support assistant. - -Each pilot should identify real workflows, work products, interaction surfaces, -and evidence of burden reduction. - -### 3. Minimal-Imposition Evaluation Needs Role Metrics - -The current minimal-imposition benchmark model is useful but too generic for -role performance. - -Role-learning evaluation should include: - -- avoided repeated explanation; -- reduced coordination burden; -- fewer unnecessary clarification or permission loops; -- work-product acceptance or revision burden; -- workflow continuity across meetings and interruptions; -- recovery cost after failure; -- process disruption; -- appropriate escalation; -- team adoption friction. - -These should be layered onto the existing evaluator and benchmark stack rather -than implemented as a separate scoring system. - -### 4. Memory, Evaluation, and Self-Improvement Need a Role Pipeline - -The current memory, evaluator, and candidate-world tasks are good substrate -work. Role learning should consume them explicitly. - -The intended loop should be: - -1. observe role-relevant episodes, artefacts, interactions, and work products; -2. preserve provenance, namespace, and visibility; -3. query them through the enduring-memory manifest; -4. promote recurring facts, procedures, and guidance into semantic, - procedural, and policy memory; -5. evaluate role performance and work-product quality through multi-axis - evaluator contracts; -6. create candidate workflow, prompt, skill, routing, or memory improvements; -7. test candidate improvements in bounded worlds; -8. publish only after evaluator-backed promotion gates. - -### 5. Multi-Agent Coordination Needs Organisational Semantics - -The multi-agent coordination note currently focuses on delegation, -capability negotiation, and shared episodic context. Role learning also needs: - -- human-agent handoff patterns; -- organisational authority and accountability; -- shared team state; -- arbitration when agents or people disagree; -- role-specific escalation policy; -- traceable ownership of delegated subtasks and artefacts. - -This should be represented in Vontology and workflows, not hidden in -orchestration code. - -### 6. Role Policy Must Not Become Python Heuristics - -The same anti-patterns that apply to routing and recommendation apply to role -learning. - -Design smells include: - -- role-name conditionals in Python; -- SAIL-specific branches in orchestration code; -- persona-only prompts with no represented responsibilities, workflows, or - work-product contracts; -- work-product grading hidden in string matching; -- opaque "agent memory" that cannot explain evidence, scope, or revision. - -Role learning should be represented and evaluated, not simulated through a -larger prompt. - -## Recommended Document Changes - -The design documents should be updated in three layers. - -1. `Von_for_AgenticAI.md` - Add role learning as the near-term proving ground, with SAIL research - operations first and broader science/R&D, auxiliary roles, and general roles - as later stages. - -2. `minimal_imposition_benchmark_model.md` - Add role-performance/process-improvement dimensions once the evaluator - substrate can support them. - -3. Memory, evaluator, and self-improvement notes - Cross-link role learning to `JVNAUTOSCI-1995` through `JVNAUTOSCI-2005` so - role learning consumes the shared substrate rather than creating a parallel - stack. - -The first update has been started under `JVNAUTOSCI-2012` by updating -`docs/engineering/Von_for_AgenticAI.md` and adding this review note. - -## Jira Outcome - -Created: - -- `JVNAUTOSCI-2011`: Minimal-imposition role learning for SAIL research - operations and generalisable work roles. -- `JVNAUTOSCI-2012`: Update Von design framing for minimal-imposition role - learning and SAIL research operations. - -`JVNAUTOSCI-2011` is linked to `JVNAUTOSCI-1960`. The relationship is: - -- `JVNAUTOSCI-1960` remains the broad minimal-imposition agentic AI substrate - programme. -- `JVNAUTOSCI-2011` is the role-learning and SAIL research-operations - application spine that consumes that substrate. - -## Acceptance Direction - -The next design and implementation work should be accepted only when it shows: - -- real SAIL research-operations roles and work products, not only abstract - personas; -- represented role artefacts in Vontology/workflow/prompt/KB surfaces; -- minimal-imposition evidence that the system reduced burden or improved - operational effectiveness; -- evaluator-backed learning from role episodes; -- candidate-world or equivalent gated promotion before role-policy changes - become canonical; -- no durable role policy hidden in Python. diff --git a/docs/engineering/jvnautosci_2391_mongo_hot_path_audit.md b/docs/engineering/jvnautosci_2391_mongo_hot_path_audit.md deleted file mode 100644 index 7b95dce05..000000000 --- a/docs/engineering/jvnautosci_2391_mongo_hot_path_audit.md +++ /dev/null @@ -1,153 +0,0 @@ -# JVNAUTOSCI-2391 Mongo Hot-Path Attribution Audit - -Date: 2026-06-03 - -## Purpose - -JVNAUTOSCI-2391 is the evidence-baseline slice of JVNAUTOSCI-2388. Its -purpose is to make MongoDB cost and latency attributable to Von routes and -service calls before later subtasks optimise storage, hydration, polling, and -guardrails. - -This is support-surface work only. It must not reduce retained history, -telemetry, LLM interaction data, tool evidence, workflow evidence, or replay -inspectability. - -## Staleness Review - -The parent issue JVNAUTOSCI-2388 was created after several adjacent fixes -landed. Current code already includes: - -- JVNAUTOSCI-2382: async local spillway and migrator for durable blob writes. -- JVNAUTOSCI-2383: request-thread starvation and hot-poller mitigation, - including the `/api/settings/db/info` TTL cache and workflow definitions - stale-while-refresh cache. -- JVNAUTOSCI-2389: local-first debug blob cache for turn history/debug blobs. -- JVNAUTOSCI-2314: bounded durable workflow checkpoint context. - -So this task should not reimplement storage offload. The current useful gap is -attribution: when Atlas profiler or local logs show a costly operation, future -agents need to know which route/service operation caused it. - -## Observed Evidence - -The parent issue records these concrete symptoms: - -- A history/diagnostic path for - `request_id=dd7ff7b0-6749-43bb-a3f3-14625da10af1` returned the useful user - answer, but later diagnostic/history loading spent tens of seconds hydrating - debug blobs and touching object-store metadata. -- Logs showed `GET von.history took 47455.8ms`. -- JVNAUTOSCI-2383 recorded hundreds of `/api/settings/db/info` calls in a - single log window, plus slow `history_sessions`, `rag_status`, - `get_all_settings`, and capability-index/status paths. -- JVNAUTOSCI-2307 remains the dedicated Atlas query-targeting alert task; this - audit should feed it route/service labels rather than replacing index work. - -## Implemented Attribution Surface - -`src/backend/services/mongo_observability_service.py` now provides: - -- PyMongo/Atlas query comments with safe route, endpoint, service, collection, - and operation labels. -- Lightweight in-process counters for operation count, failures, slow - operation count, total/average/max elapsed time, and last error type. -- Environment flags: - - `VON_MONGO_QUERY_ATTRIBUTION_ENABLED` defaults to `true`. - - `VON_MONGO_OPERATION_AUDIT_ENABLED` defaults to `true`. - - `VON_MONGO_OPERATION_AUDIT_SLOW_MS` defaults to `500`. - -The comments deliberately avoid query bodies, document bodies, prompt text, -email bodies, raw debug payloads, Mongo URIs, credentials, and secrets. Flask -route metadata uses `request.path`, not the query string. - -## Labelled Hot Paths - -### Chat History - -`src/backend/services/chat_history_service.py` now labels the existing bounded -read wrappers: - -- `get_chat_history.find_session` -- `get_chat_history_segments.aggregate_tail` -- `get_chat_history_segments.tail_fallback` -- `get_chat_history_segments.find_session` -- `get_chat_history_debug_entry.slice_entry` -- `get_chat_history_length.aggregate` -- `get_chat_history_session_count.aggregate` -- `get_chat_history_session_summaries.metadata_find` -- `get_chat_history_session_summaries.full_find` -- `has_chat_history_session.exists` -- `get_chat_history_session_summary.metadata_find` -- `get_chat_history_session_summary.full_find` -- `get_chat_history_session_summary.metadata_fallback` - -These cover the main ordinary-history, session-list, and explicit-debug read -surfaces implicated in the `von.history` latency incident. - -### Turn Execution Records - -`src/backend/services/turn_execution_record_service.py` now labels: - -- `upsert_turn_execution_record_projection.update_one` -- `get_latest_turn_execution_record_projection.find_latest` -- `get_latest_turn_execution_record_projection.fallback_find_latest` - -These cover the diagnostic projection writes and latest-record reads that -connect chat history to turn-execution evidence. - -### Settings / DB Info - -`src/backend/server/routes/settings_routes.py` labels the live Mongo ping in -`/api/settings/db/info`: - -- `get_db_location_info.ping` - -`src/backend/services/settings_service.py` labels application-settings reads -and writes: - -- `get_setting.find_one` -- `get_settings_batch.find` -- `update_setting.precheck_find_one` -- `update_setting.update_one` - -These cover the named `get_all_settings`/settings hot path and the repeated DB -info poller evidence from JVNAUTOSCI-2383. - -## Remaining Audit Targets - -The following areas are intentionally left for follow-on implementation tasks -because they require more targeted measurement or behaviour changes: - -- JVNAUTOSCI-2392: ordinary history and Thinking-card reads should remain - compact unless diagnostics are explicitly requested. -- JVNAUTOSCI-2393: poller coalescing/caching for RAG status, capability-index - status, settings/status, main health, and frontend visibility-aware loops. -- JVNAUTOSCI-2394: durable workflow checkpoint and turn-record storage pressure. -- JVNAUTOSCI-2395: operator-visible guardrails and diagnostic views for runaway - local workloads. -- JVNAUTOSCI-2307: index/query-targeting fixes using Atlas profiler evidence. - -Workflow-use episode queries and capability-index status should be revisited in -JVNAUTOSCI-2393 after polling frequency is measured. They are plausible cost -surfaces, but the immediate 2391 patch focuses on the already-evidenced -history, settings, and turn-record paths. - -## Recommended Next Order - -1. JVNAUTOSCI-2392, because it addresses the direct `GET von.history took - 47455.8ms` class of latency and hydration amplification. -2. JVNAUTOSCI-2393, because repeated small reads can generate cost while a - browser is idle. -3. JVNAUTOSCI-2395, because once the labels and poller changes exist, Von can - surface early warnings for expensive local workloads. -4. JVNAUTOSCI-2394, unless new evidence from 2391/2392 shows workflow/TER - storage pressure is the immediate largest contributor. -5. JVNAUTOSCI-2396, after the implementation subtasks can be measured together. - -## Evidence Gap - -This pass adds route/service attribution and local counters but does not access -Atlas profiler data directly. When Atlas profiler access is available, filter -for query comments containing `jira: JVNAUTOSCI-2391` and group by -`service`, `operation`, and `route.endpoint`. diff --git a/docs/engineering/jvnautosci_2720_learning_cycle_2026-09-05.md b/docs/engineering/jvnautosci_2720_learning_cycle_2026-09-05.md deleted file mode 100644 index ae4d0ed35..000000000 --- a/docs/engineering/jvnautosci_2720_learning_cycle_2026-09-05.md +++ /dev/null @@ -1,151 +0,0 @@ -# JVNAUTOSCI-2720: completed negative learning cycle - -- **Kind:** Experiment dossier -- **Lifecycle:** Frozen -- **Authority:** Evidence only; delivery status belongs to - [JVNAUTOSCI-2720](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-2720) -- **Evidence date:** 5 September 2026 UTC -- **Scope:** One actor-visible message-channel practice, one model and one - frozen read world; not general organisational competence - -## Finding - -Von authored a candidate from the designated real discussion, exposed its exact -revision in later independent experimental turns, evaluated the resulting work, -and recorded a negative use disposition from the canonical evidence. Two fresh -subsequent turns verified that the rejected revision was absent. The source, -candidate text and contrary evidence remain inspectable; rejection does not -erase the lesson or assert that every part of it is false. - -Advice B did not earn use over the no-advice baseline A. Both passed 5 of 8 -applicable trials; B had three paired improvements and three regressions there. -All 24 trials and evaluations completed, with 12 valid, comparable pairs and no -inconclusive evaluations. No represented Arm C or ordinary automatic advice -consumer was activated. This closes the issue's negative-evidence path, not its -conditional positive path or a claim of improved task performance. - -## Exact evidence and inspection - -- Run: `#V#jvnautosci_2720_learning_advice_run_9f3584c89db54ccabdc3bc0c`. -- Candidate: `#V#learning_candidate_b0fda0e1f472b895d5ee5aab5a412251`, revision 1. -- Body SHA-256: `6e2d47acba92877acd1c2f80fc0917e5d8e76bb2c8a0e73875d275159aa785b4`. -- Revision identity: `7f6ea87ac259aabd01f3e3eb13613acf1b0b907cbf73b993334290b316648cb7`. -- Source locator SHA-256: `309b70ab7bd0606382c2f984790de69698b8aa56f21bde3d4a464789fccf66f2`. -- Manifest SHA-256: `a4c2d581f60c1b603d39a2edabe3b10d1370d02d1318ae2befd9e6aad7bc0a3e`. -- Plan SHA-256: `036d688c7e7896817c84526c0a2b8e7c904ec61aeb2f7e2781f6ccf18c102f00`. -- Result SHA-256: `fe0ce020c7e627dd29786aca50cd3cfbb624d4b4338199f4af950c5cfc00bbdb`. -- Acting/evaluation revision: `aceb1d4bd82c72c2c96483a7f43dc2ddfccef196`. -- Disposition repair and later checks: `93a883fd`. - -The source is the user-designated “Testing Von messaging” session -`1eec920c-0be0-4fcf-a045-6733768c8667`; the -[frozen fixture](../../scripts/fixtures/jvnautosci_2720_message_channel_experiment.json) -identifies the exact selected messages, trusted scope, cases and criteria. -Formation used those messages only, before candidate efficacy was observed. -Contributor, actor, organisation, target, audience and beneficiary are separate; -the target is communicating and beneficiaries are the person and household. -No broader purpose or authority was invented. - -Use the existing actor-scoped `learning_candidate_get` and `experiment_run_get` -surfaces for the candidate body/provenance and individual observations, or the -[live-cycle reader](../../scripts/run_jvnautosci_2720_learning_cycle.py) with -`--readback --run-id` and the exact run above in the authorised project -environment. Private source and model-visible message bodies are not duplicated -in this public dossier. The run retains 24 trial observations, their execution -record locators, evaluator provenance and one final result observation. Generic -run verdict `partial` is distinct from the decisive content verdict -`does_not_support_use`. - -## Individual paired outcomes - -Each row is one case/repeat, containing one independently executed trial per -arm. Case contracts and scoring thresholds were unchanged across the repairs. - -| Case | Repeat | A | B | -| --- | --- | --- | --- | -| Household Von catch-up | 1 | pass | pass | -| Household Von catch-up | 2 | fail | pass | -| Task switch from Gmail | 1 | partial | pass | -| Task switch from Gmail | 2 | pass | pass | -| Channel-neutral digest | 1 | partial | pass | -| Channel-neutral digest | 2 | pass | partial | -| Channel-neutral brief | 1 | pass | partial | -| Channel-neutral brief | 2 | pass | partial | -| Explicit Gmail control | 1 | partial | partial | -| Explicit Gmail control | 2 | partial | partial | -| Explicit Von direct control | 1 | pass | pass | -| Explicit Von direct control | 2 | pass | fail | - -Gmail was a frozen typed dependency failure, while the authorised direct-message -snapshot contained five messages. Neutral clarification endpoints were partial -under the predeclared rubric: three in B and one in A. A's household failure -selected Gmail first but recovered to a grounded final work product; its task -switch partial reported a failed direct read. Capability choice and work-product -dimensions remain separately visible in the observations. - -The B-only control failure was labelled `wrong_namespace_scope` by the semantic -evaluator. It is not independently demonstrated unauthorised access: successful -replay reads required the exact pre-authorised request and returned its frozen -result. Do not infer a privacy incident or add a compulsory control from this -label. Even if that control judgement were set aside, B independently missed -the required six applicable passes, two-pass improvement, and excess of paired -improvements over regressions. The bounded non-use decision does not depend on -resolving that evaluator judgement. The recorded scores were not rewritten. - -## Disposition and subsequent use - -The canonical candidate service recomputed the result from its bound trials and -recorded `rejected` / `does_not_support_use` against the unchanged revision. -Disposition identity: -`4f836bfb5e46df8a684e756c69cbb928d7e953289fb33d3f518cca2eab72a94f`. - -The canonical projection path then reported `learning_advice_candidate_ineligible` -because of that rejection. Actual later provider requests contained none of the -rejected body or identifiers, rather than merely reporting an empty sidecar. - -| Later check | Execution record | Observed requests | Advice exposures | -| --- | --- | --- | --- | -| Trigger | `jvnautosci-2720-post-trigger-1f16190e7d9e810265847cea` | 7 | 0 | -| Neighbouring Gmail control | `jvnautosci-2720-post-control-cf82feb4a6cee1a529e88380` | 8 | 0 | - -Canonical execution-record SHA-256 values are respectively -`e976355964cf32a5da682f2f3d86d4c53e6133eb236c9732c1975be51ed7f5d3` and -`8041934e517a071663c4c82198b7dd14b5124c34011d8d49c78608e47c16b60b`. -The resumed command returned `completed_negative_learning_cycle` with -`rejected_revision_absence_verified`. It did not repeat the A/B comparison. - -## Method, repairs and limitations - -Three earlier complete comparisons were inconclusive because the evaluator -self-invalidated on common metadata or lacked evidence the acting model saw. -Those records remain unchanged. The final correction removed a lossy evidence -whitelist and silent truncation, and left model/exposure/contamination checks to -the existing harness. Four synthetic live calibrations distinguished grounded -output, fabrication, dependency-limited reporting and focused clarification. -Calibration was not counted as learning efficacy. - -The final comparison initially could not record disposition: its reader omitted -the schema version from the evaluator-runtime hash preimage. A shared identity -builder corrected this interface mismatch without changing the producer's -identity bytes, trial scores, decision thresholds or stored result. The exact -canonical disposition operation was retried, followed by the existing later-use -checks. This repair required no new acting comparison or retrospective rescoring. - -All acting and evaluating calls used fixed allowed `gpt-5.6-luna`; requested, -selected, effective and provider-observed identities were checked separately. -The final affected regression suite passed 544 tests using the isolated mock -test database. This complements, rather than substitutes for, the live records. - -Mean acting-turn latency was 19.68 s for A and 13.90 s for B; model-call totals -were 81 and 66. These exclude acquisition, evaluation and persistence overhead, -and B more often ended with clarification. They do not establish a useful -speedup. Token/cost aggregation remains unavailable, not zero. - -This was an operator-run, stripped read-only replay, not byte-equivalent to -ordinary production. Represented discovery/projection were disabled equally. -The small, visible, repeatedly exercised decision set is not an untouched -statistical holdout. The result supports a bounded decision about this revision -under this configuration, not that the lesson is universally false or that -advice cannot help. [JVNAUTOSCI-2721](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-2721) -owns the separate longitudinal role pilot; it can reuse the delivered candidate, -evidence and disposition surfaces without assuming positive advice efficacy. diff --git a/docs/engineering/jvnautosci_2721_continuity_2026-09-05.md b/docs/engineering/jvnautosci_2721_continuity_2026-09-05.md deleted file mode 100644 index 2f0ce1dce..000000000 --- a/docs/engineering/jvnautosci_2721_continuity_2026-09-05.md +++ /dev/null @@ -1,153 +0,0 @@ -# Continuing responsibility: bounded integration evidence - -- **Kind:** Implementation and developmental evidence record -- **Lifecycle:** Frozen -- **Authority:** Evidence only; current selection and delivery status live in [JVNAUTOSCI-2721](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-2721) -- **Owner:** Michael Witbrock -- **Evidence date:** 5 September 2026 UTC -- **Review trigger:** Changed workflow, prompt, tool contract or model; prospective organisational use -- **Design direction:** [Role-learning convergence](role_learning_convergence.md) - -This increment connects a saved responsibility to fresh evidence, revised -conclusions and a verified updated work product. It extends the existing -`#V#lab_project_status_digest_workflow`; it does not create a second role or -learning platform. It does not establish real lab adoption, learned benefit, -cross-domain transfer or autonomous monitoring by a deployed worker. - -## Findings and changes - -The inspected baseline was `00f9acb5`, including completed 2720. That experiment -rejected its tested advice candidate and verified subsequent absence. This -increment preserves that result: it neither activates the candidate nor -introduces an ordinary advice consumer. - -Four concrete integration problems impeded continuing responsibility: - -1. The digest always queried the latest 12 JVNAUTOSCI issues and used one fixed - work-product ID. Callers could not select separate responsibilities; an older - unresolved issue could fall outside the evidence window. -2. Prior-digest reads disabled Markdown reconstruction. With text stored in - `hasContent` relations, the next encounter did not receive the saved text. -3. The content tool description explicitly recommended the metadata-only mode - to avoid masking missing Markdown. A real model followed that advice and - reported readable stored sources as empty. -4. Completion checked for nonempty read-back, rather than the exact written - content. Connecting the existing exact-text verifier also exposed a writer/ - verifier disagreement between `hasContent` and its equivalent concept ID. - -The release seed now accepts an existing authorised `work_product_id`, an -initial `jira_jql`, and `evidence_sources` describing source locators and their -purpose. Omission preserves the old work-product and Jira-query defaults. -An initial scope state uses the existing context-set primitive. Both foreground -and scheduled launches use the same inputs and capability. - -The existing tool-capable model step can investigate material source references, -older obligations and missing facts. It has read capabilities; the deterministic -workflow owns the selected work-product write. Prompt output cannot redirect -that write to another product. The prompt preserves commitment identity across -date/owner changes, distinguishes internal/external deadlines and estimates/ -approvals, and asks for refreshed evidence before a premise is repeated as -current. Unavailable evidence must remain qualified, not silently promoted. - -Persistence uses the same actor-bound canonical singleton-text service and the -existing exact-text verifier. The only verifier algorithm change recognises -equivalent core predicate IDs; unrelated predicate mismatches remain invalid. -A new continuity prompt identity preserves locally authored old prompt text. -Seed version 8 recognises the reviewed version-7 authority digest; upgrade and -repeat-bootstrap tests read back the resulting authority and preserve existing -digest text. This is a release input, not evidence of shared runtime activation. - -## Exercised path and observations - -The [development rehearsal](role_convergence_rehearsal.md) runs the actual -represented workflow, model/tool loop and canonical text services in an -in-process mock database. Jira and repository sources are fictional adapters. -Each encounter starts afresh, with source locators and the saved work product, -without a conversation transcript or future-event text. - -The [evidence extract](../generated/role_continuity_2026-09-05.json) retains exact -canonical brief text, retrieved source IDs, source/asset digests, model identity -triples, per-call reported token usage and individual elapsed times. Review was -Codex inspection of those artefacts and sources, not a blind efficacy evaluator. - -| Encounter | Observed outcome after the repair | -|---|---| -| A | Reads call, office and project sources; distinguishes 23 October internal and 30 October external deadlines; retains open TASK-1 and missing compute cost/statement | -| B | Recovers the previous brief; reads office-v2; revises the internal deadline to 20 October while retaining 30 October externally and the same open commitment | -| C | Retrieves the newly available 8,000-unit quote; revises the cost/readiness conclusion, retains missing budget/statement work and distinguishes the quote from approval or spending authority | - -The corrected path passed A–C once with GPT-5.5 and once with Luna. All six -encounters passed exact canonical saved-text read-back. GPT-5.5 explicitly -identified the old project source's compute-cost unknown as stale/conflicting -with the quote. The partial empty Jira window did not resolve the obligation. -Only the selected internal brief was written; no external systems or task -statuses were changed. - -A further GPT-5.5 A–C sequence joined B to the existing scheduler, worker claim, -production definition loader, actor binding and terminal-state persistence. -The worker completed the changed-deadline brief and the canonical instance was -`completed`; a second due-schedule poll left exactly one occurrence. C then -recovered that saved brief and incorporated the quote. This sequence took -55.93 seconds and reported 142,087 total tokens across calls. It establishes -the isolated scheduled path, not unattended operation in a running lab. - -These are developmental observations, not an estimated success rate. Initial -Luna output retained the superseded internal deadline; initial GPT-5.5 output -followed the misleading metadata-only tool description. Both failed work -products remain in the extract. Earlier preflights encountered incompatible -model/API parameters and missing standard backfill prompt fixtures. An Astra -attempt failed at the API interface and supplies no model-quality comparison. -The successful calls used the recorded Chat Completions surface with reasoning -set to `none`; requesting another surface did not itself establish its use. - -A later Luna sequence omitted office/project reads and honestly labelled the -corresponding knowledge unavailable. It therefore failed full A–C coverage, -despite exact persistence. This partial result remains in the extract. Joined -rehearsal setup also exposed harness mistakes: absent actor binding, an -executor-only invocation leaving its instance pending, and an incompatible -definition-loader signature. The final run uses the existing worker and its -production loader, rather than manually declaring completion. No production -worker patch was needed. These observations limit any reliability claim; -successful developmental runs are not a claim that every recurrence succeeds. - -The foreground GPT-5.5 sequence took 54.38 seconds; Luna took 37.06 seconds. -Reported per-call total tokens sum to 158,663 and 129,628 respectively. These -are substantial context costs for small briefs, not a speed or cost advantage -claim; monetary cost is unavailable. Setup, fixture repair, prompt/tool-contract -repair and review effort are excluded from those execution times. No human -reply was required during either final three-encounter sequence. The briefs -remain repetitive and sometimes defer decisions a capable manager could make; -that limits the low-imposition claim. - -## Verification and limits - -The targeted execution, seed, verifier and MCP-action suites passed 34 tests. -They include separate work products, actual prior-text recovery, changed text -replacement, exact stale-readback rejection, preservation of an unrelated -product, denial of another actor's private product, model-reported target-ID -mismatch, reviewed seed upgrade, preserved old prompt and digest content, and -a due schedule claimed and completed by the existing worker, with canonical -terminal read-back and no duplicate occurrence on a second poll. -Scripted synthesis in these tests proves integration contracts, not reasoning. - -A broader run had 119 passes and 11 worker-claim failures. The identical 11 -failures reproduce on archived unchanged `00f9acb5` with this machine's -`VON_DURABLE_MIN_WORKER_BUILD=durable_exact_workflow_authority_snapshot.v1`. -Without that local setting the archived suite passes all 97 tests. The failures -are a baseline/configuration interaction, not a regression caused by this -candidate; they are not repaired or hidden by changing the user's configuration. - -The work-product concept must already exist and the actor must have its actual -write authority. A source ID, role description or schedule input grants none. -The shared workflow, prompt, model settings, database and running services were -not activated or changed by these isolated rehearsals. A real operator/source -selection, deployed worker operation and interruption/resumption evidence remain -separate claims. Initial deterministic source-tool failures still stop this -workflow; the exercised incomplete/contradictory-source case was a returned -partial source window and conflicting content, not a connector outage. - -For the next missing connection and current delivery decision, use the live -2721 task rather than turning this dated record into a second implementation -plan. Learned reuse must use a fitting source-derived candidate and the 2720 -outcome/disposition protocol; this continuity result is not evidence that the -rejected candidate should be restored. diff --git a/docs/engineering/jvnautosci_2721_prospective_readiness_2026-09-05.md b/docs/engineering/jvnautosci_2721_prospective_readiness_2026-09-05.md deleted file mode 100644 index e51f1604f..000000000 --- a/docs/engineering/jvnautosci_2721_prospective_readiness_2026-09-05.md +++ /dev/null @@ -1,273 +0,0 @@ -# First prospective replay-readiness responsibility - -- **Kind:** Dated execution and evidence record -- **Lifecycle:** Frozen evidence record -- **Authority:** Observations and interpretation; not runtime policy or certification -- **Owner:** Michael Witbrock through [JVNAUTOSCI-2721](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-2721) -- **Evidence date:** 5 September 2026 UTC -- **Live selection:** 2721 and [the bounded programme](role_convergence_test_programme.md) - -## Scope and prior state - -Michael authorised refining the programme and beginning execution. The real -responsibility is maintaining readiness of Von's replay and reliability work -for Michael. Codex selected the initial sources, issued the encounters through -Michael's normal authenticated Chrome session, and independently reviewed the -results. That facilitation is not autonomous Von initiative or beneficiary -adoption. - -The existing runtime at port 5001 was clean release -`38afefd252071274a497bdc9b9be41982c4ca9e1`, including 2722's owner-bound Jira reads. -Health and worker readiness were checked. No production code or represented -prompt was changed for these encounters. Luna is the principal model; Sol is -excluded. Private diagnostic envelopes remain outside the repository. - -The frozen portfolio review supplied sequencing advice. The programme and -live Jira were refined around an initial brief, unchanged fresh encounter, -the complete 2568 family, its actual published outcome, and one scheduled -maintenance checkpoint. Later learned efficacy and transfer remain separate -claims. 2720's rejected candidate is not reactivated. - -## Encounter A: establish the responsibility - -The prompt supplied the job, exact proposed product name, seven Jira issue -identities (2721, 2722, 2720, 2568, 2577, 2578 and 2590), and authority to -maintain one internal product. It did not supply an answer or prescribed -recommendation. Jira and other products were to remain unchanged; scheduling -was explicitly excluded from this encounter. Source and outcome expectations -were retained before submission. Jira refinement was a rolling acquisition: -2568's updated description was saved just after submission, before source reads. - -Von created `#V#von_replay_and_reliability_readiness_michael_2721`, a -`#V#work_document` with Michael as owner and user-specific visibility. The -creation, owner relation and `hasContent` write have successful effect receipts; -the turn fetched the concept and text again. An independent actor-scoped -canonical service read recovered the exact 5,492-character text and matching -write-receipt SHA-256: -`2cee7f3f7c6691660cae0819deb80d25c6017370d17b34edfe26ad69df18556f`. -The canonical text relation is `6a9c167cb312f5fd99b0ceab`. - -The brief links its sources, distinguishes 2722's delivered read path from -unproved broader capabilities, and selects the four-turn Jira family. However, -it imports the real critic, durable worker and independent-case requirements -from broader certification work into that ordinary conversational replay. It -also describes 2720 as a completed learning-cycle dossier without preserving -the material fact that the candidate was rejected. These are judgement and -knowledge-maintenance deficits; a persisted, source-linked brief alone does -not pass the intended programme-management claim. - -The UI elapsed time was 7 minutes 20 seconds. Canonical turn timing records -405.241 seconds, so the UI and recorded execution spans are reported separately. -There were 136 tool invocations: 14 Jira reads, 100 `turn_read_evidence` calls, -and 22 discovery, creation, relationship, text and read-back calls. All 100 -evidence reads returned `ok`; they included repeated field and keyword -projections of the acquired issue/comment payloads. This is observed retrieval -burden, not evidence that all those reads were redundant or that Atlas alone -caused the latency. The recorded totals include 156.908 seconds in tools and -198.339 seconds in model calls; overlapping totals are not added. - -Nineteen model calls were recorded. The UI reported 1,060,153 input tokens, -15,458 output tokens and a known cost subtotal of US$0.119615, with partial -coverage. Missing auxiliary usage is not zero cost. No Jira write, import, -reconciliation, schedule creation or mutation of another work product appears -in the executed-tool record. - -Locator: session `a1fb157e-a4b2-4bd7-b45b-ddb21acff5fd`, request -`ea086bbd-a3d8-4f50-bd2f-520a7deb942f`, assistant history index 137. - -**Merge decision does not change:** the programme and honest initial evidence -can be published; competence, efficient routine maintenance and beneficial -learning remain unproved. Do not add a semantic code restriction merely to -make this brief match an authored answer. - -## Encounter B: recover in a fresh conversation - -The second prompt named the continuing responsibility, asked Von to recover -the saved product and check current evidence, and permitted only that product's -maintenance. It supplied neither A's transcript nor an authored correction. -The A receipt and persisted trace were retained before B; the separate full -canonical text read, made while B was underway, matched A's write hash. - -Von recovered the same product and read current Jira sources. The source set -was not unchanged: 2577's canonical `updated` timestamp is -`2026-09-06T01:13:34.521+1200` (5 September 13:13 UTC, during A), and its current -status is SUPERSEDED. B recognised that supersession and acquired its successor -2579. It also found newer 2578 evidence which predated both encounters but had -been missed in A. Thus this is useful recovery and source reconciliation, not -a clean no-change control or proof of autonomous noticing. B retained the -over-broad recommended replay prerequisites. - -B reported an update and successful read-back. Independent canonical reads -found **two** active `hasContent` relations: A's text remained alongside B's -6,411-character revision, relation `6a9c1aacb312f5fd99b0ceda`, SHA-256 -`b2bf391e1bd214d64c8ef4fceab269914bff28dd5d6134e50b9211b0d12e0edb`. -The normal adaptive turn used `upsert_text_relation`, whose text-specific -upsert does not replace a prior different text. The existing digest workflow -instead selects `upsert_singleton_text_relation`. Whether subsequent consumers -receive an unambiguous current brief must be checked; the two rows are not -silently described as a completed replacement. - -UI elapsed time was 3 minutes 39 seconds; recorded elapsed time was 209.403 -seconds. There were 27 tool calls and 14 model calls: 13 provider-observed Luna -calls plus one Luna-requested auxiliary call with missing observed identity -and usage. Known cost subtotal was US$0.042780 with partial coverage. A likewise -had 18 provider-observed Luna calls and one partially observed auxiliary call. -The two encounters were sequential. They are not a matched architectural or -model comparison: setup and maintenance have different jobs and evidence. - -Locator: session `b9e97d77-32aa-4f3f-aae0-f3d67c3026a0`, request -`2fd4ccbc-4cb6-4a3b-bb1c-56ab9e3ed6f6`, assistant history index 28. - -## Four-turn Jira family - -The exact four prompts came from `jira_lookup_followup_family` in the existing -prompt bank, SHA-256 -`601e8a729c9797553bc7f1f29dd3558315482115f61ca0e1217d9b9aec9a9ef6`. -They ask for the newest Task, its represented concept, full direct Jira details, -and separate summaries of that issue and its predecessor. Normal owner-issued -browser turns were used; no fixture identity or imported browser credential -substituted for the supported owner binding. - -Canonical Jira queries before and during the family identified 2722 (Done) -and predecessor 2721 (In Progress). Issue contents remained live; the user was -also editing Jira and explicitly confirmed that concurrent work. The two model -arms are sequential operational observations with the same actor, prompts and -deployed code, not a frozen-source, randomised model benchmark. - -### Luna - -The persisted answers carry the correct issue identity across all four turns, -give Michael as assignee and the full 2722 description, and separately describe -2721 and 2722. Turn 3 used an exact Jira search with the requested fields; a -particular tool name is not necessary for the direct-source outcome. -Turn 2 performed exact key and full-title lookups with zero matches, plus name -resolution and a broader partial search. Its retained answer says a represented -concept was not found and is not currently verified, rather than claiming an -exhaustive proof of absence. No import or other semantic effect was executed. - -However, **turn 2 failed visibly**: the browser displayed “The final response -did not return from Von.” The exact server record subsequently contained a -completed 424-character answer, no effects and `turn_output_health.status=ok`. -The copied browser diagnostic instead had `status=error` at 110.741 seconds. -This is an answer-delivery/recovery discrepancy, not evidence that Luna failed -to answer internally. The later prompts continued in the same conversation -after the failure was retained; no retry was disguised as the original turn. - -Current `chatTab.js` catches a failed generate transport, reports an error, and -then stops foreground result polling in `finally`. That is a reachable reason -why the existing alternate result route may not recover a late persisted -answer. The precise underlying transport exception was not captured. The -observed discrepancy justifies a focused recovery investigation; it does not -justify blaming model capability or changing semantic reasoning policy. - -The existing evaluator reports four canonical-record passes, including its -legacy obligation checks. Those checks do not verify browser delivery and -accept empty legacy obligation fields. They are supporting diagnostics only: -**the user-visible Luna family does not pass**. No model promotion or ordinary -four-turn reliability claim follows from its four machine passes. - -| Turn | Request | History index | Recorded elapsed | UI elapsed/outcome | -| --- | --- | ---: | ---: | --- | -| 1 | `5ed2c659-a6db-4744-964c-4163e42304dc` | 4 | 50.585 s | 54 s; answer delivered | -| 2 | `359677d2-07d7-4d7c-a258-7d1dc7eb7937` | 13 | 109.236 s | 110.741 s; error displayed | -| 3 | `97b20653-95b6-4e63-ba7e-1073b81193de` | 20 | 133.071 s | 155 s; answer delivered | -| 4 | `23701987-5211-4440-8f07-39f4188fd3a6` | 27 | 75.434 s | 80 s; answer delivered | - -Session: `1aa92c43-14c9-4b4d-9232-4e2a441ea5d9`. All 21 fully observed calls were -Luna; four auxiliary calls requested/selected Luna with missing observed -identity and usage. The known family cost subtotal is partial. There was a -substantial wall-clock gap between turns 2 and 3; elapsed measurements describe -individual turns, not a continuous unattended family or clean latency ranking. - -### Terra and the bounded comparison - -Terra delivered all four answers and retained the same issue reference. Exact -key/title searches supported its qualified not-found answer. Turn 3 retrieved -2722's full description and assignee directly from Jira; turn 4 returned the -two actual target issues separately. No semantic writes occurred. The complete -Terra family passes this bounded observed case; it does not establish a -population reliability rate, an architectural advantage or a routing rule. - -| Turn | Request | History index | Recorded elapsed | UI elapsed | -| --- | --- | ---: | ---: | ---: | -| 1 | `bd03d86f-4f5f-44ab-a007-07eb5eb5d236` | 4 | 54.517 s | 58 s | -| 2 | `3b2409a8-4526-453d-94d9-541db3a8b481` | 10 | 52.730 s | 56 s | -| 3 | `a1902362-5d25-456a-bf16-c88bcd8ad0c3` | 14 | 43.599 s | 47 s | -| 4 | `0319faa4-857c-48d9-84a0-4d7a0212c9d2` | 20 | 48.172 s | 62 s | - -Session: `9b69af39-2a26-4a1e-9f9e-38cfbb7aeb08`. Sixteen provider-observed calls -were Terra; four auxiliary calls requested/selected Terra but lacked observed -identity and usage. The existing canonical evaluator again gave four passes, -supplemented here by source, effect and delivered-answer review. - -Known family cost subtotals were **US$0.040571 for Luna** and **US$0.2804134 for -Terra**, both partial telemetry estimates. Terra used two representation lookup -calls to Luna's four, and its answer arrived. Luna also produced a supported -answer internally. This sample does not identify an inadequate Luna reasoning -capability or justify replacing the principal model. The browser preference -was restored to Luna after the pair. All observed calls used the intended -non-Sol model; unobserved auxiliary identity remains explicitly unavailable. - -## Checkpoint attempt after publishing the actual result - -The complete family result was written into 2568's description and read back -at its canonical update timestamp `2026-09-06T02:48:17.943+1200`. It explicitly -retains In Progress for Luna's visible delivery failure and records Terra's -bounded pass. This was the actual changed source for the planned checkpoint, -not a fabricated successful lesson. - -Before the attempt, the authoritative registry returned Vontology workflow -`#V#lab_project_status_digest_workflow` with 11 states and definition hash -`e34c44ab88224020ebcca9524a3325131e3dfe6035198286f87074faa6587445`. -The live continuity prompt text had SHA-256 -`a39941a5bae51044298308ee5be9e675b68faa0da4f937a2e680fd18101dfbc5` and matched -the release seed after trimming boundary whitespace. No seed was activated or -overwritten for this test. - -The attached Von MCP `workflow_create_schedule` request selected one `once` -occurrence, Michael's intended scope, the existing digest workflow, this exact -work-product ID, current source locators and Luna. The prompt asked to maintain -the responsibility; it did not supply the expected revised text. The tool -rejected the request with **`workflow_actor_authority_required`**, reporting -that launch scope conflicted with ambient authority (`mismatch_fields=[]`). -The handler returns this error before schedule construction/persistence. -No schedule ID or worker occurrence was created and no scheduled model ran. - -This establishes a missing usable actor binding on the attached scheduling -route. It does not establish that every trusted operator route or the scheduler -itself is broken. Separately, the catalogue and live -[2698](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-2698) confirm that -ordinary-chat schedule creation remains unavailable. Comment 37303 records this -new pilot consumer and failed attempt on that existing task. The checkpoint -claim remains blocked at launch; revision through the digest's singleton writer -was not exercised. The two foreground text relations remain retained evidence. - -## Delivery decision and limits - -**Merge decision does not change:** publish the refined programme and this -bounded execution record. It establishes a real owned work product, fresh -recovery and actual source reconciliation, a full paired Jira-family observation, -and two discriminating integration failures. It does not establish dependable -routine programme management, a completed scheduled checkpoint, low human -burden, beneficial learning or transfer. - -The current decisions remain in 2721, 2568 and 2698. Repair of answer recovery -must reconcile the exact persisted request and delivered UI outcome; supported -schedule issuance must precede a positive checkpoint claim. These observed -connections, and the ambiguous foreground text revision, matter before adding -a general role, theory or learning mechanism. Neither a universal certification -campaign nor a model switch is supported as the next remedy by these results. - -Facilitation included source selection, ten foreground prompt submissions, -model selection/restoration, evidence inspection, Jira planning/result updates, -and one unsuccessful schedule-creation request. No corrective answer was fed -to Von. Michael's later use or feedback on the brief remains unobserved. -The concurrent portfolio reconciliation was checked against live Jira and -incorporated into the programme; its local audit files were preserved. - -The [selected evidence](../generated/jvnautosci_2721_prospective_readiness_2026-09-05.json) -retains locators, response/source hashes, effect summaries, model identities, -cost coverage and the rejected checkpoint receipt. Full actor-scoped traces, -canonical text snapshots and diagnostic envelopes remain in the private local -execution archive. No application code, credentials or running model policy -was changed by this delivery, and no deployment is required for these documents. diff --git a/docs/engineering/jvnautosci_2721_replay_programme_2026-09-05.md b/docs/engineering/jvnautosci_2721_replay_programme_2026-09-05.md deleted file mode 100644 index 244baa3dd..000000000 --- a/docs/engineering/jvnautosci_2721_replay_programme_2026-09-05.md +++ /dev/null @@ -1,103 +0,0 @@ -# Replay programme: deployment and first operational witness - -- **Kind:** Dated operational evidence record -- **Lifecycle:** Frozen -- **Authority:** Evidence only; current repair selection lives in [2722](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-2722), programme allocation in [2721](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-2721) -- **Owner:** Michael Witbrock -- **Evidence date:** 5 September 2026 UTC -- **Review trigger:** New release, actor/resource binding or repeated paired replay -- **Protocol:** [Role convergence test programme](role_convergence_test_programme.md) - -## Deployment and represented activation - -The user authorised deployment. Before deployment, port 5001 reported old build -`613e729d49d8.dirty`; its startup-seed receipts were missing. Canonical -`./run.sh deploy-main` installed clean `16e4fc79089e8480237e22bc3021874ced5f2cd4` -in detached `Von-runtime-main`. The launcher verified server PID 48688, RAG -worker 49352, concept-index worker 49367, current startup-seed receipts and -running durable worker/scheduler. An initial healthy HTTP response preceded -durable readiness; it was not treated as completed deployment. - -The selected digest family was also reconciled through its canonical bootstrap -service, without force. It recognised the reviewed version-7 authority digest, -published version 8 and created the new continuity prompt. Subsequent canonical -read-back found `initialise_scope` and exact prompt text matching the release -(SHA-256 `a39941a5bae51044298308ee5be9e675b68faa0da4f937a2e680fd18101dfbc5`). -The old authored prompt was not overwritten. Existing startup purity drift -was reported as advisory; no baseline was reset to hide it. - -## First witness and model observations - -The existing 2568 Jira family was selected on actual `/von/generate`, using -normal localhost browser-test session issuance. The actor was the existing -Zhan browser-test identity in SAIL. This proves a test path, not Michael's -personal profile or prospective adoption. The intended first answer was the -newest Jira Task's key, summary and status, grounded in a current read. - -| Attempt | Actual result | Interpretation | -|---|---|---| -| GPT-5.5 | `model_not_enabled` before reasoning | Setup failure; provider availability did not establish actor eligibility | -| Terra | Completed after four successful `turn_capabilities` calls, but said read-only Jira search was unavailable and gave no issue answer | Useful-answer failure; no evidence of expired Jira authentication | -| Luna caller | Later canonical result correctly named 2721 as newest at query time, after launching the lab-status digest workflow | Answer reached, but not a clean read-only or Luna-only success: the child used Terra and persisted a digest | - -The [bounded evidence extract](../generated/role_programme_replay_2026-09-05.json) -retains request/instance identifiers, actual model-call identities and usage, -catalogue fingerprints, answers, the write receipt and source-artifact hashes. -Private raw task/history/instance material remains in local operator evidence. -These are individual developmental observations, not success-rate estimates. -The live Jira source was not frozen, and planning updates were concurrent. - -Michael clarified that Luna is his ordinary Von model and requested interest -in Terra/Luna differences. The programme therefore uses Luna as principal and -paired Terra trials on selected cases. This first Luna attempt demonstrates why -the child model must also be recorded: its outer calls used Luna, while two -digest calls used the test actor's active Terra selection. It cannot establish -that Luna alone solved a case Terra could not. - -The Luna observer returned pending/inconclusive and did not execute the remaining -three follow-ups. A subsequent authenticated read found the first turn completed -and retrieved its actual result. No new competing task was submitted to replace -that turn. The collector's interim result and later canonical result are both -retained; the four-turn family is not passed. - -## Connector versus capability authority - -The live Terra trace reported `gateway_present=true`, `gateway_enabled=true` -and a populated tool catalogue. Its capability-discovery calls succeeded. -The deployed catalogue explicitly excludes `jira_search`, `jira_get_issue` and -`jira_get_comments` from ordinary-turn projection with -`deployment_global_account`. - -Both the connected Von MCP and a trusted-operator gateway invocation from the -deployed checkout successfully read 2721. The latter used that checkout's -configured Jira proxy and returned no authentication error. This operator probe -does not grant ordinary users access, but distinguishes working connectivity -and credentials from the separate capability projection. Reauthentication was -not indicated; no credential, model-pool or Rovo configuration was changed. - -Luna found a different route: the existing represented digest capability could -read Jira. Its durable instance `0204da15-3c9a-498a-b79f-01299f91a691` reached -canonical `completed` and exact text verification. Its actual write targeted -the browser-test actor's existing `#V#operational_reliability_status_digest`. -The receipt records a succeeded text upsert and one replaced relation. The -prior content projection exposed only the product description, so it cannot -establish the previous content bytes or support a speculative exact rollback. -No claim is made that this was zero-write execution or that nothing pre-existed. - -This is a concrete reason for unique test products, pre-state snapshots and -effect inspection in the next cycle. It is also a reason to make authorised -Jira reads directly available through the correct resource boundary instead of -rewarding a workaround with unrelated durable work. Do not infer that all -authenticated users should receive the deployment account's access. - -## Scope of the handoff - -The test-programme document and existing replay-input correction passed all -19 tests in `tests/test_run_live_multi_turn_followup_replay.py`; document links -and diff checks passed. This supports the programme/input change, not a repaired -Jira capability or role certification. - -The full witness and real-role claim remain open. The observations identify a -specific integration decision and a mixed-model/effect confound to remove -before interpreting model differences. The next implementation decision belongs -in 2722; this dated record does not prescribe a second repair plan. diff --git a/docs/engineering/jvnautosci_799_documentation_index.md b/docs/engineering/jvnautosci_799_documentation_index.md deleted file mode 100644 index d2a0edacd..000000000 --- a/docs/engineering/jvnautosci_799_documentation_index.md +++ /dev/null @@ -1,422 +0,0 @@ -# JVNAUTOSCI-799 Documentation Index - -> **Document status: Historical, task-local documentation index for the 2025 -> JVNAUTOSCI-799 milestone.** This is not the Von-wide document index. “Current -> status”, “complete”, and readiness wording below is bounded to evidence at -> that milestone. Use [`../design_index.md`](../design_index.md) for current -> documentation authority and navigation. - -**Structured Tool Calling for Internal MCP - Complete Documentation** - -This index organizes all documentation for JVNAUTOSCI-799 (Phases 0-2 Complete). - ---- - -## 📋 Quick Links - -### For Different Audiences - -**🚀 Getting Started** -- Start here: [Quick Reference Card](jvnautosci_799_quick_reference.md) (5 min read) -- Then read: [Usage Guide](structured_tool_calling_guide.md) (15 min read) - -**📊 Project Status** -- Current status: [Final Status Report](jvnautosci_799_final_status.md) (10 min read) -- Timeline: [Implementation Roadmap](jvnautosci_799_implementation_roadmap.md) (20 min read) - -**🔍 Technical Deep Dive** -- Architecture: [Phases 1-2 Summary](jvnautosci_799_phases_1-2_summary.md) (15 min read) -- Current state: [Phase 0 Audit](jvnautosci_799_phase0_audit.md) (20 min read) - -**💻 Code Reference** -- Source: [src/backend/languagemodels/structured_tool_calling/](../../src/backend/languagemodels/structured_tool_calling/) -- Tests: [tests/backend/test_structured_tool_calling_*.py](../../tests/backend/) - ---- - -## 📚 Documentation Files - -### 1. Quick Reference Card -**File**: `jvnautosci_799_quick_reference.md` -**Length**: ~200 lines -**Purpose**: Fast lookup for developers -**Best For**: "How do I...?" questions - -**Contents**: -- Import statements -- Client creation examples -- Tool definition examples -- Common patterns -- API reference table -- Feature flags -- Troubleshooting quick fixes - -**Read Time**: 5 minutes - ---- - -### 2. Complete Usage Guide -**File**: `structured_tool_calling_guide.md` -**Length**: ~400 lines -**Purpose**: Comprehensive user documentation -**Best For**: Understanding how to use the system - -**Contents**: -- Overview and benefits -- Quick start (basic & async) -- Architecture diagram -- Core types documentation -- Configuration guide -- Integration with orchestrator -- Testing guide -- Best practices -- Migration guide -- Troubleshooting section -- References - -**Read Time**: 15 minutes - ---- - -### 3. Final Status Report -**File**: `jvnautosci_799_final_status.md` -**Length**: ~300 lines -**Purpose**: Executive summary of completed work -**Best For**: Project status, what was built, validation results - -**Contents**: -- Executive summary -- Quality metrics -- Detailed deliverables -- Technical architecture -- Integration with JVNAUTOSCI-803 -- Success validation (tests, types, quality) -- File structure -- Code quality standards -- Next steps (Phase 3) -- Statistics - -**Read Time**: 10 minutes - ---- - -### 4. Phases 1-2 Implementation Summary -**File**: `jvnautosci_799_phases_1-2_summary.md` -**Length**: ~350 lines -**Purpose**: Technical summary of what was implemented -**Best For**: Understanding implementation details - -**Contents**: -- What was built (overview) -- Core package structure -- Type system documentation -- Client interface details -- Provider implementations (3 providers) -- Testing summary (23/23 passing) -- Validation instructions -- Architecture diagram -- Design decisions -- File locations - -**Read Time**: 15 minutes - ---- - -### 5. Implementation Roadmap -**File**: `jvnautosci_799_implementation_roadmap.md` -**Length**: ~400 lines -**Purpose**: Comprehensive project roadmap and timeline -**Best For**: Understanding phases, timeline, acceptance criteria - -**Contents**: -- Executive summary -- Phase timeline (Phase 0-5) -- Phase 0: Audit (✅ Complete) -- Phase 1: Core types (✅ Complete) -- Phase 2: Providers (✅ Complete) -- Phase 3: Orchestrator integration (⏳ In Progress) -- Phase 4: Evaluation & regression (⏳ Planned) -- Phase 5: Deprecation (⏳ Planned) -- Acceptance criteria for each phase -- Architecture decisions & rationale -- Code quality standards -- Rollout strategy -- Risk mitigation -- Metrics and success measures -- Timeline estimates (37 hours total) -- References - -**Read Time**: 20 minutes - ---- - -### 6. Phase 0 Audit -**File**: `jvnautosci_799_phase0_audit.md` -**Length**: ~400 lines -**Purpose**: Detailed analysis of current state before implementation -**Best For**: Understanding existing tool-call patterns and issues - -**Contents**: -- Current tool-call patterns - - `_extract_json_blob()` details - - Annotation service path - - Fast-path endpoint -- Safety constraints - - Tool whitelist - - Namespace injection rules - - Single-call contract -- Failure modes with examples - - Hallucination - - Parsing ambiguity - - Truncation - - Multiple JSON objects -- Provider-specific considerations - - OpenAI function calling - - Gemini function calling - - Ollama constraints -- Integration points with JVNAUTOSCI-803 -- Feature flag strategy -- Test coverage requirements -- Backward compatibility approach - -**Read Time**: 20 minutes - ---- - -## 🗂️ Code Files - -### Implementation - -**Directory**: `src/backend/languagemodels/structured_tool_calling/` - -| File | Lines | Purpose | -|------|-------|---------| -| `__init__.py` | 20 | Public API exports | -| `types.py` | 150 | Core data types (ToolCall, ToolDefinition, etc.) | -| `client.py` | 200 | Abstract LLMClient interface | -| `factory.py` | 30 | get_llm_client() factory function | -| `providers/__init__.py` | 10 | Provider exports | -| `providers/openai_client.py` | 180 | OpenAI GPT-4/3.5 Turbo | -| `providers/gemini_client.py` | 220 | Google Gemini | -| `providers/ollama_client.py` | 280 | Local models (Llama, Mistral, etc.) | - -**Total Core Code**: ~700 lines - -### Tests - -**Directory**: `tests/backend/` - -| File | Tests | Purpose | -|------|-------|---------| -| `test_structured_tool_calling_types.py` | 14 | Type validation tests | -| `test_structured_tool_calling_client.py` | 9 | Client interface tests | - -**Total Tests**: 23/23 passing (100%) - ---- - -## 📊 Reading Recommendations by Role - -### For Developers Using the System -1. [Quick Reference Card](jvnautosci_799_quick_reference.md) - 5 min -2. [Usage Guide](structured_tool_calling_guide.md) - 15 min -3. Code examples in docstrings - 5 min -4. [Troubleshooting section](structured_tool_calling_guide.md#troubleshooting) - 5 min - -**Total**: ~30 minutes to productive usage - -### For System Integrators (Phase 3) -1. [Final Status Report](jvnautosci_799_final_status.md) - 10 min -2. [Phases 1-2 Summary](jvnautosci_799_phases_1-2_summary.md) - 15 min -3. [Implementation Roadmap - Phase 3 section](jvnautosci_799_implementation_roadmap.md#phase-3-orchestrator-integration) - 10 min -4. Code review of providers - 20 min - -**Total**: ~55 minutes to understand integration points - -### For Project Managers -1. [Final Status Report](jvnautosci_799_final_status.md) - 10 min -2. [Implementation Roadmap](jvnautosci_799_implementation_roadmap.md) - 20 min -3. [Timeline and Metrics sections](jvnautosci_799_implementation_roadmap.md#timeline-estimates) - 5 min - -**Total**: ~35 minutes for complete project overview - -### For Architects/Decision-Makers -1. [Final Status Report - Architecture section](jvnautosci_799_final_status.md#technical-architecture) - 5 min -2. [Implementation Roadmap - Architecture decisions](jvnautosci_799_implementation_roadmap.md#architecture-decisions) - 10 min -3. [Phase 0 Audit - Integration points](jvnautosci_799_phase0_audit.md) - 15 min -4. [Phases 1-2 Summary - Architecture diagram](jvnautosci_799_phases_1-2_summary.md#architecture-diagram) - 5 min - -**Total**: ~35 minutes for architectural understanding - -### For Code Reviewers -1. [Phases 1-2 Summary](jvnautosci_799_phases_1-2_summary.md) - 15 min -2. [Quality standards section](jvnautosci_799_implementation_roadmap.md#code-quality-standards) - 5 min -3. Source code review - 30 min -4. Test review - 20 min - -**Total**: ~70 minutes for thorough code review - ---- - -## 🎯 Navigation by Task - -### "I want to use structured tool calling in my code" -→ [Quick Reference](jvnautosci_799_quick_reference.md) + [Usage Guide](structured_tool_calling_guide.md) - -### "I need to integrate this with the orchestrator (Phase 3)" -→ [Roadmap Phase 3](jvnautosci_799_implementation_roadmap.md#phase-3-orchestrator-integration) + [Final Status](jvnautosci_799_final_status.md) - -### "I need to understand what was built" -→ [Phases 1-2 Summary](jvnautosci_799_phases_1-2_summary.md) + [Final Status](jvnautosci_799_final_status.md) - -### "I need to understand the current state before this work" -→ [Phase 0 Audit](jvnautosci_799_phase0_audit.md) - -### "I need the complete project overview" -→ [Implementation Roadmap](jvnautosci_799_implementation_roadmap.md) - -### "I need to understand the architecture" -→ [Phases 1-2 Summary - Architecture](jvnautosci_799_phases_1-2_summary.md#architecture-diagram) + [Final Status - Technical Architecture](jvnautosci_799_final_status.md#technical-architecture) - -### "I want to write a test" -→ [Usage Guide - Testing](structured_tool_calling_guide.md#testing) + Look at existing tests - -### "I'm troubleshooting an issue" -→ [Quick Reference - Troubleshooting](jvnautosci_799_quick_reference.md#troubleshooting) + [Usage Guide - Troubleshooting](structured_tool_calling_guide.md#troubleshooting) - ---- - -## 📈 Document Map - -``` -JVNAUTOSCI-799 Documentation -│ -├── 🚀 Getting Started -│ ├── jvnautosci_799_quick_reference.md (5 min) -│ └── structured_tool_calling_guide.md (15 min) -│ -├── 📊 Project Overview -│ ├── jvnautosci_799_final_status.md (10 min) -│ └── jvnautosci_799_implementation_roadmap.md (20 min) -│ -├── 🔍 Technical Details -│ ├── jvnautosci_799_phases_1-2_summary.md (15 min) -│ └── jvnautosci_799_phase0_audit.md (20 min) -│ -└── 💻 Code - ├── src/backend/languagemodels/structured_tool_calling/ - │ ├── types.py (Core types) - │ ├── client.py (LLMClient interface) - │ └── providers/ (OpenAI, Gemini, Ollama) - │ - └── tests/backend/ (23 tests, 100% passing) -``` - ---- - -## 🔗 Cross-References - -### Related JIRA Issues -- **JVNAUTOSCI-799**: This issue (Structured tool calling) -- **JVNAUTOSCI-803**: LLM Workflows (depends on JVNAUTOSCI-799) -- **JVNAUTOSCI-698**: JSON action output prevention (original issue) -- **JVNAUTOSCI-640**: Von Agent Foundations (epic) - -### External Resources -- [OpenAI Function Calling](https://platform.openai.com/docs/guides/function-calling) -- [Gemini Function Calling](https://ai.google.dev/docs/function_calling) -- [JSON Schema](https://json-schema.org/) -- [Python asyncio](https://docs.python.org/3/library/asyncio.html) - ---- - -## 📝 Document Maintenance - -### When to Update Which Documents - -**After Code Changes**: -- Update: Docstrings in source code -- Update: [Usage Guide](structured_tool_calling_guide.md) if API changes -- Update: [Quick Reference](jvnautosci_799_quick_reference.md) if API changes -- Update: [Final Status](jvnautosci_799_final_status.md) if stats change - -**After Phase Completion**: -- Update: [Implementation Roadmap](jvnautosci_799_implementation_roadmap.md) - Status column -- Create: Phase summary document -- Update: [Final Status](jvnautosci_799_final_status.md) - Overall status - -**For New Discoveries**: -- Update: [Usage Guide - Troubleshooting](structured_tool_calling_guide.md#troubleshooting) -- Update: [Phase 0 Audit](jvnautosci_799_phase0_audit.md) if finding affects architecture - ---- - -## ✅ Document Completion Status - -| Document | Status | Length | Purpose | -|----------|--------|--------|---------| -| Quick Reference | ✅ Complete | 200 lines | Fast lookup | -| Usage Guide | ✅ Complete | 400 lines | Developer guide | -| Final Status | ✅ Complete | 300 lines | Project summary | -| Phases 1-2 Summary | ✅ Complete | 350 lines | Implementation details | -| Implementation Roadmap | ✅ Complete | 400 lines | Timeline & phases | -| Phase 0 Audit | ✅ Complete | 400 lines | Current state analysis | -| This Index | ✅ Complete | 300 lines | Navigation guide | - -**Total Documentation**: ~2400 lines (1000+ in this directory) - ---- - -## 🎓 Learning Path - -### Beginner (1 hour) -1. [Quick Reference](jvnautosci_799_quick_reference.md) -2. [Usage Guide - Quick Start](structured_tool_calling_guide.md#quick-start) -3. Explore examples in docstrings - -### Intermediate (3 hours) -1. Complete [Usage Guide](structured_tool_calling_guide.md) -2. Read [Phases 1-2 Summary](jvnautosci_799_phases_1-2_summary.md) -3. Review code in `src/backend/languagemodels/structured_tool_calling/` -4. Study test examples - -### Advanced (6+ hours) -1. [Implementation Roadmap](jvnautosci_799_implementation_roadmap.md) -2. [Phase 0 Audit](jvnautosci_799_phase0_audit.md) -3. Deep code review of all providers -4. Architecture discussions - ---- - -## 📞 Support & Questions - -### Documentation Coverage -- ✅ How to use the system - [Usage Guide](structured_tool_calling_guide.md) -- ✅ Quick API reference - [Quick Reference](jvnautosci_799_quick_reference.md) -- ✅ Common patterns - [Usage Guide - Best Practices](structured_tool_calling_guide.md#best-practices) -- ✅ Troubleshooting - [Quick Reference - Troubleshooting](jvnautosci_799_quick_reference.md#troubleshooting) -- ✅ Architecture - [Final Status - Architecture](jvnautosci_799_final_status.md#technical-architecture) -- ✅ Timeline - [Implementation Roadmap - Timeline](jvnautosci_799_implementation_roadmap.md#timeline-estimates) - -### If Your Question Isn't Answered -1. Check [Quick Reference - Troubleshooting](jvnautosci_799_quick_reference.md#troubleshooting) -2. Check [Usage Guide - Troubleshooting](structured_tool_calling_guide.md#troubleshooting) -3. Review docstrings in source code -4. Check test examples -5. Create issue on JIRA (JVNAUTOSCI-799) - ---- - -## 📄 Document License & Attribution - -All documentation and code in JVNAUTOSCI-799 is created as part of the Von research project. - -**Created**: 2025-11-15 -**By**: GitHub Copilot (AI Implementation Assistant) -**For**: Michael Witbrock, Von Lab, University of Auckland - ---- - -**Last Updated**: 2025-11-15 -**Status**: ✅ Phases 0-2 Complete -**Ready For**: Phase 3 - Orchestrator Integration diff --git a/docs/engineering/jvnautosci_799_final_status.md b/docs/engineering/jvnautosci_799_final_status.md deleted file mode 100644 index 129c1a242..000000000 --- a/docs/engineering/jvnautosci_799_final_status.md +++ /dev/null @@ -1,634 +0,0 @@ -# JVNAUTOSCI-799 Phases 0-2: Final Status Report - -> **Document status: Historical JVNAUTOSCI-799 implementation record -> (November–December 2025).** “Complete”, “validated”, “current”, coverage, and -> readiness claims below refer only to the named phase and evidence available -> then. They are not claims about current Von behaviour, broader reliability, -> present test coverage, or production readiness. Revalidate the current code -> path, targeted tests, and live authority surfaces before relying on details. - -**Date**: 2025-11-15 -**Status**: ✅ COMPLETE AND VALIDATED -**Tests**: 23/23 Passing (100%) -**Type Coverage**: 100% (Pyright) - ---- - -## Executive Summary - -**JVNAUTOSCI-799 Phases 0-2 have been successfully completed.** - -We have implemented a **unified, provider-agnostic structured tool calling system** for Von, replacing JSON-in-text parsing with provider-native function calling. This foundation enables deterministic tool invocation and provides the execution tracing required by JVNAUTOSCI-803 (LLM Workflows). - -### What Was Delivered - -✅ **Core Types System** (types.py, 150 lines) -- `ToolDefinition`: Tool metadata with JSON Schema -- `ToolCall`: Canonical tool invocation with UUID call_id (for execution tracing) -- `LLMResponse`: Unified response format (text + tool calls) -- `ToolCallError`: Exception class for failures - -✅ **Abstract Client Interface** (client.py, 200 lines) -- Async-first `generate_with_tools()` method -- Sync wrapper `generate_with_tools_sync()` for compatibility -- Configuration management (`LLMClientConfig`) -- Schema validation infrastructure - -✅ **Three Provider Implementations** (providers/, 700 lines) -- **OpenAI**: Native GPT-4/3.5 function calling (180 lines) -- **Gemini**: Native Gemini function calling with schema mapping (220 lines) -- **Ollama**: Constrained prompting + JSON extraction for local models (280 lines) - -✅ **Factory Function** (factory.py, 30 lines) -- Auto-selects provider by model name -- Extensible pattern matching -- Fallback to Ollama for unknown models - -✅ **Comprehensive Test Suite** (400 lines, 23/23 passing) -- Type validation tests (14 tests) -- Client interface tests (9 tests) -- All edge cases covered -- Zero test failures - -✅ **Complete Documentation** (1000+ lines) -- Phase 0 Audit (400 lines) -- Implementation Roadmap (300 lines) -- Usage Guide (400 lines) -- Quick Reference Card (200 lines) -- This Status Report - -### Quality Metrics - -| Metric | Value | Status | -|--------|-------|--------| -| **Tests Passing** | 23/23 | ✅ 100% | -| **Type Hints** | 100% | ✅ Complete | -| **Code Lines** | ~2100 | ✅ Well-scoped | -| **Documentation** | 1000+ lines | ✅ Comprehensive | -| **External Dependencies** | 0 new | ✅ Using existing SDKs | -| **Type Checking (Pyright)** | 0 errors | ✅ Clean | -| **Backward Compatibility** | Preserved | ✅ Feature flags for rollout | - ---- - -## Detailed Deliverables - -### 1. Core Implementation (~700 lines) - -#### File Structure -``` -src/backend/languagemodels/structured_tool_calling/ -├── __init__.py (20 lines) - Public API exports -├── types.py (150 lines) - Core canonical types -├── client.py (200 lines) - Abstract LLMClient interface -├── factory.py (30 lines) - Factory function -└── providers/ - ├── __init__.py (10 lines) - ├── openai_client.py (180 lines) - OpenAI GPT-4/3.5 Turbo - ├── gemini_client.py (220 lines) - Google Gemini - └── ollama_client.py (280 lines) - Local models (Llama, Mistral, etc.) -``` - -#### Core Types (`types.py`) - -**`ToolDefinition`** - Describes available tools -```python -@dataclass(frozen=True) -class ToolDefinition: - name: str # Unique tool identifier - description: str # Purpose and usage - input_schema: Dict[str, Any] # JSON Schema for inputs - output_schema: Optional[Dict[str, Any]] = None # Expected output -``` -Features: -- ✅ Immutable (frozen dataclass) -- ✅ Validation: name non-empty, schema well-formed -- ✅ Supports complex nested schemas - -**`ToolCall`** - Canonical tool invocation with tracing -```python -@dataclass(frozen=True) -class ToolCall: - tool_name: str # Tool to invoke - payload: Dict[str, Any] # Arguments - call_id: str = field(...) # UUID for execution tracing ← CRITICAL - timestamp: datetime = field(...) # UTC timestamp -``` -Features: -- ✅ Auto-generated UUID call_id (enables JVNAUTOSCI-803 execution tracing) -- ✅ Auto-generated UTC timestamp -- ✅ Immutable (prevent accidental mutation) - -**`LLMResponse`** - Unified response format -```python -@dataclass -class LLMResponse: - text_response: str # Natural language output - tool_calls: List[ToolCall] = field(...) # Structured invocations - raw_response: Optional[Dict] = None # Provider-specific (debug) - model: Optional[str] = None # Model name - usage: Optional[Dict] = None # Token counts -``` -Methods: -- ✅ `has_tool_calls()`: Check if tools were invoked -- ✅ Works with text-only, tool-calls-only, or mixed responses - -**`ToolCallError`** - Exception for failures -```python -class ToolCallError(Exception): - """Raised when tool calling fails""" -``` - -#### Abstract Client Interface (`client.py`) - -**`LLMClientConfig`** - Configuration for all clients -```python -@dataclass -class LLMClientConfig: - model: str # Model ID (required) - api_key: Optional[str] = None # Auth token - base_url: Optional[str] = None # API endpoint - temperature: float = 0.7 # Sampling - max_tokens: Optional[int] = None # Output limit - enable_structured_calling: bool = True # Feature flag - fallback_to_json_text: bool = True # Fallback option -``` - -**`LLMClient`** - Abstract base class -```python -class LLMClient(ABC): - async def generate_with_tools( - self, - prompt: str, - available_tools: List[ToolDefinition], - system_message: Optional[str] = None, - context: Optional[Dict[str, Any]] = None, - ) -> LLMResponse: - """Async tool invocation (provider-specific implementation)""" - - def generate_with_tools_sync( - self, - prompt: str, - available_tools: List[ToolDefinition], - system_message: Optional[str] = None, - context: Optional[Dict[str, Any]] = None, - ) -> LLMResponse: - """Sync wrapper (uses asyncio.run())""" - - def _validate_input_schema(self, schema: Dict[str, Any]) -> None: - """Validate JSON Schema format""" - - def _tool_definition_to_dict(self, tool: ToolDefinition) -> Dict[str, Any]: - """Convert to provider format (overrideable)""" -``` - -Methods: -- ✅ Async-first architecture (required for JVNAUTOSCI-803 workflows) -- ✅ Sync wrapper for backward compatibility -- ✅ Schema validation at adapter layer - -#### Provider Implementations - -**OpenAI Client** (openai_client.py, 180 lines) -```python -class OpenAIClient(LLMClient): - """OpenAI GPT-4, GPT-3.5 Turbo with native function calling""" -``` -Features: -- ✅ Native `tools` parameter (deterministic) -- ✅ Full token usage tracking -- ✅ Message building (system, user, assistant, tool roles) -- ✅ Tool call parsing from `message.tool_calls` -- ✅ Both async and sync support - -**Gemini Client** (gemini_client.py, 220 lines) -```python -class GeminiClient(LLMClient): - """Google Gemini with native function calling""" -``` -Features: -- ✅ Native function calling API -- ✅ Schema conversion (JSON Schema → Gemini types) -- ✅ Role normalization -- ✅ Thread pool execution (async compatibility) -- ✅ Graceful error handling - -**Ollama Client** (ollama_client.py, 280 lines) -```python -class OllamaClient(LLMClient): - """Local models with constrained prompt engineering""" -``` -Features: -- ✅ No native function calling (supports local models) -- ✅ Constrained prompt engineering (guides JSON formatting) -- ✅ JSON extraction from text with strict validation -- ✅ Stream-based response handling -- ✅ Graceful degradation (text-only fallback) - -#### Factory Function (factory.py, 30 lines) - -```python -def get_llm_client(config: LLMClientConfig) -> LLMClient: - """Factory function to select provider by model name""" - # Pattern matching: gpt-* → OpenAI, gemini-* → Gemini, llama*/mistral/* → Ollama - # Unknown models default to Ollama -``` - -Features: -- ✅ Model name pattern matching -- ✅ Fallback to Ollama for unknown models -- ✅ Extensible for new providers - -### 2. Test Suite (400 lines, 23/23 passing ✅) - -#### Type Tests (test_structured_tool_calling_types.py, 14 tests) - -``` -✅ test_tool_definition_valid -✅ test_tool_definition_with_output_schema -✅ test_tool_definition_missing_name -✅ test_tool_definition_empty_name -✅ test_tool_definition_invalid_schema_no_type -✅ test_valid_tool_call -✅ test_tool_call_with_explicit_call_id -✅ test_invalid_tool_name -✅ test_invalid_payload -✅ test_text_only_response -✅ test_tool_call_response -✅ test_response_with_usage -✅ test_invalid_text_response -✅ test_invalid_tool_calls_list -✅ test_tool_call_error_message -``` - -Coverage: -- ✅ ToolDefinition validation (valid, with output, invalid) -- ✅ ToolCall creation and validation -- ✅ UUID call_id generation -- ✅ UTC timestamp creation -- ✅ LLMResponse variants (text, tools, usage) -- ✅ Error handling - -#### Client Tests (test_structured_tool_calling_client.py, 9 tests) - -``` -✅ test_minimal_config -✅ test_full_config -✅ test_openai_client_creation -✅ test_gemini_client_creation -✅ test_ollama_client_creation -✅ test_unknown_model_defaults_to_ollama -✅ test_validate_input_schema_valid -✅ test_validate_input_schema_with_properties -✅ test_validate_input_schema_invalid -``` - -Coverage: -- ✅ Configuration creation -- ✅ Factory function (all providers) -- ✅ Schema validation (valid/invalid cases) - -**Test Execution Results**: -``` -============================= 23 passed in 5.87s ============================= -``` - -All tests passing, no failures, 0 regressions. - -### 3. Documentation (1000+ lines) - -#### Phase 0 Audit (jvnautosci_799_phase0_audit.md, 400+ lines) -Documents current state: -- ✅ Current tool-call patterns (`_extract_json_blob()` at lines 889-969) -- ✅ Safety constraints (whitelist, namespace, single-call contract) -- ✅ Failure modes with examples (hallucination, parsing ambiguity, multiple JSON) -- ✅ Provider-specific considerations -- ✅ Integration points with JVNAUTOSCI-803 - -#### Implementation Roadmap (jvnautosci_799_implementation_roadmap.md, 300+ lines) -Comprehensive roadmap: -- ✅ Phase breakdown (0-5 with status) -- ✅ Acceptance criteria for each phase -- ✅ Architecture decisions and rationale -- ✅ Code quality standards -- ✅ Rollout strategy -- ✅ Known risks and mitigations -- ✅ Timeline estimates (37 hours total) - -#### Usage Guide (structured_tool_calling_guide.md, 400+ lines) -Complete user documentation: -- ✅ Quick start examples -- ✅ Architecture overview -- ✅ Configuration options -- ✅ Error handling -- ✅ Best practices -- ✅ Migration guide -- ✅ Troubleshooting -- ✅ Provider reference - -#### Quick Reference Card (jvnautosci_799_quick_reference.md, 200+ lines) -Developer quick reference: -- ✅ Import statements -- ✅ Client creation -- ✅ Tool definition examples -- ✅ Common patterns -- ✅ API reference -- ✅ Feature flags -- ✅ Troubleshooting - ---- - -## Technical Architecture - -### Provider Adapter Pattern - -``` -┌─────────────────────────────────────────────────────┐ -│ Canonical Interface (LLMClient) │ -│ generate_with_tools(prompt, tools) → LLMResponse │ -└────────────────────────┬────────────────────────────┘ - │ - ┌───────────────┼───────────────┐ - │ │ │ - ┌────▼────┐ ┌──────▼──────┐ ┌──▼──────────┐ - │ OpenAI │ │ Gemini │ │ Ollama │ - │ Adapter │ │ Adapter │ │ Adapter │ - └────┬────┘ └──────┬──────┘ └──┬──────────┘ - │ │ │ - Native Function Native Function Prompt Engineering - Calling Calling + JSON Extraction - │ │ │ - └───────────────┼─────────────┘ - │ - ┌───────────────▼───────────────┐ - │ Canonical ToolCall │ - │ - tool_name: str │ - │ - payload: Dict │ - │ - call_id: UUID (tracing) │ - │ - timestamp: UTC │ - └───────────────────────────────┘ - │ - ┌───────────────▼───────────────┐ - │ Orchestrator / Agent │ - │ (Phase 3 Integration) │ - └───────────────────────────────┘ -``` - -### Why This Architecture? - -**Canonical Types** (ToolCall, ToolDefinition, LLMResponse) -- ✅ Provider-agnostic -- ✅ Enables polymorphism -- ✅ Orchestrator doesn't need provider-specific code - -**Abstract LLMClient Interface** -- ✅ All providers implement same contract -- ✅ Factory enables dynamic provider selection -- ✅ Easy to add new providers - -**Async-First Design** -- ✅ Required for JVNAUTOSCI-803 workflow concurrency -- ✅ Sync wrapper maintains backward compatibility -- ✅ Reduces latency in multi-step workflows - -**UUID call_id** -- ✅ Enables execution tracing (critical for JVNAUTOSCI-803) -- ✅ Correlates tool calls with workflow steps -- ✅ Auto-generated (no manual tracking) - ---- - -## Integration with JVNAUTOSCI-803 - -This implementation provides the **execution layer foundation** required by JVNAUTOSCI-803 (LLM Workflows). - -### How JVNAUTOSCI-803 Will Use This - -1. **Workflow Step Execution** - ```python - async def execute_step(step: WorkflowStep): - response = await client.generate_with_tools( - prompt=step.prompt, - available_tools=step.tools, - ) - # Tool call call_id enables execution tracing - trace.add_step(tool_calls=response.tool_calls) - ``` - -2. **Execution Tracing** - ```python - for tool_call in response.tool_calls: - trace.add( - call_id=tool_call.call_id, # ← Unique identifier - tool=tool_call.tool_name, - args=tool_call.payload, - timestamp=tool_call.timestamp, - ) - ``` - -3. **Multi-Step Orchestration** - ```python - # Workflow engine executes steps in sequence/parallel - # Each step's tool calls are traced via call_id - # Execution history is deterministic and replayable - ``` - -### Call_id: The Critical Connection - -The `call_id` field in `ToolCall` is **essential for JVNAUTOSCI-803**: -- ✅ Correlates tool invocation with workflow execution trace -- ✅ Enables execution history and replay -- ✅ Auto-generated (UUID4) - globally unique -- ✅ Immutable - cannot be changed - ---- - -## Success Validation - -### Tests: 23/23 Passing ✅ - -```powershell -pdm run pytest tests/backend/test_structured_tool_calling_types.py tests/backend/test_structured_tool_calling_client.py -v -``` - -Result: -``` -============================= 23 passed in 5.87s ============================= -``` - -All tests passing, zero failures, comprehensive coverage. - -### Type Coverage: 100% ✅ - -```powershell -pdm run pyright src/backend/languagemodels/structured_tool_calling/ -``` - -Result: 0 errors, 100% type hints - -### Import Validation ✅ - -```powershell -pdm run python -c "from src.backend.languagemodels.structured_tool_calling import *; print('OK')" -``` - -Result: OK (all exports working) - -### Code Quality ✅ - -- ✅ Black formatting (automatic) -- ✅ New Zealand English spelling throughout -- ✅ Comprehensive docstrings -- ✅ No external dependency additions -- ✅ Backward compatible (feature flags) - ---- - -## File Structure (Phases 0-2 Complete) - -``` -src/backend/languagemodels/structured_tool_calling/ -├── __init__.py ✅ 20 lines -├── types.py ✅ 150 lines -├── client.py ✅ 200 lines -├── factory.py ✅ 30 lines -└── providers/ - ├── __init__.py ✅ 10 lines - ├── openai_client.py ✅ 180 lines - ├── gemini_client.py ✅ 220 lines - └── ollama_client.py ✅ 280 lines - -tests/backend/ -├── test_structured_tool_calling_types.py ✅ 14 tests -└── test_structured_tool_calling_client.py ✅ 9 tests - -docs/engineering/ -├── jvnautosci_799_phase0_audit.md ✅ 400+ lines -├── jvnautosci_799_implementation_roadmap.md ✅ 300+ lines -├── structured_tool_calling_guide.md ✅ 400+ lines -├── jvnautosci_799_quick_reference.md ✅ 200+ lines -└── jvnautosci_799_phases_1-2_summary.md ✅ This report -``` - ---- - -## Code Quality Standards Met - -### Style & Formatting -- ✅ New Zealand English spelling (behaviour, colour, organisation) -- ✅ Black formatter compliant -- ✅ Pyright type checking (100%) -- ✅ Google-style docstrings -- ✅ Comprehensive comments - -### Testing -- ✅ 23 unit tests (100% passing) -- ✅ Edge case coverage -- ✅ No test failures or regressions -- ✅ Type validation in tests - -### Documentation -- ✅ Docstrings on all public methods -- ✅ Type hints on all parameters -- ✅ Usage examples in docstrings -- ✅ Architecture diagrams -- ✅ Comprehensive integration guide - -### Dependencies -- ✅ No new external dependencies added -- ✅ Uses existing SDKs (OpenAI, Gemini, Ollama) -- ✅ Backward compatible -- ✅ Feature flags for safe rollout - ---- - -## Next Steps: Phase 3 - -**Phase 3: Orchestrator Integration** is ready to begin. - -### Phase 3 Scope -1. Examine `InternalMCPChatOrchestrator.run()` method -2. Replace `_extract_json_blob()` with `generate_with_tools()` -3. Create `MCPToolWorkflowExecutor` wrapper -4. Implement feature flag integration -5. Add regression tests - -### Success Criteria -- ✅ Orchestrator uses structured calling as primary path -- ✅ All existing tool invocations work identically -- ✅ Tool call `call_id` available for execution tracing -- ✅ Feature flags enable safe rollout -- ✅ Fallback to JSON-in-text works if structured fails - -### Estimated Effort -- 10 hours (code + tests + validation) - ---- - -## Summary Statistics - -| Category | Metric | Value | -|----------|--------|-------| -| **Code** | Lines in core | 700 | -| | Lines in tests | 400 | -| | Lines in docs | 1000+ | -| | Total | ~2100 | -| **Tests** | Passing | 23/23 | -| | Pass rate | 100% | -| | Execution time | 5.87s | -| **Quality** | Type coverage | 100% | -| | External deps | 0 new | -| | Test categories | 4 | -| **Documentation** | Audit | ✅ | -| | Roadmap | ✅ | -| | Usage guide | ✅ | -| | Quick ref | ✅ | -| **Providers** | Implemented | 3 | -| | OpenAI | ✅ | -| | Gemini | ✅ | -| | Ollama | ✅ | - ---- - -## Known Limitations & Future Work - -### Phase 3-5 (Planned) -- [ ] Orchestrator integration (Phase 3, 10h) -- [ ] Regression tests (Phase 4, 7h) -- [ ] Legacy path deprecation (Phase 5, 5h) -- [ ] Performance benchmarking (Phase 4) -- [ ] Workflow step execution tests (Phase 4) - -### Intentional Scope Limits -- Provider implementations are functional, not fully-featured -- Schema validation is basic (no deep JSON Schema validation) -- Performance optimization deferred to Phase 4 -- Distributed tracing integration deferred to Phase 5 - ---- - -## Approval & Handoff - -✅ **Phases 0-2 Complete and Validated** - -**Created**: 2025-11-15 -**By**: GitHub Copilot (AI Implementation) -**For**: JVNAUTOSCI-799, JVNAUTOSCI-803 -**Status**: Ready for Phase 3 - Orchestrator Integration - -**Ready To Begin**: Phase 3: Orchestrator Integration - ---- - -## References - -- [Phase 0 Audit](jvnautosci_799_phase0_audit.md) - Current state analysis -- [Implementation Roadmap](jvnautosci_799_implementation_roadmap.md) - Full timeline -- [Usage Guide](structured_tool_calling_guide.md) - Developer documentation -- [Quick Reference](jvnautosci_799_quick_reference.md) - Quick lookup -- JIRA Issue: [JVNAUTOSCI-799](https://jira.example.com/browse/JVNAUTOSCI-799) -- Related Issue: [JVNAUTOSCI-803](https://jira.example.com/browse/JVNAUTOSCI-803) (LLM Workflows) - ---- - -**End of Report** diff --git a/docs/engineering/jvnautosci_799_implementation_roadmap.md b/docs/engineering/jvnautosci_799_implementation_roadmap.md deleted file mode 100644 index 4268c7a72..000000000 --- a/docs/engineering/jvnautosci_799_implementation_roadmap.md +++ /dev/null @@ -1,525 +0,0 @@ -# JVNAUTOSCI-799: Structured Tool Calling - Implementation Roadmap - -> **Document status: Historical JVNAUTOSCI-799 implementation record -> (November–December 2025).** Phase status, “current state”, completion, and -> next-step language below records that milestone, not current work sequencing -> or system behaviour. Revalidate against live Jira, code, tests, and authority -> surfaces before acting. - -**Issue**: JVNAUTOSCI-799 (Structured tool calling for internal MCP) -**Epic**: JVNAUTOSCI-640 (Von Agent Foundations) -**Related**: JVNAUTOSCI-803 (LLM Workflows) - **DEPENDS ON THIS** -**Status**: Phases 0-2 Complete, Phase 3 In Progress - -## Executive Summary - -This document tracks the implementation of **unified, provider-agnostic structured tool calling** for the Von system. The work eliminates JSON-in-text parsing ambiguity, enables deterministic tool invocation, and provides the execution tracing foundation required by JVNAUTOSCI-803 (LLM Workflows). - -### Why This Matters - -**Current State (Problem)**: -- `_extract_json_blob()` parses JSON from unstructured text -- Ambiguous parsing: multiple JSON objects, truncated output, hallucinated tools -- No execution tracing for workflow steps -- Provider-specific workarounds scattered throughout code - -**Target State (Solution)**: -- Use provider-native function calling (OpenAI, Gemini) -- Fall back to constrained prompting (Ollama, local models) -- Unified `LLMClient` interface across all providers -- Tool call `call_id` for execution tracing (required by JVNAUTOSCI-803) - -**Impact**: -- ✅ Deterministic tool invocation -- ✅ Foundation for JVNAUTOSCI-803 (LLM Workflows) -- ✅ Execution tracing support -- ✅ Graceful local model support (Ollama) - -## Phase Timeline & Status - -### Phase 0: Audit & Documentation ✅ COMPLETE - -**Goal**: Document current state, identify failure modes, establish baseline. - -**Deliverables**: -- ✅ [jvnautosci_799_phase0_audit.md](jvnautosci_799_phase0_audit.md) - 400+ line comprehensive audit -- ✅ Current tool-call patterns documented -- ✅ Failure modes with examples -- ✅ Provider-specific considerations catalogued -- ✅ Integration points with JVNAUTOSCI-803 identified - -**Key Findings**: -- `_extract_json_blob()` handles ~95% of tool calls (lines 889-969) -- Annotation service provides ~5% (alternative path) -- Multiple JSON objects cause parsing ambiguity (safety issue) -- No provider-native function calling currently used - -### Phase 1: Core Types & Interface ✅ COMPLETE - -**Goal**: Define canonical types and abstract interface for all implementations. - -**Deliverables**: - -1. **`types.py`** (150 lines) - - ✅ `ToolDefinition`: name, description, input_schema, output_schema - - ✅ `ToolCall`: tool_name, payload, call_id (UUID), timestamp (UTC) - - ✅ `LLMResponse`: text_response, tool_calls, raw_response, model, usage - - ✅ `ToolCallError`: Custom exception - - ✅ Validation: schema format checking, non-empty strings - - ✅ Immutability: frozen dataclasses - -2. **`client.py`** (200 lines) - - ✅ `LLMClientConfig`: model, api_key, base_url, temperature, max_tokens - - ✅ `LLMClient`: Abstract base class - - ✅ `async generate_with_tools()`: Async method (provider-specific) - - ✅ `generate_with_tools_sync()`: Sync wrapper (uses asyncio.run()) - - ✅ `_validate_input_schema()`: Base validation method - - ✅ `_tool_definition_to_dict()`: Override point for providers - -3. **`__init__.py`** (20 lines) - - ✅ Public API exports - -**Test Coverage**: -- ✅ 14 unit tests for types (14/14 passing) - - ToolDefinition: valid, with output schema, invalid cases - - ToolCall: creation, call_id auto-gen, timestamp - - LLMResponse: text-only, tool calls, usage tracking - - ToolCallError: exception creation - -- ✅ 9 unit tests for client (9/9 passing) - - LLMClientConfig: creation, defaults - - Factory: OpenAI, Gemini, Ollama detection - - Schema validation: valid/invalid cases - -### Phase 2: Provider Implementations ✅ COMPLETE - -**Goal**: Implement adapters for OpenAI, Gemini, and Ollama providers. - -**Deliverables**: - -1. **`providers/openai_client.py`** (180 lines) - - ✅ Native OpenAI function calling - - ✅ Message building (system, user, assistant, tool roles) - - ✅ Tool invocation via `tools` parameter - - ✅ Response parsing: `message.tool_calls` extraction - - ✅ Token usage tracking - - ✅ Async/sync support (asyncio.run() for sync) - - ✅ Error handling for API failures - - **Providers Supported**: gpt-4, gpt-3.5-turbo - -2. **`providers/gemini_client.py`** (220 lines) - - ✅ Google Gemini function calling - - ✅ Schema conversion: JSON Schema → Gemini types - - ✅ Role normalization: Gemini-specific formatting - - ✅ Thread pool execution (async-blocking compatibility) - - ✅ Tool call parsing - - ✅ Graceful error handling - - **Providers Supported**: gemini-pro, gemini-1.5-pro - -3. **`providers/ollama_client.py`** (280 lines) - - ✅ No native function calling (local models) - - ✅ Constrained prompt engineering - - ✅ JSON instruction embedding in prompt - - ✅ Stream-based response handling - - ✅ JSON extraction from text - - ✅ Strict validation (rejects malformed JSON) - - ✅ Graceful degradation (text-only fallback) - - **Providers Supported**: llama2, mistral, neural-chat, etc. - -4. **`factory.py`** (30 lines) - - ✅ `get_llm_client()`: Factory function - - ✅ Model name pattern matching - - ✅ Provider auto-selection - - ✅ Unknown model fallback (Ollama) - -**Design Decisions**: -- Async-first architecture (required for workflow concurrency) -- Sync wrapper using `asyncio.run()` for compatibility -- Abstract `_tool_definition_to_dict()` for provider customization -- Schema validation at adapter layer (provider-specific needs) - -**Test Coverage**: -- ✅ 23 unit tests total (all passing) -- ✅ Type validation tests (14/14) -- ✅ Client interface tests (9/9) -- ✅ No integration tests yet (planned for Phase 4) - -### Phase 3: Orchestrator Integration ⏳ IN PROGRESS - -**Goal**: Replace `_extract_json_blob()` with structured tool calling. - -**Scope**: -- Modify `InternalMCPChatOrchestrator.run()` to use `LLMClient.generate_with_tools()` -- Preserve existing safety constraints (whitelist, namespace rules, single-call contract) -- Add execution trace correlation (tool call `call_id`) -- Create `MCPToolWorkflowExecutor` wrapper for JVNAUTOSCI-803 compatibility -- Ensure orchestrator loop supports reentrancy (nested workflows) - -**Subtasks**: - -1. **Examine Orchestrator Structure** - - Read full `orchestrator.py` (examine `run()` loop) - - Identify all `_extract_json_blob()` call sites - - Map existing safety constraints - -2. **Create MCPToolWorkflowExecutor** - - Adapter class for workflow step execution - - Wraps orchestrator loop for reentrancy - - Provides execution trace interface - -3. **Modify Orchestrator Loop** - - Replace `generate()` + `_extract_json_blob()` with `generate_with_tools()` - - Map available MCP tools to `ToolDefinition` list - - Extract tool calls from structured response - -4. **Implement Safety Constraints** - - Tool whitelist validation - - Namespace injection rules (preserve existing) - - Single-call contract enforcement - - Parameter validation - -5. **Add Execution Tracing** - - Log tool call `call_id` in execution trace - - Correlate with JVNAUTOSCI-803 workflow steps - - Preserve timing information - -6. **Feature Flag Integration** - - `VON_INTERNAL_MCP_STRUCTURED_TOOL_CALLING`: Enable/disable - - `VON_LEGACY_JSON_TEXT_PARSING`: Force legacy path (debugging) - - Fallback to `_extract_json_blob()` if structured fails - -7. **Testing** - - Regression tests for existing chat interactions - - Fallback path tests (graceful degradation) - - Workflow integration tests (JVNAUTOSCI-803) - -**Success Criteria**: -- ✅ Orchestrator uses `generate_with_tools()` as primary path -- ✅ All existing tool invocations work identically -- ✅ Tool call `call_id` available for execution tracing -- ✅ Feature flags enable safe rollout -- ✅ Fallback to JSON-in-text works if structured fails - -**Estimated Effort**: 8-12 hours (code + tests + validation) - -### Phase 4: Evaluation & Regression Tests ⏳ PLANNED - -**Goal**: Validate structured tool calling fixes JVNAUTOSCI-799 issues. - -**Test Coverage**: - -1. **Runtime State Hallucination** (JVNAUTOSCI-799 #1) - - Model claims "tool unavailable" when it exists - - Test: Model MUST invoke available tool (not refuse) - - Success metric: 100% invocation rate (vs ~90% with JSON parsing) - -2. **Malformed JSON Handling** (JVNAUTOSCI-799 #2) - - Multiple JSON objects in response → ambiguous parsing - - Test: Structured calling prevents multi-JSON ambiguity - - Fallback: gracefully degrade to text-only response - - Success metric: 0 invalid tool executions - -3. **Tool Name Validation** (JVNAUTOSCI-799 #3) - - Model invokes non-existent tool - - Test: System rejects unknown tools, requests correction - - Success metric: 0 crashes from unknown tools - -4. **Workflow Step Execution** (JVNAUTOSCI-803 Integration) - - Multi-step workflows execute reliably - - Tool call `call_id` preserved across steps - - Success metric: 95%+ workflow completion rate - -5. **Performance Benchmarking** - - Overhead comparison: Structured vs JSON-in-text - - Target: <10ms overhead per tool call - - Measure: end-to-end latency, parsing time, validation time - -**Test Dataset**: -- 50+ chat interactions (existing test corpus) -- 20+ workflow scenarios (from JVNAUTOSCI-803) -- Edge cases: truncated output, multiple JSON, malformed input - -**Estimated Effort**: 6-8 hours (test implementation + analysis) - -### Phase 5: Deprecation & Cleanup ⏳ PLANNED - -**Goal**: Remove legacy JSON-in-text paths after validation. - -**Actions**: - -1. **Disable JSON-in-text by default** - - `VON_LEGACY_JSON_TEXT_PARSING=0` (no longer auto-enabled) - - Structured calling becomes primary path - -2. **Deprecate `_extract_json_blob()`** - - Mark method as `@deprecated` with migration guide - - Move to `legacy_tool_parsing.py` (preserve for fallback) - - Remove from orchestrator main loop - -3. **Remove Fast-Path** - - `/von/generate` prompt-introspection endpoint no longer special-cased - - Unified orchestrator loop for all paths - -4. **Documentation & Migration Guide** - - Publish integration guide for external agents - - Update orchestrator documentation - - Examples: Using `LLMClient` directly - -**Success Criteria**: -- ✅ Zero references to `_extract_json_blob()` in main code -- ✅ Legacy path available but clearly deprecated -- ✅ Migration guide published -- ✅ No performance regression (vs Phase 4) - -**Estimated Effort**: 4-6 hours (code cleanup + documentation) - -## Acceptance Criteria - -### Phase 1-2: Core Implementation -- ✅ All core types implement (`ToolCall`, `ToolDefinition`, `LLMResponse`) -- ✅ All providers implemented (OpenAI, Gemini, Ollama) -- ✅ Factory function selects correct provider by model name -- ✅ 23+ unit tests, all passing -- ✅ No external dependency failures -- ✅ Code follows New Zealand English spelling & style guide -- ✅ Type hints complete (Pyright validation) - -### Phase 3: Orchestrator Integration -- ✅ Orchestrator uses `generate_with_tools()` as primary path -- ✅ All existing tool invocations work without modification -- ✅ Tool call `call_id` available for execution tracing -- ✅ Safety constraints enforced (whitelist, namespace, single-call) -- ✅ Feature flags enable safe rollout -- ✅ Fallback to JSON-in-text works if structured fails -- ✅ No regression in existing chat/tool functionality - -### Phase 4: Evaluation -- ✅ Runtime state hallucination issue resolved (100% invocation rate) -- ✅ Malformed JSON handling prevents invalid tool execution -- ✅ Unknown tool invocation rejected cleanly -- ✅ Workflow step execution reliable (95%+ completion) -- ✅ Performance overhead <10ms per tool call -- ✅ Regression tests in permanent test suite - -### Phase 5: Cleanup -- ✅ Legacy `_extract_json_blob()` moved to `legacy_tool_parsing.py` -- ✅ Deprecation warning on legacy path -- ✅ Migration guide published -- ✅ Zero test failures - -## Architecture Decisions - -### Why Structured Tool Calling? - -**Alternative 1: Continue JSON-in-text parsing** -- ❌ Ambiguous (multiple JSON objects) -- ❌ Fragile (truncation, hallucination) -- ❌ No provider leverage (wasteful) -- ❌ No execution tracing - -**Alternative 2: Use LLM function calling** -- ✅ Deterministic (provider handles parsing) -- ✅ Reliable (structured contract) -- ✅ Provider-native (leverage APIs) -- ✅ Execution tracing ready (call_id) - -**Decision**: Alternative 2 (structured tool calling) is selected. - -### Why Separate Types from Providers? - -**Rationale**: -- `ToolCall`, `ToolDefinition`, `LLMResponse` are canonical (provider-agnostic) -- Providers translate provider-specific responses → canonical types -- New providers can implement without modifying core types -- Enables provider-agnostic orchestrator code - -### Why Async-First? - -**Rationale**: -- JVNAUTOSCI-803 (workflows) requires concurrent task execution -- Async enables parallel tool invocation across workflow steps -- Sync wrapper provides backward compatibility -- Async reduces latency in multi-step workflows - -### Why `call_id` in `ToolCall`? - -**Rationale**: -- JVNAUTOSCI-803 requires execution tracing -- `call_id` (UUID) correlates tool call with execution trace -- Auto-generated on creation (no manual tracking) -- Immutable (frozen dataclass) for integrity - -## Code Quality Standards - -All code in Phases 1-5 must adhere to: - -### Style & Formatting -- ✅ New Zealand English spelling (behaviour, colour, organisation) -- ✅ Black formatter (automatic via `pdm run black .`) -- ✅ Pyright type checking (100% coverage) -- ✅ Docstrings for all public methods -- ✅ Comments for non-obvious logic - -### Testing -- ✅ Unit tests for all public methods -- ✅ Integration tests for provider implementations -- ✅ Regression tests for existing functionality -- ✅ Edge case coverage (malformed input, failures) - -### Documentation -- ✅ Docstrings follow Google style -- ✅ Type hints on all parameters -- ✅ Usage examples in module docstring -- ✅ Architecture diagram in `ARCHITECTURE.md` (planned) - -### Dependencies -- ✅ No new external dependencies (use existing SDKs) -- ✅ Backward compatibility maintained -- ✅ Feature flags for safe rollout - -## File Structure (As Implemented) - -``` -src/backend/languagemodels/ -├── structured_tool_calling/ # Phase 1-2: Core implementation -│ ├── __init__.py # Public exports -│ ├── types.py # ToolCall, ToolDefinition, LLMResponse -│ ├── client.py # LLMClient abstract interface -│ ├── factory.py # get_llm_client() factory -│ └── providers/ -│ ├── __init__.py -│ ├── openai_client.py # OpenAI GPT-4, GPT-3.5 Turbo -│ ├── gemini_client.py # Google Gemini -│ └── ollama_client.py # Local models -│ -├── orchestrator_integration/ # Phase 3: TBD -│ ├── __init__.py -│ ├── workflow_executor.py # MCPToolWorkflowExecutor -│ └── legacy_tool_parsing.py # Deprecated _extract_json_blob() -│ -└── llm_interface.py # Existing (to be refactored in Phase 5) - -tests/backend/ -├── test_structured_tool_calling_types.py # 14 tests ✅ -├── test_structured_tool_calling_client.py # 9 tests ✅ -├── test_structured_tool_calling_providers/ # TBD (Phase 3-4) -│ ├── test_openai_client.py -│ ├── test_gemini_client.py -│ └── test_ollama_client.py -└── test_orchestrator_integration.py # TBD (Phase 3) - -docs/engineering/ -├── jvnautosci_799_phase0_audit.md # ✅ Phase 0 -├── jvnautosci_799_implementation_roadmap.md # ← This file -├── structured_tool_calling_guide.md # ← Usage guide -└── ARCHITECTURE.md # TBD (Phase 3) -``` - -## Rollout Strategy - -### Safe Activation (Phase 3) - -Feature flag `VON_INTERNAL_MCP_STRUCTURED_TOOL_CALLING`: -- **Default**: Enabled (1) in development -- **Default**: Disabled (0) in production until Phase 4 tests pass -- **Override**: `VON_LEGACY_JSON_TEXT_PARSING=1` forces legacy path - -### Gradual Migration -1. Phase 3: Structured calling as primary, JSON-in-text as fallback -2. Phase 4: Validation tests pass → promote to production -3. Phase 5: Legacy path deprecated, structured calling mandatory - -### Rollback Plan -If structured calling fails in Phase 3: -1. Set `VON_INTERNAL_MCP_STRUCTURED_TOOL_CALLING=0` (env var) -2. Orchestrator reverts to `_extract_json_blob()` -3. No code changes required (feature flag handles it) - -## Known Risks & Mitigations - -### Risk 1: Provider API Changes -- **Impact**: New provider version breaks compatibility -- **Mitigation**: Pin SDK versions in `pyproject.toml`, test against latest quarterly - -### Risk 2: Model Behavior Regression -- **Impact**: Model no longer invokes tools reliably -- **Mitigation**: JVNAUTOSCI-799 Phase 4 regression tests, rollback plan - -### Risk 3: Orchestrator Reentrance -- **Impact**: Nested workflows don't work (JVNAUTOSCI-803) -- **Mitigation**: Phase 3 explicitly design for reentrance, Phase 4 tests - -### Risk 4: Performance Regression -- **Impact**: Structured calling slower than JSON-in-text -- **Mitigation**: Phase 4 benchmarking, <10ms target, optimization in Phase 5 - -## Metrics & Success Measures - -### Code Quality -- ✅ 100% type hint coverage (Pyright) -- ✅ 95%+ test coverage (pytest) -- ✅ Zero external dependency additions - -### Functionality -- ✅ 100% tool invocation success rate (vs ~90% with JSON parsing) -- ✅ 0 invalid tool executions (malformed JSON) -- ✅ 0 crashes from unknown tools -- ✅ 95%+ workflow completion rate (JVNAUTOSCI-803) - -### Performance -- ✅ <10ms overhead per tool call -- ✅ No increase in token usage -- ✅ Sub-100ms roundtrip latency (unchanged) - -### Reliability -- ✅ Feature flag enables safe rollout -- ✅ Fallback path always available -- ✅ Comprehensive regression test coverage - -## Timeline Estimates - -| Phase | Description | Effort | Status | ETA | -|-------|-------------|--------|--------|-----| -| **0** | Audit & Documentation | 3h | ✅ Complete | Oct 2025 | -| **1-2** | Core Types & Providers | 12h | ✅ Complete | Oct 2025 | -| **3** | Orchestrator Integration | 10h | ⏳ In Progress | Nov 2025 | -| **4** | Evaluation & Regression | 7h | ⏳ Planned | Nov 2025 | -| **5** | Deprecation & Cleanup | 5h | ⏳ Planned | Dec 2025 | -| **Total** | | 37h | | | - -## References - -### Related Issues -- **JVNAUTOSCI-799**: Structured tool calling (this issue) -- **JVNAUTOSCI-803**: LLM Workflows (depends on JVNAUTOSCI-799) -- **JVNAUTOSCI-698**: JSON action output prevention (original issue) -- **JVNAUTOSCI-640**: Von Agent Foundations (epic) - -### Documentation -- [Phase 0 Audit](jvnautosci_799_phase0_audit.md) -- [Usage Guide](structured_tool_calling_guide.md) -- [CONTRIBUTING.md](../../CONTRIBUTING.md) -- [AI Agent Guide](../../AGENTS.md) - -### External References -- OpenAI Function Calling: https://platform.openai.com/docs/guides/function-calling -- Gemini Function Calling: https://ai.google.dev/docs/function_calling -- JSON Schema: https://json-schema.org/ - -## Approval & Sign-Off - -**Issue**: JVNAUTOSCI-799 -**Epic**: JVNAUTOSCI-640 -**Created**: 2025-11-15 -**Last Updated**: 2025-11-15 -**Owner**: GitHub Copilot (AI Implementation) -**Reviewer**: Michael Witbrock (Von Lab) - ---- - -**Next Action**: Begin Phase 3 - Orchestrator Integration -- [ ] Examine full orchestrator structure -- [ ] Identify all tool invocation points -- [ ] Create MCPToolWorkflowExecutor wrapper -- [ ] Implement feature flag integration -- [ ] Add regression tests diff --git a/docs/engineering/jvnautosci_799_lessons_learned.md b/docs/engineering/jvnautosci_799_lessons_learned.md deleted file mode 100644 index 598e738ac..000000000 --- a/docs/engineering/jvnautosci_799_lessons_learned.md +++ /dev/null @@ -1,266 +0,0 @@ -# Lessons Learned - JVNAUTOSCI-799 - -> **Document status: Historical JVNAUTOSCI-799 implementation record from -> December 2025.** The design lessons may remain useful, but coverage, -> implementation-time, compatibility, and code-path claims are bounded to that -> milestone and require current verification. - -**Project:** Structured Tool Calling for Internal MCP -**Date:** December 2025 -**Implementation Time:** ~4 hours across all phases -**Code Impact:** ~500 lines production code, ~320 lines test code -**Test Coverage:** 100% of new code paths - ---- - -## Architecture & Design - -### 1. Bridge Pattern Over Direct Integration - -**Decision:** Created `generate_with_tools()` bridge method in `LLMInterface` rather than directly modifying existing `generate()` method. - -**Why It Worked:** -- Preserved backward compatibility completely -- Enabled feature flag control without code branching in consumers -- Made testing dual paths straightforward -- Reduced risk of breaking existing workflows - -**Lesson:** When integrating new capabilities into established systems, **bridge patterns enable safe, gradual migration** rather than risky big-bang replacements. - ---- - -### 2. Feature Flags Are Essential for Research Prototypes - -**Decision:** Made structured calling opt-in via `VON_INTERNAL_MCP_STRUCTURED_TOOL_CALLING` environment variable. - -**Why It Worked:** -- Enabled deployment without forcing all users to new path -- Allowed A/B testing structured vs legacy performance -- Provided instant rollback mechanism (set flag to "0") -- Reduced deployment anxiety - -**Lesson:** **Always ship new capabilities behind feature flags** in research systems where stability is critical but innovation is constant. - ---- - -### 3. Automatic Fallback > Manual Error Handling - -**Decision:** Structured calling failures automatically fall back to legacy path with warning log. - -**Why It Worked:** -- No service interruption when structured calling has issues -- Gradual degradation instead of hard failures -- Simplified error handling logic (no complex retry loops) -- Built confidence in deployment - -**Lesson:** **Design for graceful degradation** - new features should fail back to proven implementations rather than blocking workflows. - ---- - -## Implementation Challenges - -### 4. Schema Format Mismatch Discovery - -**Challenge:** MCP's internal Schema format (required/optional dicts) didn't match JSON Schema (properties + required array). - -**Solution:** Created `_mcp_schema_to_json_schema()` converter with type mapping. - -**Lesson:** **Always verify schema compatibility early** when integrating systems. What looks like "just JSON" often has subtle format differences that cause runtime errors. - ---- - -### 5. Python Class Structure Pitfall - -**Error Made:** Initially created module-level function `_python_type_to_json_schema_type()` that broke class method definitions following it. - -**Why It Failed:** Python classes can't be "resumed" after module-level code - once a class definition ends, it's final. - -**Fix:** Changed to static method `@staticmethod _python_type_to_json_schema_type()` within class. - -**Lesson:** **Keep helper functions as static methods** within classes unless they genuinely need to be module-level. Avoids class structure fragmentation. - ---- - -### 6. Dataclass Parameter Naming Matters - -**Error Made:** Used `ToolCall(name=..., arguments=...)` when actual parameters were `tool_name` and `payload`. - -**Root Cause:** Assumed naming convention without checking actual `@dataclass` definition. - -**Fix:** Read `types.py` to verify exact parameter names before creating instances. - -**Lesson:** **Don't assume dataclass parameter names** - always check the source definition, especially with frozen dataclasses that fail fast on wrong parameters. - ---- - -## Testing Strategy - -### 7. Test Both Paths Independently - -**Approach:** Created separate tests for: -- Structured calling path enabled -- Feature flag disabled (legacy only) -- Client without structured support -- Exception fallback - -**Why It Worked:** -- Verified both paths work independently -- Ensured feature flag actually controls behaviour -- Caught edge cases (e.g., what if client doesn't have method?) -- Provided regression coverage for legacy path - -**Lesson:** **When implementing dual paths, test each path independently** plus the switching logic between them. - ---- - -### 8. Mock Complexity vs. Reality - -**Challenge:** Mock LLM clients needed to realistically simulate both `generate()` and `generate_with_tools()` responses. - -**Solution:** Created `MockLLMClientWithTools` class that tracks which methods were called, plus separate `MockLLMClientLegacyOnly` for old-style client. - -**Lesson:** **Design test mocks to mirror actual usage patterns** - don't just return static values, simulate the actual call semantics your code expects. - ---- - -### 9. Schema Conversion Tests Are Critical - -**Insight:** Schema conversion (`_mcp_schema_to_json_schema()`) had the highest bug potential due to: -- Type mapping complexity -- Required vs optional field handling -- Union type edge cases - -**Solution:** Dedicated test (`test_mcp_schema_to_json_schema_conversion`) with explicit assertions on output structure. - -**Lesson:** **Test data transformation functions thoroughly** - they're often the source of subtle runtime bugs when schemas evolve. - ---- - -## Code Quality - -### 10. Preserve Safety Constraints Explicitly - -**Approach:** Documented in code comments and tests that namespace injection, Gmail profile handling, and whitelist validation must work in both paths. - -**Why It Matters:** Security constraints are easy to accidentally bypass when adding new code paths. - -**Verification:** Created `test_namespace_injection_preserved()` to ensure safety constraints survived refactoring. - -**Lesson:** **Make safety constraints testable and test them explicitly** - don't rely on manual code review to catch security regressions. - ---- - -### 11. call_id Tracing Enables Debugging - -**Decision:** Preserved `ToolCall.call_id` throughout the execution flow and logged it. - -**Future Benefit:** Enables end-to-end tracing correlation (JVNAUTOSCI-803) once workflow system uses it. - -**Lesson:** **Add execution identifiers early** even if downstream systems don't consume them yet - they're much harder to retrofit later. - ---- - -## Process Insights - -### 12. Incremental Implementation Beats Big Bang - -**Approach:** Implemented in phases: -1. Phase 0: Core types and unified client -2. Phase 1: Provider implementations -3. Phase 2: Validation and cross-provider testing -4. Phase 3: Orchestrator integration - -**Why It Worked:** -- Each phase was independently testable -- Early phases caught issues before they propagated -- Could deploy phases incrementally -- Reduced cognitive load (focus on one layer at a time) - -**Lesson:** **Break large refactorings into independently valuable phases** - each phase should leave the system in a working state. - ---- - -### 13. Documentation During Development, Not After - -**Approach:** Created `phase3_orchestrator_integration_summary.md` while implementing, not as post-work documentation. - -**Benefits:** -- Captured design decisions while fresh in mind -- Served as implementation checklist -- Caught missing test coverage early -- Reduced knowledge transfer burden - -**Lesson:** **Write design docs during implementation** - they're more accurate and easier to produce than post-hoc documentation. - ---- - -## What We'd Do Differently - -### 14. Type Hints Could Be Stricter - -**Observation:** Some methods use `Any` for flexibility (e.g., `llm_client: Any` in `run()`). - -**Improvement:** Could define `ProtocolLLMClient` with `generate()` and optional `generate_with_tools()` for better type safety. - -**Trade-off:** Research prototype needs flexibility; premature type constraints can slow experimentation. - -**Lesson:** **Balance type safety with research velocity** - add constraints when patterns stabilize, not during initial exploration. - ---- - -### 15. Performance Metrics From Day One - -**Missing:** We don't have structured/legacy path performance comparison data yet. - -**Should Have:** Built-in latency/token usage logging to compare approaches objectively. - -**Lesson:** **Instrument code for performance comparison during development** - harder to add metrics after deployment when baselines are lost. - ---- - -## Key Takeaways - -1. ✅ **Bridge patterns enable safe migration** in production systems -2. ✅ **Feature flags are essential** for research prototypes -3. ✅ **Automatic fallback prevents outages** when new features fail -4. ✅ **Test both paths independently** when implementing dual code paths -5. ✅ **Preserve safety constraints explicitly** with dedicated tests -6. ✅ **Document during implementation**, not after -7. ✅ **Incremental phases reduce risk** compared to big-bang rewrites -8. ✅ **Execution IDs enable tracing** - add them early, use them later - ---- - -## Impact Summary - -| Metric | Value | -|--------|-------| -| **Total Implementation Time** | ~4 hours across all phases | -| **Production Code** | ~500 lines | -| **Test Code** | ~320 lines | -| **Test Coverage** | 100% of new code paths | -| **Deployment Risk** | Low (feature flag + automatic fallback) | -| **Breaking Changes** | Zero | -| **Legacy Path Preserved** | Yes, with automatic fallback | - ---- - -## Related Documentation - -- [Phase 3 Orchestrator Integration Summary](phase3_orchestrator_integration_summary.md) -- [Structured Tool Calling Guide](structured_tool_calling_guide.md) -- [Implementation Roadmap](jvnautosci_799_implementation_roadmap.md) -- [Final Status Report](jvnautosci_799_final_status.md) - ---- - -## Applicability to Future Work - -These lessons are particularly relevant for: - -- **JVNAUTOSCI-803**: LLM Workflows - execution tracing architecture -- **Future provider integrations**: Claude, Cohere, etc. -- **Research feature deployment**: Any new capabilities needing gradual rollout -- **System refactoring**: Large-scale changes to established codebases - -The bridge pattern + feature flag + automatic fallback strategy should be the **default approach** for introducing new capabilities in Von's research environment. diff --git a/docs/engineering/jvnautosci_799_phase0_audit.md b/docs/engineering/jvnautosci_799_phase0_audit.md deleted file mode 100644 index 5e72b3bb4..000000000 --- a/docs/engineering/jvnautosci_799_phase0_audit.md +++ /dev/null @@ -1,317 +0,0 @@ -# PHASE 0 AUDIT: Current Tool-Call Patterns - -> **Document status: Historical JVNAUTOSCI-799 implementation record from -> November 2025.** “Current” below means current at that audit date. Line -> numbers, invocation paths, gaps, and baseline findings are not guaranteed to -> describe present Von. Revalidate the current code path, targeted tests, and -> live authority surfaces before relying on them. - -This document captured the tool-invocation baseline for JVNAUTOSCI-799 Phase 0. - -## 1. Current Invocation Paths - -### 1.1 Internal MCP Orchestrator (`src/backend/integrations/internal_mcp/orchestrator.py`) - -**JSON-in-Text Parsing**: -- Method: `_extract_json_blob()` (line 889-969) -- Pattern: Model outputs tool calls as JSON object embedded in text -- Format: `{"action": "call_tool", "tool": "...", "payload": {...}}` -- Validation: Tolerates Markdown fences and trailing prose - -**Safety Constraints** (must preserve in Phase 3): -1. Single-tool-call contract: Only one tool call per response -2. Namespace injection rules: Tool names must match registered tools -3. Untrusted content: No tool execution from user-pasted JSON -4. Strict JSON parsing: Malformed JSON raises `ToolCallParsingError` - -**Failure Modes** (documented in JVNAUTOSCI-698, JVNAUTOSCI-799): -1. Model hallucination: "I cannot access tools" despite server truth showing available tools -2. Parsing ambiguity: Trailing `}x` or multiple JSON objects cause failures -3. Non-determinism: Different runs of same prompt may parse differently - -### 1.2 Prompt-Introspection Fast-Path (`/von/generate` endpoint) - -**Location**: `src/backend/server/chat_routes.py` or `routes.py` - -**Purpose**: Deterministically answer runtime state questions without parsing ambiguity -- Returns: Prompt IDs, text, model configuration -- Avoids: Model self-reporting runtime state - -**Temporary Mitigation**: Fast-path should be removed post-JVNAUTOSCI-799 (Phase 5) - -### 1.3 Annotation Extraction Service - -**Location**: `src/backend/services/annotation_extraction_service.py` - -**Current Pattern**: -1. Load prompt from Vontology (`#V#find_concepts_in_text_prompt`) -2. Invoke LLM -3. Parse JSON response with fallback to regex extraction -4. Map type labels → concept IDs - -**Tool Interaction**: Doesn't currently invoke tools, but could benefit from structured LLM responses - -### 1.4 Tell-Von Agent (Task Inference) - -**Status**: Under development (JVNAUTOSCI-552) - -**Anticipated Tool Usage**: -- Natural language → structured task specification -- May require tool calling for intent clarification - -## 2. Current Contract & Semantics - -### 2.1 Single-Tool-Call Semantics - -**Current Assumption**: -- Each LLM response contains at most one tool call -- Multiple tools require sequential rounds - -**Implication for Workflows** (JVNAUTOSCI-803): -- Conditional workflows: IF tool X succeeds THEN tool Y -- Sequential workflows: tool A → tool B → tool C -- Parallel workflows: NOT SUPPORTED YET (future feature) - -### 2.2 Tool Namespace Isolation - -**Current Pattern**: -- Internal MCP tools (gateway provides ~50+ tools) -- User-provided tools (via stored prompts): FUTURE -- External tools (Zapier, etc.): FUTURE (Phase 4+) - -**Safety Model**: -- Whitelist of allowed tools per user/org/context -- No direct access to arbitrary executables -- Tool execution logged and auditable - -### 2.3 Retry Semantics - -**Current Behaviour**: -- No automatic retry at tool call level -- Orchestrator implements max_tool_invocations loop (default 1, max 8) -- Model can decide to call another tool or respond with text - -**For Workflows** (JVNAUTOSCI-803): -- Retry logic should be at step level (workflow step executor) -- Exponential backoff, max retries, timeout safeguards - -## 3. Observed Failure Examples - -### 3.1 Runtime State Hallucination (JVNAUTOSCI-799) - -**Symptom**: Model claims "no user-specific prompt active" -**Server Truth**: `user_prompt.loaded=true`, concept IDs present -**Root Cause**: Model trained on limited context; cannot introspect runtime state - -**Mitigation** (Temporary - Phase 5 removes fast-path): -``` -/von/generate?query=is_prompt_active -Response: {"active": true, "prompt_id": "#V#my_prompt", "text": "..."} -``` - -### 3.2 JSON Parsing Ambiguity (JVNAUTOSCI-698) - -**Symptom**: Tool call with trailing `}x` fails parsing -**Current Handling**: Strict JSONDecodeError raises `ToolCallParsingError` -**Root Cause**: Model sometimes appends explanatory text without proper separation - -**Mitigation** (JVNAUTOSCI-799 Phase 1-2): -Structured tool calling replaces this with provider-native API - -### 3.3 Multiple JSON Values in Response - -**Symptom**: Model outputs multiple tool calls or multiple JSON objects -**Current Handling**: Only first valid JSON object accepted (strict mode) -**Future Handling**: Structured calling returns List[ToolCall] - -## 4. Provider-Specific Considerations - -### 4.1 OpenAI (GPT-4, GPT-3.5 Turbo) - -**Native Support**: Function calling API (via `tools` parameter) -**Advantages**: -- Deterministic structured output -- Reliable tool invocation -- Token usage tracking - -**Adoption Strategy** (Phase 2): -1. Implement `OpenAIClient.generate_with_tools()` -2. Map internal `ToolDefinition` → OpenAI `tools` schema -3. Parse `message.tool_calls` → canonical `ToolCall` list -4. Enable via feature flag `VON_INTERNAL_MCP_STRUCTURED_TOOL_CALLING=1` - -### 4.2 Gemini (Gemini Pro, Pro Vision) - -**Native Support**: Function calling API (via `tools` parameter) -**Implementation**: Similar to OpenAI but with Gemini-specific schema mapping - -### 4.3 Ollama (Local Models) - -**Native Support**: None (GGML models don't have function calling) -**Fallback Strategy**: -1. Grammar-based constrained generation (if GGML grammar available) -2. JSON-in-text parsing with strict validation (default) -3. Prompt engineering to elicit proper JSON format - -**Migration Path**: Ollama users can upgrade to models with better JSON support - -## 5. Integration Points with JVNAUTOSCI-803 (LLM Workflows) - -### 5.1 Workflow Step Executor Requirements - -**What Workflows Need** (from JVNAUTOSCI-803): -1. Structured tool calls (not text parsing) -2. Deterministic call_id for execution tracing -3. ToolDefinition schema compatible with workflow step definitions -4. Async/sync support for concurrent steps (parallel workflows) - -**What JVNAUTOSCI-799 Provides**: -1. `LLMClient` abstract interface with `generate_with_tools()` -2. `ToolCall` with `call_id` for tracing -3. `ToolDefinition` with `output_schema` for validation -4. Both async and sync methods - -### 5.2 Execution Trace Recording - -**Workflow Execution Traces Need**: -- `ToolCall.call_id` to correlate with step results -- Deterministic tool invocation (no parsing ambiguity) -- Clear success/failure semantics (tool executed vs tool call invalid) - -**JVNAUTOSCI-799 Enables This**: -- Structured `LLMResponse.tool_calls` (not text extraction) -- Preserves call_id through entire execution pipeline -- Provider-native APIs guarantee determinism - -## 6. Feature Flag Strategy - -### 6.1 Rollout Plan - -**Default**: `VON_INTERNAL_MCP_STRUCTURED_TOOL_CALLING=1` (enabled) - -**Disable for Debugging**: Set to `0` to revert to JSON-in-text parsing - -**Legacy Fallback**: `VON_LEGACY_JSON_TEXT_PARSING=1` (disabled by default) - -### 6.2 Transition Timeline - -- Phase 1-2 (Weeks 1-2): Implement `OpenAIClient` with feature flag enabled -- Phase 2 (Weeks 2-3): Add Gemini/Ollama adapters -- Phase 3 (Weeks 3-4): Integrate with orchestrator -- Phase 4-5 (Weeks 5+): Evaluate and deprecate legacy paths - -## 7. Test Coverage Strategy - -### 7.1 Regression Tests - -Required tests to ensure JVNAUTOSCI-799 fixes original issues: - -1. **Runtime State Hallucination**: - - Test: "is_prompt_active" question must use tool-backed evidence - - Verify: Model cannot claim "prompt not active" when it is - -2. **JSON Parsing Reliability**: - - Test: Malformed JSON with trailing `}x` does not execute tool - - Verify: `ToolCallError` raised, not silent failure - -3. **Tool Name Validation**: - - Test: Requesting unknown tool does not execute - - Verify: Error logged, tool call skipped - -### 7.2 Acceptance Tests (from JVNAUTOSCI-799) - -- Orchestrator uses structured calls (not `_extract_json_blob`) -- Prompt/model/RAG introspection returns tool-backed evidence -- Safety constraints preserved (whitelist, namespace rules) -- Tests cover observed failure modes - -### 7.3 Integration Tests (for JVNAUTOSCI-803) - -- Workflow step executes LLM action with tool calling -- Tool call `call_id` roundtrips through execution trace -- Multi-step workflows with tool dependencies work reliably - -## 8. Backward Compatibility - -### 8.1 Adapter Pattern - -Existing code continues using `llm_interface.py`: - -```python -# OLD API (still works) -from src.backend.languagemodels.llm_interface import get_llm_client -client = get_llm_client() -response = client.generate(prompt, context) - -# NEW API (structured tool calling) -from src.backend.languagemodels.structured_tool_calling import get_llm_client, LLMClientConfig -config = LLMClientConfig(model="gpt-4") -client = get_llm_client(config) -response = client.generate_with_tools_sync(prompt, tools, context) -``` - -### 8.2 Gradual Migration - -1. Orchestrator: Switch to `LLMClient.generate_with_tools()` (Phase 3) -2. Annotation: Optional migration to structured responses (Phase 3) -3. New workflows: Use structured calling by default (JVNAUTOSCI-803) -4. Legacy paths: Remove post-Phase 5 (deprecation notice added to `llm_interface.py`) - -## 9. Known Limitations & Future Work - -### 9.1 Current Limitations - -1. **Ollama**: No native function calling → relies on prompt engineering + JSON parsing -2. **Multiple Tools**: Single-call contract → parallel workflows require sequence -3. **Tool Composition**: No tool chaining (tool A output → tool B input) - -### 9.2 Future Enhancements (Phase 4+) - -1. **Parallel Tool Execution**: Multiple tools in single LLM response -2. **Tool Chaining**: Automatic dataflow between tool calls -3. **Conditional Execution**: Tools selected based on input classification -4. **Adaptive Retry**: Retry strategy learns from success/failure patterns - -## Conclusion - -This audit confirms that JVNAUTOSCI-799's structured tool calling directly addresses -three categories of current problems: - -1. **Reliability**: Deterministic tool invocation replaces ambiguous JSON parsing -2. **Observability**: Structured `ToolCall` objects enable accurate execution traces -3. **Workflow Foundation**: Canonical interfaces enable JVNAUTOSCI-803 (LLM Workflows) - -The implementation prioritises OpenAI first (highest reliability requirement), then -adds Gemini and Ollama support for breadth of coverage. - -# Summary Table: Current Patterns vs Proposed - -```text -Current State (JSON-in-Text): -┌─────────────────────┬─────────────────────────────┐ -│ Aspect │ Current Implementation │ -├─────────────────────┼─────────────────────────────┤ -│ Tool Invocation │ JSON extracted from text │ -│ Parser │ Custom _extract_json_blob() │ -│ Reliability │ Prone to hallucination │ -│ Tracing │ No call_id or structured │ -│ Multi-Tool Support │ Single tool per response │ -│ Provider Support │ All (via text parsing) │ -│ Async Support │ None │ -│ Retry Logic │ In orchestrator loop │ -└─────────────────────┴─────────────────────────────┘ - -Proposed State (JVNAUTOSCI-799): -┌─────────────────────┬─────────────────────────────┐ -│ Aspect │ Structured Calling │ -├─────────────────────┼─────────────────────────────┤ -│ Tool Invocation │ Provider-native API │ -│ Parser │ Provider SDK (no custom) │ -│ Reliability │ Deterministic │ -│ Tracing │ call_id on ToolCall │ -│ Multi-Tool Support │ Future (Phase 4) │ -│ Provider Support │ OpenAI, Gemini, Ollama │ -│ Async Support │ async/sync both available │ -│ Retry Logic │ At adapter layer │ -└─────────────────────┴─────────────────────────────┘ -``` diff --git a/docs/engineering/jvnautosci_799_phases_1-2_summary.md b/docs/engineering/jvnautosci_799_phases_1-2_summary.md deleted file mode 100644 index 1706739ee..000000000 --- a/docs/engineering/jvnautosci_799_phases_1-2_summary.md +++ /dev/null @@ -1,556 +0,0 @@ -# JVNAUTOSCI-799 Phases 1-2: Implementation Summary - -> **Document status: Historical JVNAUTOSCI-799 implementation record -> (November–December 2025).** Completion, coverage, code-size, and replacement -> claims below are bounded to that milestone and are not current system-level -> evidence. Revalidate the current implementation and targeted tests. - -**Status**: ✅ Phases 0-2 Complete -**Date Completed**: 2025-11-15 -**Lines of Code**: ~2000 (core + tests + documentation) -**Tests Passing**: 23/23 (100%) -**Type Coverage**: 100% (Pyright) - -## What Was Built - -### Core Package: `src/backend/languagemodels/structured_tool_calling/` - -A unified, provider-agnostic structured tool calling system that replaces JSON-in-text parsing with provider-native function calling. - -#### 1. **Type System** (`types.py`, 150 lines) - -**`ToolDefinition`**: Describes available tools -```python -@dataclass(frozen=True) -class ToolDefinition: - name: str # Unique identifier - description: str # Purpose & usage - input_schema: Dict[str, Any] # JSON Schema for inputs - output_schema: Optional[Dict[str, Any]] = None # Expected output format -``` -- ✅ Validation: Name must be non-empty, schema must have type/properties -- ✅ Immutable: frozen dataclass prevents accidental modification - -**`ToolCall`**: Canonical tool invocation -```python -@dataclass(frozen=True) -class ToolCall: - tool_name: str # Which tool to invoke - payload: Dict[str, Any] # Arguments (matched against input_schema) - call_id: str = field(default_factory=lambda: str(uuid4())) # UUID for tracing - timestamp: datetime = field(default_factory=lambda: datetime.now(timezone.utc)) # UTC -``` -- ✅ Auto-generated `call_id`: Enables execution tracing (required by JVNAUTOSCI-803) -- ✅ UTC timestamp: Consistent time tracking across timezones - -**`LLMResponse`**: Unified response format -```python -@dataclass -class LLMResponse: - text_response: str # Natural language output - tool_calls: List[ToolCall] = field(default_factory=list) # Structured invocations - raw_response: Optional[Dict[str, Any]] = None # Provider-specific (debug) - model: Optional[str] = None # Model name - usage: Optional[Dict[str, Any]] = None # Token counts -``` -- ✅ Supports text-only, tool-calls-only, or combined responses -- ✅ Optional fields for flexibility - -**`ToolCallError`**: Exception for failures -```python -class ToolCallError(Exception): - """Raised when tool calling fails (provider error, validation failure, etc.)""" -``` - -#### 2. **Client Interface** (`client.py`, 200 lines) - -**`LLMClientConfig`**: Configuration for all clients -```python -@dataclass -class LLMClientConfig: - model: str # Model identifier (required) - api_key: Optional[str] = None # Authentication - base_url: Optional[str] = None # API endpoint (proxies, self-hosted) - temperature: float = 0.7 # Sampling (0-2) - max_tokens: Optional[int] = None # Output limit - enable_structured_calling: bool = True # Use structured calling (feature flag) - fallback_to_json_text: bool = True # Fallback if structured fails -``` -- ✅ Sensible defaults (temperature, flags) -- ✅ Optional auth/endpoint for flexibility - -**`LLMClient`**: Abstract base class -```python -class LLMClient(ABC): - """Abstract interface for all provider implementations""" - - async def generate_with_tools( - self, - prompt: str, - available_tools: List[ToolDefinition], - system_message: Optional[str] = None, - context: Optional[Dict[str, Any]] = None, - ) -> LLMResponse: - """Generate response with access to tools (provider-native or fallback)""" - ... - - def generate_with_tools_sync( - self, - prompt: str, - available_tools: List[ToolDefinition], - system_message: Optional[str] = None, - context: Optional[Dict[str, Any]] = None, - ) -> LLMResponse: - """Synchronous wrapper (uses asyncio.run())""" - ... -``` -- ✅ Async-first (required for workflow concurrency) -- ✅ Sync wrapper for backward compatibility -- ✅ Provider implementations override `async generate_with_tools()` - -#### 3. **Provider Implementations** (`providers/`, 700 lines) - -**OpenAI** (`openai_client.py`, 180 lines) -```python -class OpenAIClient(LLMClient): - """OpenAI GPT-4, GPT-3.5 Turbo with native function calling""" - - async def generate_with_tools(self, ...): - # Build message list with tools - tools_param = [self._tool_definition_to_dict(tool) for tool in available_tools] - - # Call OpenAI with tools parameter - response = await self.client.chat.completions.create( - model=self.config.model, - messages=messages, - tools=tools_param, - ... - ) - - # Parse response - tool_calls = [ - ToolCall(tool_name=tc.function.name, payload=json.loads(tc.function.arguments)) - for tc in response.choices[0].message.tool_calls or [] - ] - - return LLMResponse( - text_response=response.choices[0].message.content or "", - tool_calls=tool_calls, - model=response.model, - usage={ - "prompt_tokens": response.usage.prompt_tokens, - "completion_tokens": response.usage.completion_tokens, - }, - ) -``` -- ✅ Uses native `tools` parameter (deterministic) -- ✅ Full token usage tracking -- ✅ Both async and sync support - -**Gemini** (`gemini_client.py`, 220 lines) -```python -class GeminiClient(LLMClient): - """Google Gemini with native function calling""" - - async def generate_with_tools(self, ...): - # Convert ToolDefinition → Gemini schema - tools = [ - genai.types.Tool( - function_declarations=[ - genai.types.FunctionDeclaration( - name=tool.name, - description=tool.description, - parameters=genai.types.Schema( - type=genai.types.Type.OBJECT, - properties={...}, - ), - ) - for tool in available_tools - ] - ) - ] - - # Call Gemini (execute in thread pool to avoid blocking) - loop = asyncio.get_event_loop() - response = await loop.run_in_executor( - None, - lambda: self.model.generate_content( - messages, - tools=tools, - ... - ), - ) - - # Parse tool calls - tool_calls = [ - ToolCall(tool_name=tc.name, payload=tc.args) - for tc in response.candidates[0].content.parts - if isinstance(tc, genai.types.FunctionCall) - ] - - return LLMResponse(text_response=..., tool_calls=tool_calls, model="gemini-pro") -``` -- ✅ Native function calling (Gemini API) -- ✅ Schema mapping (JSON Schema → Gemini types) -- ✅ Thread pool execution (async-blocking compatibility) - -**Ollama** (`ollama_client.py`, 280 lines) -```python -class OllamaClient(LLMClient): - """Local models (Llama 2, Mistral, etc.) - no native function calling""" - - async def generate_with_tools(self, ...): - # Build constrained prompt with tool definitions - prompt = self._build_constrained_prompt(available_tools, prompt) - - # Stream response from Ollama - response_text = "" - async for chunk in self.client.generate(model=self.config.model, prompt=prompt): - response_text += chunk.response - - # Extract JSON tool calls from text - tool_calls = self._extract_json_tool_calls(response_text, available_tools) - - return LLMResponse(text_response=response_text, tool_calls=tool_calls) - - def _build_constrained_prompt(self, tools: List[ToolDefinition], prompt: str) -> str: - """Embed tool definitions and JSON instructions in prompt""" - instructions = """ - Available tools: - {tools_list} - - If you need to use a tool, respond with a JSON object: - {{ - "action": "use_tool", - "tool_name": "tool_name", - "arguments": {{...}} - }} - - You can use multiple tools by responding with JSON objects separated by newlines. - """ - return instructions + prompt - - def _extract_json_tool_calls(self, text: str, available_tools: List[ToolDefinition]) -> List[ToolCall]: - """Extract JSON tool calls from text response""" - tool_calls = [] - # Find all JSON objects in response - for json_match in re.finditer(r'\{.*?\}', text, re.DOTALL): - try: - obj = json.loads(json_match.group()) - if obj.get("action") == "use_tool" and obj.get("tool_name"): - # Validate tool exists - if any(t.name == obj["tool_name"] for t in available_tools): - tool_calls.append(ToolCall( - tool_name=obj["tool_name"], - payload=obj.get("arguments", {}), - )) - except json.JSONDecodeError: - continue # Skip malformed JSON - - return tool_calls -``` -- ✅ Graceful fallback (no native function calling) -- ✅ Constrained prompt engineering (guides JSON formatting) -- ✅ JSON extraction with strict validation -- ✅ Supports local models (Llama 2, Mistral, etc.) - -#### 4. **Factory Function** (`factory.py`, 30 lines) - -```python -def get_llm_client(config: LLMClientConfig) -> LLMClient: - """Factory function to select provider by model name""" - - model_lower = config.model.lower() - - if model_lower.startswith("gpt-"): - return OpenAIClient(config) - elif model_lower.startswith("gemini-"): - return GeminiClient(config) - elif model_lower.startswith(("llama", "mistral", "neural-chat")): - return OllamaClient(config) - else: - # Default to Ollama for unknown models - return OllamaClient(config) -``` -- ✅ Pattern matching on model name -- ✅ Fallback to Ollama for unknown models -- ✅ Extensible (add new patterns for new providers) - -### Testing: 23/23 Passing ✅ - -**Type Tests** (`test_structured_tool_calling_types.py`, 14 tests) -``` -test_tool_definition_valid PASSED -test_tool_definition_with_output_schema PASSED -test_tool_definition_missing_name PASSED -test_tool_definition_empty_name PASSED -test_tool_definition_invalid_schema_no_type PASSED -test_tool_call_creation PASSED -test_tool_call_auto_generates_call_id PASSED -test_tool_call_has_timestamp PASSED -test_tool_call_timestamp_is_utc PASSED -test_llm_response_text_only PASSED -test_llm_response_with_tool_calls PASSED -test_llm_response_usage_tracking PASSED -test_tool_call_error_exception PASSED -test_llm_response_has_tool_calls_method PASSED -``` - -**Client Interface Tests** (`test_structured_tool_calling_client.py`, 9 tests) -``` -test_llm_client_config_creation PASSED -test_llm_client_config_defaults PASSED -test_get_llm_client_openai PASSED -test_get_llm_client_gemini PASSED -test_get_llm_client_ollama_explicit PASSED -test_get_llm_client_unknown_defaults_to_ollama PASSED -test_validate_input_schema_valid PASSED -test_validate_input_schema_missing_type PASSED -test_validate_input_schema_empty_schema PASSED -``` - -**Validation**: All tests run with 0 failures, 0 warnings (after datetime fix) - -### Documentation - -1. **Phase 0 Audit** (`jvnautosci_799_phase0_audit.md`, 400+ lines) - - Current tool-call patterns documented - - Failure modes with examples - - Provider-specific considerations - - Integration points with JVNAUTOSCI-803 - -2. **Implementation Roadmap** (`jvnautosci_799_implementation_roadmap.md`, 300+ lines) - - Phase breakdown (0-5) - - Status tracking - - Acceptance criteria - - Timeline estimates - -3. **Usage Guide** (`structured_tool_calling_guide.md`, 400+ lines) - - Quick start examples - - Architecture overview - - Best practices - - Troubleshooting - -## How to Validate - -### 1. Run All Tests - -```powershell -# Types tests -pdm run pytest tests/backend/test_structured_tool_calling_types.py -v - -# Client interface tests -pdm run pytest tests/backend/test_structured_tool_calling_client.py -v - -# All together -pdm run pytest tests/backend/test_structured_tool_calling*.py -v -``` - -Expected output: -``` -23 passed in 3.81s -``` - -### 2. Check Type Hints - -```powershell -# Validate type coverage -pdm run pyright src/backend/languagemodels/structured_tool_calling/ - -# Should show 0 errors -``` - -### 3. Verify Imports - -```powershell -# Test that all imports work -pdm run python -c "from src.backend.languagemodels.structured_tool_calling import *; print('OK')" - -# Should output: OK -``` - -### 4. Quick Integration Test - -```python -# Example: Create a client and define a tool -from src.backend.languagemodels.structured_tool_calling import ( - LLMClientConfig, - ToolDefinition, - get_llm_client, -) - -config = LLMClientConfig(model="ollama:llama2") # Ollama local model -client = get_llm_client(config) - -tools = [ - ToolDefinition( - name="search", - description="Search the knowledge base", - input_schema={ - "type": "object", - "properties": { - "query": {"type": "string", "description": "Search query"}, - }, - "required": ["query"], - }, - ), -] - -# This will work (Ollama client will gracefully handle) -response = client.generate_with_tools_sync( - prompt="Search for information about concepts", - available_tools=tools, -) - -print(f"Response: {response.text_response}") -if response.has_tool_calls(): - for tc in response.tool_calls: - print(f"Tool called: {tc.tool_name}") -``` - -## Architecture Diagram - -``` -┌─────────────────────────────────────────────────────┐ -│ LLMClient (Abstract Interface) │ -├─────────────────────────────────────────────────────┤ -│ async generate_with_tools(prompt, tools) │ -│ generate_with_tools_sync(prompt, tools) ← wrapper │ -└────────────────────────┬────────────────────────────┘ - │ - ┌───────────────┼───────────────┐ - │ │ │ - ┌────▼────┐ ┌──────▼──────┐ ┌──▼──────────┐ - │ OpenAI │ │ Gemini │ │ Ollama │ - │ Client │ │ Client │ │ Client │ - └────┬────┘ └──────┬──────┘ └──┬──────────┘ - │ │ │ - [GPT-4] [Gemini Pro] [Local Models] - │ │ │ - Native Tools Native Tools Prompt Engineering - │ │ │ - └───────────────┼─────────────┘ - │ - ┌───────────────▼───────────────┐ - │ Canonical LLMResponse │ - │ - text_response: str │ - │ - tool_calls: List[ToolCall] │ - │ - model: str │ - │ - usage: Dict │ - └───────────────────────────────┘ - │ - ┌───────────────▼───────────────┐ - │ Orchestrator/Agent │ - │ (Phase 3 Integration) │ - └───────────────────────────────┘ -``` - -## Key Design Decisions - -### 1. Frozen Dataclasses for Immutability -```python -@dataclass(frozen=True) -class ToolCall: - ... -``` -**Why**: Prevents accidental mutation of tool calls (important for tracing, replay) - -### 2. Auto-Generated UUID call_id -```python -call_id: str = field(default_factory=lambda: str(uuid4())) -``` -**Why**: Enables execution tracing (critical for JVNAUTOSCI-803 workflows) - -### 3. Abstract LLMClient Base Class -```python -class LLMClient(ABC): - @abstractmethod - async def generate_with_tools(self, ...): - ... -``` -**Why**: Ensures all providers implement same interface, enables polymorphism - -### 4. Async-First with Sync Wrapper -```python -async def generate_with_tools(self, ...): - ... # Provider-specific async implementation - -def generate_with_tools_sync(self, ...): - return asyncio.run(self.generate_with_tools(...)) -``` -**Why**: Async required for workflow concurrency, sync wrapper for compatibility - -### 5. Provider-Agnostic Canonical Types -```python -# ToolCall, ToolDefinition, LLMResponse are provider-agnostic -# Providers translate provider-specific responses → canonical types -``` -**Why**: Orchestrator code doesn't need provider-specific knowledge - -## What's Next: Phase 3 - -The implementation is ready for **Phase 3: Orchestrator Integration**. - -**Next Steps**: -1. ✅ Examine `InternalMCPChatOrchestrator.run()` method -2. ✅ Replace `_extract_json_blob()` with `generate_with_tools()` -3. ✅ Create `MCPToolWorkflowExecutor` wrapper for JVNAUTOSCI-803 -4. ✅ Add feature flag integration -5. ✅ Implement regression tests - -**Success Criteria** (Phase 3): -- Orchestrator uses structured calling as primary path -- All existing tool invocations work identically -- Tool call `call_id` available for execution tracing -- Feature flags enable safe rollout -- Fallback to JSON-in-text if structured fails - -## File Locations - -``` -Core Implementation: - src/backend/languagemodels/structured_tool_calling/ - __init__.py (20 lines) - types.py (150 lines) - client.py (200 lines) - factory.py (30 lines) - providers/ - __init__.py - openai_client.py (180 lines) - gemini_client.py (220 lines) - ollama_client.py (280 lines) - -Tests: - tests/backend/ - test_structured_tool_calling_types.py (14 tests) - test_structured_tool_calling_client.py (9 tests) - -Documentation: - docs/engineering/ - jvnautosci_799_phase0_audit.md (Phase 0) - jvnautosci_799_implementation_roadmap.md (Timeline) - structured_tool_calling_guide.md (Usage) - jvnautosci_799_phases_1-2_summary.md (This file) -``` - -## Summary Statistics - -| Metric | Value | -|--------|-------| -| **Lines of Code (Core)** | ~700 | -| **Lines of Code (Tests)** | ~400 | -| **Lines of Code (Docs)** | ~1000 | -| **Total Lines** | ~2100 | -| **Test Coverage** | 23/23 passing (100%) | -| **Type Coverage** | 100% (Pyright) | -| **Providers Implemented** | 3 (OpenAI, Gemini, Ollama) | -| **Public Methods** | 8 (generate_with_tools, factory, etc.) | -| **Data Classes** | 4 (Config, ToolDefinition, ToolCall, LLMResponse) | -| **Estimated Phase 3 Effort** | 10 hours | - ---- - -**Status**: Ready for Phase 3 - Orchestrator Integration -**Created**: 2025-11-15 -**By**: GitHub Copilot (AI Implementation Assistant) -**For**: JVNAUTOSCI-799, JVNAUTOSCI-803 diff --git a/docs/engineering/jvnautosci_799_quick_reference.md b/docs/engineering/jvnautosci_799_quick_reference.md index b42b260ba..5fda006c5 100644 --- a/docs/engineering/jvnautosci_799_quick_reference.md +++ b/docs/engineering/jvnautosci_799_quick_reference.md @@ -396,9 +396,9 @@ VON_LEGACY_JSON_TEXT_PARSING # Force legacy (default: 0) ## Links - [Full Guide](structured_tool_calling_guide.md) -- [Implementation Roadmap](jvnautosci_799_implementation_roadmap.md) -- [Phase 0 Audit](jvnautosci_799_phase0_audit.md) -- [Phases 1-2 Summary](jvnautosci_799_phases_1-2_summary.md) +- [Implementation Roadmap](https://github.com/Strong-AI-Lab/Von/blob/5b2f8b976069935da73b82bca433fb3cefffcd3d/docs/engineering/jvnautosci_799_implementation_roadmap.md) +- [Phase 0 Audit](https://github.com/Strong-AI-Lab/Von/blob/5b2f8b976069935da73b82bca433fb3cefffcd3d/docs/engineering/jvnautosci_799_phase0_audit.md) +- [Phases 1-2 Summary](https://github.com/Strong-AI-Lab/Von/blob/5b2f8b976069935da73b82bca433fb3cefffcd3d/docs/engineering/jvnautosci_799_phases_1-2_summary.md) - JIRA: JVNAUTOSCI-799 - Related: JVNAUTOSCI-803 (LLM Workflows) diff --git a/docs/engineering/llm_workflows_design.md b/docs/engineering/llm_workflows_design.md deleted file mode 100644 index 9eb0ad9e1..000000000 --- a/docs/engineering/llm_workflows_design.md +++ /dev/null @@ -1,1239 +0,0 @@ -# LLM Workflows: Design Document - -> **Document status: Historical design precursor; non-normative.** Current-state -> and gap descriptions reflect December 2025 and have been overtaken by later -> implementation. Preserve this as design rationale, but use the current VWL -> manual and live code/Vontology state for workflow planning. - -**Lifecycle**: Superseded design precursor -**Authority**: Historical only -**Version**: 1.0 -**Date**: 2025-12-21 -**JIRA**: [JVNAUTOSCI-803](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-803) -**Authors**: Von AI Assistant, Michael Witbrock - ---- - -## Executive Summary - -This document outlines the design for **LLM Workflows** — a compositional framework that transforms Von's currently ad-hoc LLM invocations into structured, observable, and reusable patterns. The key insight is that every LLM operation (chat, annotation, rumination, stored prompts) can be modelled as a workflow comprising one or more actions with explicit control flow. - -**Core Benefits**: -- **Observability**: Complete execution traces for debugging and optimisation -- **Reusability**: Common patterns (retry, validation, caching) extracted once -- **Composability**: Build complex multi-step reasoning from simpler components -- **Security**: Unified access control and audit trail -- **Performance**: Systematic identification and elimination of bottlenecks - ---- - -## Table of Contents - -1. [Background & Context](#1-background--context) -2. [Current State Analysis](#2-current-state-analysis) -3. [Design Principles](#3-design-principles) -4. [Conceptual Model](#4-conceptual-model) -5. [Vontology Schema](#5-vontology-schema) -6. [Workflow Definition Language](#6-workflow-definition-language) -7. [Control Flow Patterns](#7-control-flow-patterns) -8. [Implementation Architecture](#8-implementation-architecture) -9. [Migration Strategy](#9-migration-strategy) -10. [Integration Points](#10-integration-points) -11. [Performance Considerations](#11-performance-considerations) -12. [Security & Access Control](#12-security--access-control) -13. [Testing Strategy](#13-testing-strategy) -14. [Operational Concerns](#14-operational-concerns) -15. [Future Extensions](#15-future-extensions) -16. [Glossary](#16-glossary) - ---- - -## 1. Background & Context - -### 1.1 Current State - -Von uses large language models (LLMs) across multiple subsystems: - -- **Chat Assistant**: User dialogue with tool invocation capabilities -- **Annotation Extraction**: Identifying concepts/entities in text -- **Stored Prompts**: Vontology-backed prompt templates for specific tasks -- **Rumination**: Proactive knowledge graph enrichment -- **Tell-Von Agent**: Natural language task inference - -Each subsystem implements its own LLM invocation logic, leading to: -- Code duplication (retry logic, error handling, logging) -- Limited observability (no unified execution tracing) -- Difficult debugging (scattered LLM calls with inconsistent patterns) -- Missed optimisation opportunities (no systematic analysis of LLM usage) - -### 1.2 Existing Vontology Concept - -The Vontology already contains `#V#llm_action`: - -> "An action performed by invoking a large language model with a specific prompt and model configuration. LLM actions represent discrete computational operations that use language models for analysis, generation, or decision-making." - -**Key Instance**: `#V#detect_missing_tool_call_action` (used in chat orchestrator recovery) - -This foundational concept provides the building block for workflows. - -### 1.3 Motivation - -**Observation**: Most LLM operations are not isolated actions but **sequential or conditional processes**: - -- Chat: [build context] → [invoke LLM] → [parse tools] → [execute tools] → [format response] -- Annotation: [load prompt] → [invoke LLM] → [parse JSON] → [map types] → [rank candidates] -- Rumination: [identify gap] → [query LLM] → [validate] → [update graph] (iterate) - -Making these implicit workflows **explicit** enables: -- Systematic analysis and optimisation -- Reusable patterns across subsystems -- Automatic instrumentation and monitoring -- Declarative specification (stored in Vontology) - ---- - -## 2. Current State Analysis - -### 2.1 LLM Invocation Inventory - -#### Chat Assistant (`InternalMCPChatOrchestrator`) - -**File**: `src/backend/integrations/internal_mcp/orchestrator.py` - -**Current Flow**: -```python -def run(prompt, context, ...): - 1. Build system message (with tool instructions) - 2. Construct conversation history - 3. LOOP up to max_tool_invocations: - a. Invoke LLM - b. Extract tool calls from response - c. IF tool calls present: - - Execute via MCP gateway - - Append results to context - d. ELSE: - - Return response - 4. Handle missing tool call detection (recovery) -``` - -**Characteristics**: -- **Control Flow**: Iterative with early exit -- **Error Handling**: Retry for missing tool calls -- **Observability**: Partial (logs exist, no structured trace) -- **LOC**: ~400 lines (orchestration + helpers) - -**Workflow Opportunity**: This is already a rudimentary workflow — formalise it. - -#### Annotation Extraction Service - -**File**: `src/backend/services/annotation_extraction_service.py` - -**Current Flow**: -```python -def extract_annotations(text): - 1. Load prompt from Vontology (#V#find_concepts_in_text_prompt) - 2. Build full prompt with text - 3. Invoke LLM - 4. Parse JSON response - IF parse fails: - - Attempt regex fallback extraction - 5. Map type labels to concept IDs - 6. Score and rank candidates - 7. Return annotation spans -``` - -**Characteristics**: -- **Control Flow**: Sequential with conditional fallback -- **Error Handling**: Graceful degradation (regex fallback) -- **Caching**: Prompt cached (TTL=300s), phrase cache for candidate matching -- **LOC**: ~500 lines (extraction + helpers) - -**Workflow Opportunity**: Sequential workflow with error recovery step. - -#### Stored Prompt Execution - -**Files**: Various (referenced in `JVNAUTOSCI-797`) - -**Current Pattern**: -- Prompts stored as Vontology concepts (via `hasContent` or `hasDescription`) -- Template expansion with user/org context -- Single LLM invocation -- No inherent retry or validation logic - -**Workflow Opportunity**: Enable multi-step prompts (e.g., "analyse text, extract entities, validate against Vontology"). - -#### Rumination & Relation-Filling - -**Context**: JIRA-371 (Proactive Relation-Filling), JIRA-377 (Annotation Rumination) - -**Anticipated Pattern**: -- Identify missing relations for concept -- LOOP for each missing field: - - Query LLM for plausible value - - Validate against schema/constraints - - IF valid: update knowledge graph -- Continue until all gaps filled or max iterations reached - -**Workflow Opportunity**: Canonical iterative workflow with validation gates. - -### 2.2 Common Patterns Identified - -Across all subsystems, the following **workflow primitives** emerge: - -1. **Sequential Execution**: Step A → Step B → Step C -2. **Conditional Branching**: IF condition THEN action_X ELSE action_Y -3. **Iteration**: WHILE condition DO action (with max iterations safeguard) -4. **Error Recovery**: TRY action CATCH error THEN fallback_action -5. **Parallel Fan-Out**: PARALLEL [action_A, action_B, action_C] THEN aggregate -6. **Caching**: Retrieve cached result if available, else compute and cache - ---- - -## 3. Design Principles - -### 3.1 Core Principles - -1. **Explicit Over Implicit**: Every LLM operation should declare its workflow structure -2. **Composability**: Workflows can be composed of smaller workflows (fractal pattern) -3. **Declarative First**: Workflow definitions stored as data (Vontology), not just code -4. **Backward Compatible**: Existing code continues to work during migration -5. **Observable by Default**: Every workflow execution produces a structured trace -6. **Fail-Safe**: Failures at any step are captured and don't crash the system -7. **User-Scoped**: Workflows respect user/org permissions and context - -### 3.2 Non-Goals (Out of Scope) - -- **General Workflow Engine**: This is not a replacement for Airflow/Temporal; scope is LLM-centric -- **Visual Workflow Builder**: UI for workflow creation deferred to Phase 4 -- **Real-Time Workflow Modification**: Runtime changes require explicit API (future feature) -- **Cross-System Orchestration**: External system integrations (beyond MCP) not covered - -### 3.3 Minimal-Imposition Mutation Policy - -Workflow policy should minimise unnecessary interruption, not default to -hesitation. - -Normative implications: - -- low-risk additive Vontology writes should default-allow when the workflow has - evidence-backed inputs and no explicit user denial; -- canonical identifiers or URLs may themselves constitute sufficient permission - for low-risk additive representation or artefact-ingestion work; -- write-policy decisions should be based on risk class, not primarily on - whether the prompt happened to include verbs such as `download`, `store`, or - `create`; -- destructive or otherwise high-risk mutations should route to explicit - approval/confirmation workflow states; -- human interruption is an exception path for destructive, ambiguous, or - genuinely safety-critical cases, not the default doctrine for ordinary Von - operation. - ---- - -## 4. Conceptual Model - -### 4.1 Workflow Hierarchy - -``` -LLMWorkflow - ├─ metadata (id, name, description) - ├─ control_flow_type (sequential | conditional | iterative | parallel | dag) - ├─ steps: List[WorkflowStep] - │ ├─ step_id - │ ├─ action (reference to LLMAction or nested LLMWorkflow) - │ ├─ condition (optional predicate) - │ ├─ retry_policy - │ └─ dependencies (for DAG flows) - └─ failure_strategy (halt | continue | retry | fallback) -``` - -### 4.2 Workflow vs Action - -**LLMAction**: Atomic unit — a single LLM invocation with specific prompt/model/params - -**LLMWorkflow**: Composition of actions (or sub-workflows) with control flow logic - -**Relationship**: Every LLMWorkflow contains ≥1 LLMAction. An LLMAction can exist standalone or within a workflow. - -### 4.3 Execution Model - -``` -WorkflowEngine.execute_workflow(workflow_id, inputs, context) -> WorkflowResult - ├─ Load workflow definition from Vontology - ├─ Initialise workflow state (inputs + context) - ├─ FOR each step in workflow: - │ ├─ Evaluate step condition (if present) - │ ├─ IF condition met: - │ │ ├─ Execute step action (dispatch to executor) - │ │ ├─ Handle retries if step fails - │ │ ├─ Capture step result in trace - │ │ └─ Update workflow state - ├─ Apply control flow logic (iteration, branching, etc.) - └─ Return WorkflowResult (output + trace + metadata) -``` - ---- - -## 5. Vontology Schema - -### 5.1 Core Concepts - -#### `#V#llm_workflow` (Type) - -**Parent**: `#V#workflow` (create if doesn't exist, else `#V#action`) - -**Description**: A compositional pattern comprising one or more LLM actions with defined sequencing and control flow. - -**Attributes** (via text relations): -- **hasDescription**: Human-readable explanation of workflow purpose -- **hasName**: Workflow display name -- **hasDefinition**: Optional *narrative* process description (not machine-parsed JSON) - -**Relationships**: -- `is_a_type_of` → `#V#workflow` -- `has_instance` → Specific workflow instances (e.g., `#V#chat_assistant_workflow`) -- `hasInitialStep` → The first step in the workflow (a `#V#workflow_step` individual) -- `hasStep` → All steps in the workflow (unordered set; ordering comes from control-flow links) - -#### `#V#llm_action` (Type) — ENHANCED - -**Existing concept, enhance with additional attributes**: - -**New Attributes**: -- **hasPromptTemplate**: Reference to stored prompt concept ID (e.g., `#V#find_concepts_in_text_prompt`) -- **hasModelSelectionStrategy**: `{"fixed", "adaptive", "user_preference"}` -- **hasInputSchema** (JSON): Expected input structure -- **hasOutputSchema** (JSON): Expected LLM response structure -- **hasValidationRules** (JSON): Post-conditions to verify - -**Example Enhancement**: -```json -{ - "action_id": "#V#extract_spans_action", - "prompt_template_id": "#V#find_concepts_in_text_prompt", - "model_selection": "user_preference", - "input_schema": {"text": "string", "max_spans": "integer"}, - "output_schema": {"spans": "array[{text, start, end, type}]"}, - "validation_rules": [ - {"rule": "output.spans must be array"}, - {"rule": "each span must have text field"} - ] -} -``` - -#### `#V#workflow_step` (Type) - -**Description**: A single step within an LLM workflow - -**Attributes** (via text relations): -- **hasName**: Short label (e.g., "Invoke LLM for spans") -- **hasDescription**: Purpose of the step -- **hasTimeoutSeconds**: Optional numeric setting -- **hasMaxRetries**: Optional numeric setting -- **hasRetryPolicy**: Optional symbolic setting (e.g., `#V#exponential_backoff_retry_policy`) - -**Relationships**: -- `is_an_instance_of` → `#V#workflow_step` -- `partOfWorkflow` → Parent workflow concept -- `invokesAction` → The action to execute (usually a `#V#llm_action`, but can be any `#V#action`) -- `nextStep` → Default control-flow successor step -- `onTrueNextStep` / `onFalseNextStep` → Conditional branching successors -- `onFailureNextStep` → Fallback/recovery successor -- `hasPrecondition` → Preconditions (logical statements, see `#V#workflow_condition`) -- `hasEffect` → Effects (state updates, see `#V#workflow_effect`) -- `readsVariable` / `writesVariable` → Dataflow dependencies - -#### `#V#workflow_execution_trace` (Individual) - -**Description**: Runtime record of a specific workflow execution - -**Attributes**: -- **workflow_id**: Reference to workflow definition -- **execution_id**: UUID for this specific run -- **start_time**: ISO 8601 timestamp -- **end_time**: ISO 8601 timestamp -- **status**: `{"running", "completed", "failed", "timeout", "cancelled"}` -- **user_id**: Authenticated user who triggered workflow -- **org_id**: Organisation context -- **step_traces**: Array of step-level execution records -- **total_tokens**: Aggregate token count across all LLM calls -- **total_cost**: Estimated cost (if available) - -**Storage**: MongoDB `workflow_executions` collection (separate from main Vontology) - -**Retention**: 30-day TTL for routine executions, indefinite for failed/timeout cases - -### 5.2 Predicate Extensions - -**New Predicates**: -- `partOfWorkflow`: Links step to parent workflow -- `hasInitialStep`: Links workflow to its entry step -- `hasStep`: Links workflow to step set (for discovery) -- `nextStep`: Default control-flow edge -- `onTrueNextStep` / `onFalseNextStep`: Conditional branch edges -- `onFailureNextStep`: Error-recovery edge -- `invokesAction`: Step → action to run -- `hasPromptTemplate`: Action → stored prompt concept -- `hasPrecondition` / `hasEffect`: Step → condition/effect objects -- `readsVariable` / `writesVariable`: Step → variable nodes - ---- - -## 6. Workflow Definition Language - -### 6.1 Vontology-Native Process Representation (No JSON Definitions) - -Workflow *structure* should be represented as explicit Vontology entities and relationships, not as a JSON blob in text. - -This aligns with classical KR approaches: -- **Scripts** (Schank & Abelson): stereotyped event sequences with roles, props, and expectations. -- **Plans** (STRIPS-style): actions with **preconditions** and **effects**, enabling reasoning about state. -- **HTN**: workflows as tasks decomposed into ordered/partially ordered subtasks. -- **Situation calculus / Golog**: explicitly modelled programmes; the graph is the programme. - -In Von terms, a workflow becomes a small, queryable graph: - -- `#V#chat_assistant_workflow` (individual; instance of `#V#llm_workflow`) - - `hasInitialStep` → `#V#chat_step_build_system_message` - - `hasStep` → (all step nodes) - -- `#V#chat_step_build_system_message` (individual; instance of `#V#workflow_step`) - - `invokesAction` → `#V#compose_chat_system_message_action` - - `nextStep` → `#V#chat_step_invoke_llm` - -- `#V#chat_step_invoke_llm` - - `invokesAction` → `#V#chat_llm_generate_action` - - `nextStep` → `#V#chat_step_extract_tool_calls` - -- `#V#chat_step_extract_tool_calls` - - `invokesAction` → `#V#parse_tool_calls_action` - - `hasPrecondition` → `#V#tool_calls_present_condition` - - `onTrueNextStep` → `#V#chat_step_execute_tools` - - `onFalseNextStep` → `#V#chat_step_return_text_response` - -- `#V#chat_step_execute_tools` - - `invokesAction` → `#V#mcp_tool_loop_action` - - `nextStep` → `#V#chat_step_synthesise_response` - -- `#V#chat_step_synthesise_response` - - `invokesAction` → `#V#synthesise_final_response_action` - -The “definition” is therefore the graph itself. - -### 6.2 Conditions, State, and Bindings - -To keep the workflow graph explicit while remaining implementable, represent *conditions* and *bindings* as first-class nodes: - -- `#V#workflow_variable` (type) with individuals like `#V#var_tool_calls`, `#V#var_tool_results` - - Steps use `readsVariable` / `writesVariable` to document dataflow. - -- `#V#workflow_condition` (type) - - Store a short declarative expression in `hasContent` (e.g., "tool_calls_nonempty") or a restricted predicate form. - - Link step → condition via `hasPrecondition`. - -- `#V#workflow_effect` (type) - - Represents a state update such as “append tool result to context” or “set final_response”. - -This is deliberately closer to planning/operator representations than to a JSON DSL. - -### 6.3 Optional: Narrative Definition Text - -If `hasDefinition` is used at all, it should be a narrative process description that complements the graph (useful for humans and LLM prompting), not a machine-parsed JSON definition. - ---- - -## 7. Control Flow Patterns - -### 7.1 Sequential Workflow - -**Definition**: Steps execute in order, each step's output becomes available to subsequent steps. - -**Example** (Annotation Extraction): -``` -1. load_prompt_from_vontology -2. build_extraction_prompt -3. invoke_llm_for_spans -4. parse_json_response (with fallback) -5. map_types_to_concepts -6. rank_candidates -``` - -**Implementation**: Simple `for` loop over steps list - -### 7.2 Conditional Workflow - -**Definition**: Steps execute based on runtime condition evaluation. - -**Example** (Chat Intent Routing): -``` -1. classify_user_intent -2. IF intent == "concept_query": - 3a. fetch_concept_details - 3b. synthesise_answer_with_context - ELSE: - 3c. generate_direct_response -``` - -**Implementation**: Evaluate `condition` field before step execution - -### 7.3 Iterative Workflow - -**Definition**: Repeat a sub-workflow until condition met or max iterations reached. - -**Example** (Rumination): -``` -WHILE missing_fields AND iterations < max_iterations: - 1. identify_next_missing_field - 2. query_llm_for_value - 3. validate_response - 4. IF valid: - update_knowledge_graph -``` - -**Implementation**: `control_flow: "iterative"` with `max_iterations` parameter - -### 7.4 Parallel Workflow - -**Definition**: Execute multiple steps concurrently, then aggregate results. - -**Example** (Multi-Model Consensus): -``` -PARALLEL: - - invoke_llm_a(prompt) - - invoke_llm_b(prompt) - - invoke_llm_c(prompt) -AGGREGATE: - - majority_vote([response_a, response_b, response_c]) -``` - -**Implementation**: `asyncio.gather()` for concurrent step execution - -### 7.5 DAG Workflow (Future) - -**Definition**: Steps form a directed acyclic graph with explicit dependencies. - -**Example** (Complex Analysis): -``` - ┌─→ step_2a ─┐ -step_1 ─┤ ├─→ step_4 - └─→ step_2b ─┘ -``` - -**Implementation**: Topological sort + parallel execution where possible - ---- - -## 8. Implementation Architecture - -### 8.1 Module Structure - -``` -src/backend/workflows/ -├── __init__.py -├── engine.py # Core WorkflowEngine class -├── executors/ -│ ├── __init__.py -│ ├── llm_executor.py # LLM action execution -│ ├── mcp_tool_executor.py # MCP tool invocation -│ ├── prompt_executor.py # Prompt composition -│ └── parse_executor.py # JSON/text parsing -├── definition.py # Workflow definition parsing -├── state.py # WorkflowState and StateManager -├── trace.py # ExecutionTrace recording -└── validators.py # Schema and condition validators -``` - -### 8.2 Core Classes - -#### `LLMWorkflowEngine` - -```python -from typing import Dict, Any, Optional -from .definition import WorkflowDefinition -from .state import WorkflowState -from .trace import ExecutionTrace - -class LLMWorkflowEngine: - def __init__(self, gateway: InternalMCPGateway): - self._gateway = gateway - self._executor_registry: Dict[str, Callable] = {} - self._register_default_executors() - - def execute_workflow( - self, - workflow_id: str, - inputs: Dict[str, Any], - context: WorkflowContext, - ) -> WorkflowResult: - """Execute a workflow from its Vontology definition.""" - # 1. Load workflow definition - workflow_def = self._load_workflow_definition(workflow_id) - - # 2. Initialise state - state = WorkflowState(inputs=inputs, context=context) - trace = ExecutionTrace(workflow_id=workflow_id, user_id=context.user_id) - - # 3. Execute according to control flow - if workflow_def.control_flow == "sequential": - result = self._execute_sequential(workflow_def, state, trace) - elif workflow_def.control_flow == "conditional": - result = self._execute_conditional(workflow_def, state, trace) - elif workflow_def.control_flow == "iterative": - result = self._execute_iterative(workflow_def, state, trace) - else: - raise ValueError(f"Unsupported control flow: {workflow_def.control_flow}") - - # 4. Persist trace - self._save_trace(trace) - - return result - - def execute_step( - self, - step: WorkflowStep, - state: WorkflowState, - trace: ExecutionTrace, - ) -> StepResult: - """Execute a single workflow step.""" - # 1. Evaluate condition - if step.condition and not self._evaluate_condition(step.condition, state): - return StepResult(skipped=True) - - # 2. Resolve inputs - resolved_inputs = self._resolve_inputs(step.inputs, state) - - # 3. Dispatch to executor - executor = self._executor_registry.get(step.action_type) - if not executor: - raise ValueError(f"Unknown action type: {step.action_type}") - - # 4. Execute with retry logic - result = self._execute_with_retry( - executor, resolved_inputs, step.max_retries, step.retry_policy - ) - - # 5. Update state and trace - state.update(step.outputs, result) - trace.add_step(step.step_id, result) - - return result -``` - -#### `WorkflowDefinition` - -```python -from dataclasses import dataclass -from typing import List, Dict, Any - -@dataclass -class WorkflowStep: - step_id: str - action_id: str # Vontology concept ID - inputs: Dict[str, str] # Variable bindings (strings or references) - outputs: Dict[str, str] # Output mappings - condition_id: Optional[str] = None # Optional #V#workflow_condition - retry_policy: str = "none" - max_retries: int = 0 - timeout_seconds: int = 30 - -@dataclass -class WorkflowGraph: - """A workflow definition represented explicitly as a graph.""" - - workflow_id: str - initial_step_id: str - steps: Dict[str, WorkflowStep] - edges: Dict[str, Dict[str, str]] # step_id -> {"next"|"on_true"|"on_false"|"on_failure": step_id} - - @classmethod - def from_vontology(cls, workflow_id: str) -> 'WorkflowGraph': - """Load workflow definition from Vontology relationships. - - Expected pattern: - - workflow --hasInitialStep--> step - - workflow --hasStep--> step - - step --nextStep/onTrueNextStep/...--> step - - step --invokesAction--> action - """ - # 1) Fetch workflow node - # 2) Resolve hasInitialStep + hasStep - # 3) For each step, resolve invokesAction and control-flow successors - # 4) Return graph - pass -``` - -#### `WorkflowState` - -```python -class WorkflowState: - """Manages workflow execution state (inputs, step outputs, context).""" - - def __init__(self, inputs: Dict[str, Any], context: WorkflowContext): - self._inputs = inputs - self._context = context - self._outputs = {} # step_id -> output - self._variables = {} # state variables - - def get(self, path: str) -> Any: - """Resolve variable path (e.g., '$inputs.user_prompt').""" - # Parse path: $namespace.key.subkey - # Return value from appropriate namespace - pass - - def update(self, mappings: Dict[str, str], result: Any): - """Update state with step outputs.""" - # Apply output mappings (e.g., {"llm_response": "$.content"}) - # Extract values from result using JSONPath - pass -``` - -#### `ExecutionTrace` - -```python -@dataclass -class StepTrace: - step_id: str - start_time: datetime - end_time: datetime - status: str # "success" | "failed" | "skipped" - inputs: Dict[str, Any] - outputs: Dict[str, Any] - error: Optional[str] = None - tokens_used: Optional[int] = None - -class ExecutionTrace: - """Records workflow execution for observability.""" - - def __init__(self, workflow_id: str, user_id: str): - self.workflow_id = workflow_id - self.execution_id = str(uuid.uuid4()) - self.user_id = user_id - self.start_time = datetime.utcnow() - self.end_time: Optional[datetime] = None - self.status = "running" - self.steps: List[StepTrace] = [] - - def add_step(self, step_id: str, result: StepResult): - """Record step execution.""" - pass - - def to_dict(self) -> Dict[str, Any]: - """Serialise for storage.""" - pass -``` - -### 8.3 Executor Registry - -**Executor Interface**: -```python -class StepExecutor(ABC): - @abstractmethod - def execute(self, inputs: Dict[str, Any], context: WorkflowContext) -> Any: - """Execute the action and return result.""" - pass -``` - -**Built-in Executors**: - -1. **LLMExecutor**: Invoke LLM with prompt/model/params -2. **MCPToolExecutor**: Call internal MCP tools -3. **PromptComposerExecutor**: Build prompts from templates -4. **JSONParserExecutor**: Parse JSON with fallback handling -5. **ValidatorExecutor**: Check post-conditions/schemas - -**Registration**: -```python -def _register_default_executors(self): - self._executor_registry["llm_generate"] = LLMExecutor(self._gateway) - self._executor_registry["mcp_tool_invocation"] = MCPToolExecutor(self._gateway) - self._executor_registry["compose_prompt"] = PromptComposerExecutor() - self._executor_registry["parse_json"] = JSONParserExecutor() - self._executor_registry["validate"] = ValidatorExecutor() -``` - ---- - -## 9. Migration Strategy - -### 9.1 Phased Approach - -**Phase 1: Foundation** (No breaking changes) -- Implement `LLMWorkflowEngine` with sequential flow only -- Create Vontology schema for workflows -- Add workflow storage/retrieval endpoints -- Introduce `WorkflowAdapter` to wrap existing functions - -**Phase 2: Chat Integration** (Gradual replacement) -- Define `#V#chat_assistant_workflow` in Vontology -- Refactor `InternalMCPChatOrchestrator` to use workflow engine internally -- Preserve existing public API (`orchestrator.run()` signature unchanged) -- Run parallel validation (old vs new implementation) - -**Phase 3: Annotation & Prompts** (Feature parity + enhancements) -- Convert annotation extraction to workflow -- Enable stored prompts to declare multi-step workflows -- Add conditional and iterative control flow - -**Phase 4: New Features** (Leverage workflow capabilities) -- Parallel execution -- Workflow optimisation (caching, batching) -- UI for workflow visualisation -- Runtime workflow customisation - -### 9.2 Adapter Pattern (Non-Breaking Wrapper) - -```python -class WorkflowAdapter: - """Wraps existing function as a single-step workflow.""" - - @staticmethod - def from_function( - func: Callable, - workflow_id: str, - input_schema: Dict = None, - output_schema: Dict = None, - ) -> LLMWorkflow: - """Convert existing function to workflow definition.""" - # Generate workflow JSON with single step - # Step action = wrapped function call - # Store in Vontology - pass - -# Example usage -annotation_workflow = WorkflowAdapter.from_function( - extract_annotations, - workflow_id="#V#annotation_extraction_workflow", - input_schema={"text": "string"}, - output_schema={"spans": "array"}, -) -``` - -### 9.3 Dual-Path Validation - -During migration, run both old and new implementations: - -```python -def extract_annotations_with_validation(text: str) -> List[Dict]: - # Old implementation - old_result = _extract_annotations_legacy(text) - - # New workflow implementation - workflow_result = workflow_engine.execute_workflow( - "#V#annotation_extraction_workflow", - inputs={"text": text}, - context=get_current_context(), - ) - new_result = workflow_result.outputs["spans"] - - # Compare results - if not _results_match(old_result, new_result): - logger.warning(f"Workflow divergence detected: {diff}") - # Emit metric for monitoring - - # Return old result (safe fallback during transition) - return old_result -``` - ---- - -## 10. Integration Points - -### 10.1 Chat Assistant - -**Before** (`orchestrator.py`): -```python -class InternalMCPChatOrchestrator: - def run(self, prompt, context, ...) -> OrchestratorResult: - # 400 lines of procedural orchestration logic - pass -``` - -**After** (workflow-based): -```python -class InternalMCPChatOrchestrator: - def __init__(self, gateway, workflow_engine=None): - self._gateway = gateway - self._workflow_engine = workflow_engine or LLMWorkflowEngine(gateway) - self._chat_workflow_id = "#V#chat_assistant_workflow" - - def run(self, prompt, context, ...) -> OrchestratorResult: - workflow_result = self._workflow_engine.execute_workflow( - self._chat_workflow_id, - inputs={"prompt": prompt, "context": context}, - context=WorkflowContext( - user_id=get_authenticated_user_id(), - org_id=get_current_organisation_id(), - ), - ) - - # Map workflow result to OrchestratorResult - return OrchestratorResult( - response_text=workflow_result.outputs["final_response"], - tool_invocations=workflow_result.trace.tool_calls, - ... - ) -``` - -### 10.2 Annotation Extraction - -**Before**: -```python -def extract_annotations(text: str) -> List[Dict]: - prompt = load_prompt_from_vontology() - llm_response = llm_client.generate(prompt + text) - spans = parse_json_or_fallback(llm_response) - return rank_candidates(spans) -``` - -**After**: -```python -def extract_annotations(text: str) -> List[Dict]: - result = workflow_engine.execute_workflow( - "#V#annotation_extraction_workflow", - inputs={"text": text}, - context=get_current_context(), - ) - return result.outputs["ranked_spans"] -``` - -### 10.3 Stored Prompts - -**Enhancement**: Prompts can now declare workflows instead of just templates. - -**Vontology Representation**: -```json -{ - "concept_id": "#V#my_custom_prompt", - "hasDescription": "Analyse research paper and extract key findings", - "hasWorkflowDefinition": { - "control_flow": "sequential", - "steps": [ - {"action_type": "llm_generate", "prompt": "Summarise abstract"}, - {"action_type": "llm_generate", "prompt": "Extract methodology"}, - {"action_type": "llm_generate", "prompt": "List key findings"}, - {"action_type": "validate", "rule": "findings.length >= 3"} - ] - } -} -``` - ---- - -## 11. Performance Considerations - -### 11.1 Overhead Analysis - -**Added Latency Sources**: -1. Workflow definition loading from Vontology (~10-20ms, cacheable) -2. State management and variable resolution (~5-10ms per step) -3. Trace recording (~2-5ms per step) -4. Condition evaluation (~1-2ms per step) - -**Total Overhead**: ~20-40ms for typical workflow (5 steps) - -**Target**: Keep overhead <5% of total workflow time (dominated by LLM calls which are 500ms-5s) - -### 11.2 Optimisation Strategies - -#### Caching - -1. **Workflow Definition Cache**: TTL=300s (same as current prompt cache) -2. **Step Result Cache**: Cache idempotent step outputs (e.g., prompt composition) -3. **LLM Response Cache**: Deduplicate identical prompts within workflow - -```python -class WorkflowCache: - def __init__(self, ttl_seconds: int = 300): - self._definitions: Dict[str, WorkflowDefinition] = {} - self._step_results: Dict[str, Any] = {} - self._llm_responses: Dict[str, str] = {} - - def get_cached_step_result(self, cache_key: str) -> Optional[Any]: - # Check if step result can be reused - pass -``` - -#### Parallel Execution - -For workflows with independent steps, execute concurrently: - -```python -async def _execute_parallel( - self, - workflow_def: WorkflowDefinition, - state: WorkflowState, - trace: ExecutionTrace, -) -> WorkflowResult: - # Identify independent steps (no cross-dependencies) - # Execute concurrently using asyncio.gather() - pass -``` - -#### Lazy Loading - -Load workflow definitions only when needed (not on server startup). - -### 11.3 Performance Benchmarks - -**Baseline** (current direct LLM calls): -- Annotation extraction: ~800ms (LLM call dominates) -- Chat response: ~1.2s (single turn, no tools) -- Chat with tools: ~3.5s (2 tool invocations) - -**Target** (workflow-based): -- Annotation extraction: <850ms (+50ms overhead acceptable) -- Chat response: <1.25s (+50ms overhead) -- Chat with tools: <3.6s (+100ms overhead for more complex workflows) - ---- - -## 12. Security & Access Control - -### 12.1 Workflow-Level Permissions - -**Principle**: Workflows inherit access control from authenticated user context. - -**Implementation**: -```python -class WorkflowContext: - user_id: str # Authenticated user (#V#person_id) - org_id: str # Current organisation (#V#org_id) - permissions: Set[str] # User's role-based permissions - - def can_execute_workflow(self, workflow_id: str) -> bool: - # Check if user has permission to run workflow - # E.g., some workflows may be admin-only - pass -``` - -### 12.2 Step-Level Access Control - -**Scenario**: Workflow contains MCP tool calls that require specific permissions. - -**Solution**: Workflow engine delegates permission checks to underlying systems (e.g., MCP gateway already enforces tool permissions). - -### 12.3 Trace Sanitisation - -**Concern**: Execution traces may contain sensitive data (user inputs, LLM responses). - -**Mitigation**: -1. **Field-Level Redaction**: Automatically redact sensitive fields (e.g., API keys, passwords) -2. **User Consent**: Require opt-in for detailed trace collection -3. **Retention Policy**: Auto-delete traces after 30 days (configurable) - -```python -class TraceSanitiser: - SENSITIVE_PATTERNS = [ - r"api[_-]?key", - r"password", - r"token", - ] - - def sanitise_trace(self, trace: ExecutionTrace) -> ExecutionTrace: - # Redact sensitive fields from step inputs/outputs - pass -``` - -### 12.4 Workflow Integrity - -**Threat**: Malicious workflow definition could execute arbitrary code. - -**Defences**: -1. **No Dynamic Code Execution**: Workflow definitions are data (JSON), not code -2. **Restricted Executor Registry**: Only pre-approved executors can be invoked -3. **Condition Evaluation Sandbox**: Use safe expression evaluator (not Python `eval`) -4. **Schema Validation**: All workflow definitions validated against schema before execution - ---- - -## 13. Testing Strategy - -### 13.1 Unit Tests - -**Workflow Engine**: -- Workflow definition parsing -- Variable resolution -- Condition evaluation -- Retry logic -- Error handling - -**Executors**: -- LLM executor (mocked LLM calls) -- MCP tool executor (mocked gateway) -- Prompt composer -- JSON parser (including fallback) - -**State Management**: -- Variable binding -- Output mapping -- State updates - -### 13.2 Integration Tests - -**End-to-End Workflows**: -- Chat assistant workflow (full conversation) -- Annotation extraction workflow -- Multi-step stored prompt execution - -**Cross-Component**: -- Workflow engine + MCP gateway -- Workflow engine + Vontology (load/store definitions) - -### 13.3 Performance Tests - -**Benchmarks**: -- Workflow overhead vs direct calls -- Parallel execution speedup -- Cache hit rates -- Trace storage impact - -**Load Testing**: -- Concurrent workflow executions -- Maximum workflow steps before degradation -- Memory usage under load - -### 13.4 Security Tests - -**Access Control**: -- Unauthorised workflow execution blocked -- Step-level permission enforcement -- Trace access restricted to workflow owner - -**Injection Attacks**: -- Malicious condition expressions rejected -- Invalid workflow definitions rejected -- Trace sanitisation verified - ---- - -## 14. Operational Concerns - -### 14.1 Monitoring & Observability - -**Metrics** (exposed via `/diag` endpoint): -- Workflow execution count (by workflow_id) -- Average execution time (by workflow_id) -- Failure rate (by workflow_id, by step_id) -- Token usage (aggregate across all workflows) -- Cache hit rates (definition cache, step result cache) - -**Dashboards**: -- Real-time workflow execution view -- Historical performance trends -- Error rate alerts - -**Logging**: -``` -[workflow_engine] Executing workflow #V#chat_assistant_workflow (execution_id=abc-123) -[workflow_engine] Step build_system_message completed (15ms) -[workflow_engine] Step invoke_llm completed (1203ms, tokens=450) -[workflow_engine] Workflow completed successfully (total=1.5s) -``` - -### 14.2 Error Handling & Debugging - -**Workflow Failure Modes**: -1. **Step Timeout**: Individual step exceeds timeout -2. **Step Error**: Executor raises exception -3. **Validation Failure**: Output doesn't match schema -4. **Workflow Timeout**: Overall workflow exceeds timeout - -**Recovery Strategies** (configured via `failure_strategy`): -- **halt**: Stop workflow immediately, return partial results -- **continue**: Skip failed step, proceed to next -- **retry**: Retry failed step (up to max_retries) -- **fallback**: Execute alternate step (if defined) - -**Debugging Tools**: -- Execution trace viewer (UI showing step-by-step execution) -- Step replay (re-execute single step with same inputs) -- Workflow diff (compare execution traces to identify regressions) - -### 14.3 Deployment & Rollback - -**Feature Flags**: -- `VON_WORKFLOWS_ENABLED`: Global enable/disable -- `VON_WORKFLOW__ENABLED`: Per-workflow feature flag - -**Rollback Plan**: -1. Disable workflow execution via feature flag -2. Fall back to legacy implementation (maintained in parallel during Phase 2) -3. Investigate failure, fix workflow definition -4. Re-enable once validated - ---- - -## 15. Future Extensions - -### 15.1 Workflow Marketplace - -**Vision**: Users can share and discover reusable workflows. - -**Features**: -- Workflow templates for common tasks -- Community ratings and reviews -- Import/export workflows between instances - -### 15.2 Visual Workflow Builder - -**Vision**: Drag-and-drop UI for workflow creation. - -**Features**: -- Node-based workflow editor (similar to n8n, Flowise) -- Real-time validation -- Step previews (execute single step to test) - -### 15.3 Adaptive Workflows - -**Vision**: Workflows that learn from execution history to optimise themselves. - -**Examples**: -- Adjust retry parameters based on success rate -- Skip unnecessary steps based on input patterns -- Switch models based on performance/cost trade-offs - -### 15.4 Cross-System Workflows - -**Vision**: Workflows that orchestrate external systems (not just LLMs). - -**Examples**: -- Workflow that queries database → invokes LLM → updates CRM -- Integration with Zapier, IFTTT for external automation - ---- - -## 16. Glossary - -**LLM Action**: Atomic unit — a single LLM invocation with specific prompt/model/params. - -**LLM Workflow**: Composition of actions (or sub-workflows) with control flow logic. - -**Workflow Step**: Single action within a workflow, potentially with condition and retry logic. - -**Workflow Definition**: Declarative specification of workflow structure (stored in Vontology). - -**Workflow Execution**: Runtime instance of a workflow being executed. - -**Execution Trace**: Complete record of a workflow execution (inputs, outputs, timing, errors). - -**Workflow Context**: User/org/permissions associated with workflow execution. - -**Workflow State**: Runtime variables and step outputs during execution. - -**Executor**: Component that executes a specific action type (e.g., LLM call, tool invocation). - -**Control Flow**: Logic determining which steps execute and in what order (sequential, conditional, iterative, etc.). - -**Failure Strategy**: How workflow handles step failures (halt, continue, retry, fallback). - ---- - -## Revision History - -| Version | Date | Changes | Author | -|---------|------------|----------------------------------|------------------| -| 1.0 | 2025-12-21 | Initial design document | Von AI Assistant | - ---- - -**End of Document** diff --git a/docs/engineering/mcp_reliability_analysis.md b/docs/engineering/mcp_reliability_analysis.md deleted file mode 100644 index ad747cf67..000000000 --- a/docs/engineering/mcp_reliability_analysis.md +++ /dev/null @@ -1,206 +0,0 @@ -# Analysis: Why MCP Service Wasn't Used & How to Fix It - -> **Document status: Historical incident reflection; evidence date not recorded.** -> It explains one mistaken implementation approach and may retain a useful MCP -> invariant, but its tool inventory and proposed fixes are not current MCP -> documentation. Re-discover the live MCP surface and use current operational -> guidance before acting. - -## What I Did (Wrong Approach) -I performed critical Vontology operations by writing manual Python scripts: -1. Used `text_value_service.upsert_text_for_concept()` to add detector prompt text -2. Used `text_value_service.get_texts_for_concept()` to query text relations -3. Used `concept_service.get_concept_by_concept_id()` to fetch concept details -4. Used `text_value_service.get_texts_for_concept()` to verify relationships - -**Problem**: These are low-level service calls that bypass the MCP service layer. - ---- - -## What I Should Have Done (MCP Service Approach) -The MCP service already has **all required tools** documented in `vontology_mcp.json` and implemented in `mcp_stdio_server.py`: - -### 1. Add Detector Prompt Text -``` -Tool: upsert_text_relation -Parameters: - - concept_id: "#V#missing_tool_call_detection_prompt" - - predicate: "hasContent" - - text: "You are a classifier that detects..." - - language: "en" -``` - -### 2. Query Text Relations -``` -Tool: get_text_relations -Parameters: - - concept_id: "#V#missing_tool_call_detection_prompt" - - predicate: "hasContent" (optional filter) -``` - -### 3. Get Full Concept Details -``` -Tool: fetch_concept -Parameters: - - concept_id: "#V#detect_missing_tool_call_action" - - include_relations_arg1: true - - include_text_relations_arg1: true -``` - ---- - -## Root Cause Analysis: Why I Didn't Use MCP - -### 1. **Availability Uncertainty** -- I wasn't confident the MCP service had text relation support -- I saw `add_names` working but didn't check if generic `upsert_text_relation` existed -- I defaulted to manual Python scripts instead of exploring MCP first - -### 2. **Tool Discovery Gap** -- The manifest (`vontology_mcp.json`) exists but isn't surfaced in my agent context -- The stdio server implementation (`mcp_stdio_server.py`) is 2000+ lines—hard to scan -- No "quick reference" of available text relation tools visible to agents - -### 3. **Implementation Verification Missing** -- I didn't systematically check: "Is this tool implemented?" before writing Python -- I wrote scripts first, verified they work, then retrospectively realized MCP existed - ---- - -## How to Fix the MCP Service for Future Reliability - -### Phase 1: Documentation & Discoverability ✅ (Already Exists But Not Surfaced) -**Current State:** -- ✅ Complete manifest in `vontology_mcp.json` -- ✅ Full implementations in `mcp_stdio_server.py` -- ✅ Handlers registered in `_TOOL_HANDLERS` dictionary - -**Issue:** -- Not visible in agent context or documentation - -**Fix:** -- Create `docs/engineering/mcp_text_relations_quick_reference.md` listing all text relation tools -- Include this in agent context/AGENTS.md -- Provide examples for each tool - -### Phase 2: HTTP Server Parity (Audit & Augment) -**Current:** Only stdio MCP server has full implementations - -**Action Required:** -1. Audit `mcp_server.py` (Flask HTTP) to see what endpoints exist -2. Add missing text relation endpoints if not present: - - `POST /upsert_text_relation` - - `GET /get_text_relations` - - `POST /update_text_relation` - - `DELETE /delete_text_relation` -3. Ensure identical behavior between HTTP and stdio interfaces - -**Why:** Users might call HTTP endpoints directly; need feature parity - -### Phase 3: Service Reliability Improvements -Add robustness features to prevent future failures: - -**3a. Validation & Error Handling** -- Verify `subject_concept_id` parameter works (current code uses `subject_concept_id` in service but accepts `concept_id` in handlers—ensure consistency) -- Add parameter validation before calling services -- Return structured error responses with hints - -**3b. Integration Tests** -- Add test: "Can upsert text with hasContent predicate and retrieve it" -- Add test: "Text retrieval filters by predicate correctly" -- Add test: "fetch_concept returns text relations in correct format" - -**3c. Fallback Graceful Degradation** -- If a text relation operation fails, log it clearly with suggestion to check: - - Concept exists - - Predicate name is valid - - No database connectivity issues - -### Phase 4: Agent Context & Guidance -**In AGENTS.md or instructions, add:** -```markdown -## Using Vontology MCP for Text Relations - -### When You Need to: -1. **Add/Update text content on a concept** → Use `upsert_text_relation` - - Example: Adding a prompt, description, or instructions - - Predicates: hasContent, hasDescription, hasNote, custom - -2. **Query what text is stored** → Use `get_text_relations` - - Example: Finding all names, descriptions, or custom text - -3. **Get full concept with relationships** → Use `fetch_concept` - - Example: Understanding concept structure before modifications - -### Available Text Relation Predicates: -- **hasName** – Alternate names, translations, abbreviations (use `add_names` helper) -- **hasContent** – Main text content, prompts, instructions -- **hasDescription** – Human-readable descriptions -- **hasNote** – Implementation notes, commentary -- Custom predicates – Any application-specific text relations - -### Never Use Python Scripts For: -- ❌ Adding text relations (use `upsert_text_relation`) -- ❌ Querying text relations (use `get_text_relations`) -- ❌ Fetching concepts (use `fetch_concept`) -- ❌ Managing relationships (use `add_relationship`, `remove_relationship`) - -### Always Use MCP Because: -✅ Tools are discoverable by LLM -✅ Operations are logged and traceable -✅ Error handling is consistent -✅ Can be extended without code changes -✅ Supports future UI frontends -``` - ---- - -## Summary: Changes Required to Fix MCP Service - -| Component | Issue | Fix | Priority | -|-----------|-------|-----|----------| -| Documentation | Text relation tools not visible to agents | Add quick reference to AGENTS.md | HIGH | -| HTTP Server | May lack text relation endpoints | Audit `mcp_server.py`, add missing endpoints | HIGH | -| Consistency | Parameter names (`concept_id` vs `subject_concept_id`) | Standardize across handlers | MEDIUM | -| Testing | No integration tests for text relations | Add E2E tests for upsert/query cycle | MEDIUM | -| Error Messages | Generic exception handling | Add hints for common failure modes | LOW | - ---- - -## Lesson Learned -**When a capability already exists in the MCP service but you don't know about it, the responsibility is on the system (documentation + discoverability), not the agent.** - -Next time: -1. Check `vontology_mcp.json` before writing Python scripts -2. Search codebase for handlers in `mcp_stdio_server.py` -3. Ask explicitly: "What Vontology operations can I do via MCP?" -4. If uncertain, request tool availability explicitly - ---- - -## DevOps note: MCP flakiness in VS Code Insiders (2026-01-10) - -### Symptom -- MCP tool visibility was inconsistent: `list_mcp_resources` only showed MongoDB even when other servers were configured. -- Atlassian MCP felt flaky; no obvious start errors in normal Output logs. - -### What we did -- Merged workspace and user MCP configs so they match: - - `C:\\Users\\witbr\\Documents\\Programming\\Strong-AI-Lab\\Von\\.vscode\\mcp.json` - - `C:\\Users\\witbr\\AppData\\Roaming\\Code - Insiders\\User\\mcp.json` -- Switched stdio servers to absolute paths for reliability: - - `pdm.exe` and `uv.exe` explicit paths - - `cwd` and arXiv storage paths set to `C:\\Users\\witbr\\Documents\\Programming\\Strong-AI-Lab\\Von` -- Verified tool-level access (not resources): - - Vontology: `mcp__vontology__get_context` - - VonRAG: `mcp__vonrag__rag_list_collections` - - Atlassian: `mcp__atlassian__getAccessibleAtlassianResources` - -### Notes from DevTools console -- Repeated warnings about user-level MCP config overwriting workspace config are expected. -- MCP registry fetch from `api.github.com/copilot/mcp_registry` returned 404; this did not block local stdio tools. -- No MCP spawn errors surfaced in the console; MCP resources list still only showed MongoDB, which is expected for tool-only servers. - -### Next time -- If Atlassian MCP misbehaves, follow the restart steps in `AGENTS.md` (MCP server restart, window reload, sign-out/in). -- Prefer tool calls to validate server health, not `list_mcp_resources`. diff --git a/docs/engineering/multi_agent_coordination_design.md b/docs/engineering/multi_agent_coordination_design.md deleted file mode 100644 index 7a27a08f6..000000000 --- a/docs/engineering/multi_agent_coordination_design.md +++ /dev/null @@ -1,51 +0,0 @@ -# Multi-Agent Coordination Design - -**Status**: Proposed design guidance -**Date**: 2026-04-04 - -## 1. Purpose - -This document outlines the architectural requirements for extending the Von Workflow Language (VWL) and the Orchestrator to support mature multi-agent coordination. The goal is to move beyond monolithic agent execution and enable a robust ecosystem of specialised agents that can dynamically delegate tasks, negotiate capabilities, and share context. - -## 2. Architectural Requirements - -### 2.1 Agent-to-Agent Delegation - -The Orchestrator must support explicit delegation from a supervisor agent to specialised sub-agents. - -- **Explicit Handoffs**: Delegation must be a first-class operation in VWL, allowing an agent to yield execution to another agent with a specific sub-task and expected output schema. -- **Hierarchical Supervision**: Supervisor agents must be able to monitor the progress of delegated tasks and intervene if a sub-agent fails or hallucinates. -- **Turn Execution Records (TERs)**: Every delegated execution must produce a standard Turn Execution Record, ensuring that sub-agent actions are fully observable by the supervisor and the broader system. - -### 2.2 Capability Negotiation - -Agents must be able to discover and negotiate capabilities dynamically. - -- **Capability Registries**: The system must maintain a registry of available agents and their documented capabilities, accessible via Vontology or a dedicated MCP tool. -- **Dynamic Routing**: The Orchestrator should support dynamic routing of tasks based on the required capabilities, cost constraints, and available agent profiles. -- **Fallback Mechanisms**: If a specialised agent is unavailable or lacks the required capability, the system must gracefully fall back to a more general agent or escalate to a human supervisor. - -### 2.3 Shared Episodic Memory Contexts - -To prevent context fragmentation, delegated agents must operate with a shared understanding of the episodic context. - -- **Context Propagation**: When delegating a task, the supervisor must pass relevant slices of the shared turn context, episodic memory, and necessary Vontology pointers to the sub-agent. -- **Result Synthesis**: Upon completion of a delegated task, the sub-agent's output and relevant memory updates must be synthesised back into the supervisor's main episodic context. -- **Lineage Tracking**: The Orchestrator must track the lineage of context across agent boundaries, ensuring that memory updates can be traced back to the specific agent that generated them. - -## 3. Extension of VWL and Orchestrator - -### 3.1 VWL Extensions - -- Introduce new VWL constructs for `Delegate`, `WaitFor`, and `Synthesise`. -- Define standard schemas for agent capability manifests. - -### 3.2 Orchestrator Enhancements - -- Extend the Orchestrator's event loop to handle asynchronous sub-agent executions. -- Implement robust telemetry for cross-agent communication to aid in debugging and policy learning. -- Ensure that the Orchestrator can enforce access controls and namespace isolation between different agents operating on behalf of the same user. - -## 4. Conclusion - -By maturing multi-agent coordination, Von will be able to tackle increasingly complex tasks through the collaborative effort of specialised, observable, and accountable agents, moving closer to its vision as a scalable neuro-symbolic assistant. diff --git a/docs/engineering/phase3_orchestrator_integration_summary.md b/docs/engineering/phase3_orchestrator_integration_summary.md deleted file mode 100644 index 2c449768d..000000000 --- a/docs/engineering/phase3_orchestrator_integration_summary.md +++ /dev/null @@ -1,159 +0,0 @@ -# Phase 3: Orchestrator Integration - Completion Summary - -> **Document status: Historical JVNAUTOSCI-799 implementation record from -> December 2025.** “Complete”, replacement, validation, and “ready for -> deployment” claims below are phase-local historical claims, not evidence of -> current architecture, reliability, or production readiness. Revalidate the -> present orchestrator path and tests before relying on them. - -**JIRA Issue**: JVNAUTOSCI-799 -**Status**: ✅ Complete -**Date**: December 2025 - -## Overview - -Phase 3 successfully integrated structured tool calling into the Internal MCP Chat Orchestrator, replacing the legacy JSON-in-text parsing approach with native structured tool calling from Phases 0-2. - -## Implementation Highlights - -### 1. LLMInterface Bridge - -Added bridge layer to existing `LLMInterface` classes: - -**`src/backend/languagemodels/llm_interface.py`**: -- `generate_with_tools()`: Delegates to `structured_tool_calling` module -- `_should_use_structured_calling()`: Feature flag check (VON_INTERNAL_MCP_STRUCTURED_TOOL_CALLING) -- `_get_structured_client_config()`: Provider-specific configuration - - **OpenAIClient**: Returns LLMClientConfig with api_key, model - - **OllamaClient**: Returns LLMClientConfig with base_url, model - - **GeminiClient**: Returns LLMClientConfig with api_key, model - -### 2. Orchestrator Integration - -**`src/backend/integrations/internal_mcp/orchestrator.py`**: - -#### New Methods: -- `_convert_mcp_tools_to_structured_definitions()`: Converts MCP catalog → ToolDefinition list -- `_mcp_schema_to_json_schema()`: Converts MCP Schema format → JSON Schema format -- `_python_type_to_json_schema_type()`: Maps Python types → JSON Schema type strings - -#### Modified run() Method: -1. **Dual Path Strategy**: - - Checks if `generate_with_tools()` available and feature flag enabled - - If yes: Use structured calling path - - If no: Fall back to legacy `_extract_tool_calls()` path - -2. **Structured Calling Flow**: - ```python - tool_definitions = self._convert_mcp_tools_to_structured_definitions() - llm_response = llm_client.generate_with_tools( - prompt=prompt, - available_tools=tool_definitions, - context=augmented_context, - model=model, - ) - ``` - -3. **Tool Call Extraction**: - - Structured path: Extract from `llm_response.tool_calls` - - Legacy path: Parse from text response JSON - -4. **Safety Constraints Preserved**: - - ✅ Namespace injection (non-Gmail tools) - - ✅ Gmail profile injection - - ✅ Tool whitelist validation - - ✅ Single-call contract enforcement - -5. **call_id Execution Tracing** (JVNAUTOSCI-803): - - Extracted from `ToolCall.call_id` in structured path - - Preserved in invocation records - - Logged for workflow correlation - -6. **Exception Handling**: - - Structured calling failures automatically fall back to legacy path - - Logged with warning for diagnostics - -### 3. Schema Conversion - -**MCP Schema Format** (required/optional dicts): -```python -{ - "required": {"query": str, "limit": int}, - "optional": {"offset": int}, -} -``` - -**JSON Schema Format** (properties + required array): -```python -{ - "type": "object", - "properties": { - "query": {"type": "string"}, - "limit": {"type": "integer"}, - "offset": {"type": "integer"}, - }, - "required": ["query", "limit"], -} -``` - -**Type Mapping**: -- `str` → `"string"` -- `int` → `"integer"` -- `float` → `"number"` -- `bool` → `"boolean"` -- `list` → `"array"` -- `dict` → `"object"` -- Handles union types (tuples): Takes first non-None type - -### 4. Comprehensive Testing - -**`tests/backend/test_orchestrator_structured_calling.py`** - 9 tests, all passing: - -1. ✅ `test_structured_calling_path_used_when_available`: Verifies structured path is used -2. ✅ `test_legacy_fallback_when_structured_disabled`: Feature flag disables structured calling -3. ✅ `test_legacy_fallback_when_structured_unavailable`: Client without structured support works -4. ✅ `test_call_id_tracing_in_structured_path`: call_id preserved in invocations (JVNAUTOSCI-803) -5. ✅ `test_namespace_injection_preserved`: Safety constraints work in structured path -6. ✅ `test_mcp_schema_to_json_schema_conversion`: Schema converter works correctly -7. ✅ `test_tool_definitions_conversion`: MCP catalog → ToolDefinition conversion -8. ✅ `test_structured_calling_with_no_tool_response`: Handles responses without tool calls -9. ✅ `test_structured_calling_exception_fallback`: Exceptions fall back gracefully - -## Feature Flag - -**Environment Variable**: `VON_INTERNAL_MCP_STRUCTURED_TOOL_CALLING` -- **Default**: `"1"` (enabled) -- **Disable**: Set to `"0"` -- **Checked by**: `LLMInterface._should_use_structured_calling()` - -## Benefits - -1. **Native Tool Calling**: LLMs use native tool calling APIs instead of JSON-in-text hacks -2. **Improved Reliability**: Structured formats reduce parsing errors -3. **Better Tracing**: call_id enables end-to-end workflow correlation -4. **Backward Compatibility**: Legacy path preserved, safe rollout via feature flag -5. **Exception Resilience**: Automatic fallback on structured calling failures - -## Files Modified - -- `src/backend/languagemodels/llm_interface.py` (extensive additions) -- `src/backend/integrations/internal_mcp/orchestrator.py` (dual-path run() method) -- `tests/backend/test_orchestrator_structured_calling.py` (new test suite) - -## Next Steps (Future Work) - -1. **Performance Metrics**: Compare structured vs legacy latency/accuracy -2. **Provider Coverage**: Ensure all three providers (OpenAI, Gemini, Ollama) tested in production -3. **Remove Legacy Path**: After stable deployment, deprecate `_extract_tool_calls()` -4. **Documentation**: Update user-facing docs with structured calling benefits - -## Related Issues - -- **JVNAUTOSCI-698**: Detect JSON action output (LLM failure mode) - Legacy path handles this -- **JVNAUTOSCI-699**: Support chained tool calls - Both paths support this -- **JVNAUTOSCI-803**: Workflow execution tracing - call_id enables this - ---- -**Implementation Date**: December 2025 -**Author**: AI Agent (GitHub Copilot) -**Status**: Ready for deployment diff --git a/docs/engineering/proactive_clarification_and_role_learning.md b/docs/engineering/proactive_clarification_and_role_learning.md index 5de9fba5e..0e37412ad 100644 --- a/docs/engineering/proactive_clarification_and_role_learning.md +++ b/docs/engineering/proactive_clarification_and_role_learning.md @@ -282,7 +282,7 @@ answer → produce the useful work → observe correction/outcome → revise the account → use it on a later independent task. Test a changed role-holder or non-applicable organisation too. Transfer useful practice, not a private alias, source-specific fact or permission. Use the existing -[represented-advice proposal](represented_advice_design.md) if retaining a +[represented-advice proposal](https://github.com/Strong-AI-Lab/Von/blob/5b2f8b976069935da73b82bca433fb3cefffcd3d/docs/engineering/represented_advice_design.md) if retaining a general lesson is justified; do not add an automatic advice layer here. ## 6. Continuity and execution contract diff --git a/docs/engineering/recent_architecture_progress_2026-04-20.md b/docs/engineering/recent_architecture_progress_2026-04-20.md deleted file mode 100644 index bcc0731bb..000000000 --- a/docs/engineering/recent_architecture_progress_2026-04-20.md +++ /dev/null @@ -1,285 +0,0 @@ -# Recent Architecture Progress — Updated April 21, 2026 - -## Purpose - -This note records the state of the `JVNAUTOSCI-1913` / `JVNAUTOSCI-1116` -anti-hack and de-bloating programme after the April 21, 2026 landings through -`JVNAUTOSCI-1972`. - -For the broader current-code review of where Von still falls short of the -agentic architecture described in `Von_for_AgenticAI.md`, and for the latest -priority ordering after the 23 April 2026 repeat review, see -`docs/engineering/agentic_architecture_status_2026-04-23.md`. - -This note remains useful as a historical anti-drift snapshot through the April -21 tranche. Its code-size measurements and "recommended next steps" should be -read as historical April 21 guidance, not as the live post-`1989` -programme-wide ordering. - -It replaces an earlier draft that had become materially stale. In particular: - -- `JVNAUTOSCI-1929` is now `Done` -- `JVNAUTOSCI-1930` is now `Done` -- `JVNAUTOSCI-1953` is now `Done` -- `JVNAUTOSCI-1954` is now `Done` -- `JVNAUTOSCI-1955` is now `Done` -- `JVNAUTOSCI-1956` has now removed the remaining turn-record response-surface fallback from the primary critic path -- the structured tool-family keyword/prefix heuristic slab is no longer an open violation - -The aim of this note is not to provide a full historical changelog. It is to state, as clearly as possible, what has actually been removed, what architectural risks remain, and what the next worthwhile structural steps are. - -## Design Intent - -Von's architectural doctrine is unchanged: - -- durable decision policy belongs in authority surfaces such as Vontology, workflows, prompts, and KB assertions -- Python should provide reusable support surfaces: execution, validation, telemetry, persistence, integration, and safety -- when a behaviour can be represented in workflow/prompt/KB/Vontology rather than encoded as a code-side semantic rule, that represented form should be preferred - -The work over April 10–21 materially moved the codebase in that direction. It did not finish the programme. - -## What Actually Landed - -### 1. Orchestrator lexical and semantic-routing cleanup - -The orchestrator no longer owns the worst of the old semantic routing policy. - -Completed work in this area: - -- `JVNAUTOSCI-1914` removed orchestrator-side lexical routing and guided-retrieval steering -- `JVNAUTOSCI-1943`, `1944`, `1948`, `1949`, `1945`, and `1951` repaired the live grounded tool path, ontology follow-up, evidence shaping, and answer-path degradation without reintroducing prompt-keyword steering -- `JVNAUTOSCI-1929` made the turn expected-outcome contract a first-class boundary object -- `JVNAUTOSCI-1930` made dispatch preflight explicitly contract-aware for multi-surface evidence requirements -- `JVNAUTOSCI-1953` removed the remaining structured tool-family prompt-keyword / prefix heuristic slab and replaced it with authority-backed family resolution derived from represented tool metadata, explicit required tools, and recent actual tool use -- `JVNAUTOSCI-1954` added a first-class predicate-incidence support surface so explicit predicate-relative turns no longer have to rely on broad relation-hit paging before narrowing to the relevant predicate -- `JVNAUTOSCI-1955` stopped grounded false-negative tool-path turns from being silently recorded as `completed_verified`; those turns now surface as `false_success` / `false_completion_claim` when retrieved evidence contradicts a low-information answer -- `JVNAUTOSCI-1956` removed the remaining turn-record response-surface fallback so `required_evidence_answer_consistency_blocker` is now produced only by the authority-backed postcondition critic path - -This is a real architectural shift. Structured tool availability is no longer widened or narrowed by Python heuristics such as matching `jira`, `task`, `epic`, or tool-name prefixes against the prompt. - -### 2. Workflow discovery and routing substrate cleanup - -The workflow-discovery path is no longer based on the old BM25 / stopword / lexical capability-index doctrine. - -Completed work in this area: - -- `JVNAUTOSCI-1781` recorded the architectural decision that workflow discovery should remain a dedicated retrieval plane rather than collapse into generic concept search -- `JVNAUTOSCI-1915` replaced the BM25 capability index with a RAG-backed workflow retrieval surface over represented workflow artefacts - -The old code-side lexical workflow-capability doctrine is gone. Python now provides the retrieval surface rather than being the durable source of recommendation policy. - -### 3. Support-surface decomposition and executor cleanup - -Several large mixed-responsibility runtime slabs were decomposed into clearer support surfaces. - -Completed work in this area: - -- `JVNAUTOSCI-1821` extracted part of the custom-workflow dispatch cluster from the worst region of the orchestrator -- `JVNAUTOSCI-1824` decomposed the base workflow executor into clearer support phases -- `JVNAUTOSCI-1950` converged the durable executor on those shared runtime support surfaces so base/durable behaviour does not drift -- `JVNAUTOSCI-1827` turned catalogue assembly into explicit capability-family builder support surfaces rather than one large inline registration slab - -This did not solve the orchestrator monolith, but it did remove multiple points where runtime behaviour was being patched inline inside oversized functions. - -### 4. Semantic-heuristic removal from domain and authoring services - -Several support services that previously contained English-language or regex-heavy interpretation policy were moved toward authority-backed interpretation. - -Completed work in this area: - -- `JVNAUTOSCI-1916` moved file-copy diagram interpretation away from regex NER / stopword / confidence heuristics into an authority-backed interpretation surface -- `JVNAUTOSCI-1952` replaced person/company/meeting file-copy representation heuristics with one shared authority-backed entity-representation interpretation surface -- `JVNAUTOSCI-1917` removed authoring semantics from English lexical fallbacks in workflow creation and annotation extraction -- `JVNAUTOSCI-1918` moved write-tool request-evidence inference out of prompt-text semantic policy and into a Vontology-backed evidence surface -- `JVNAUTOSCI-1931` hardened workflow publication so transient per-turn specifics are structurally kept out of reusable authority artefacts - -### 5. Guardrails, anti-drift, and test quality - -The work was not only code movement. Mechanical guardrails were added to stop these regressions from quietly returning. - -Completed work in this area: - -- `JVNAUTOSCI-1919` extended the workflow-purity report with anti-drift contracts for key support surfaces -- `JVNAUTOSCI-1947` repaired stale tests that were still pinning removed heuristic behaviour or bypassing the authority-backed paths they were supposed to exercise -- `JVNAUTOSCI-1953` added purity-report anti-drift coverage for the retired structured tool-family heuristic symbols - -### 6. Operational and tooling cleanup - -Completed work in this area: - -- `JVNAUTOSCI-1946` migrated the Atlassian MCP path from deprecated HTTP+SSE transport to Streamable HTTP - -That is not central to `1913`, but it removed an operational liability and aligned the checked-in control surface with the real supported transport. - -## What Changed in Practical Terms - -The most important difference from ten days earlier is not a line-count win. It is that several major code-side semantic policy surfaces are now gone. - -Before this round: - -- workflow discovery still depended on lexical / BM25 logic -- orchestrator tool-family exposure still depended on prompt-keyword and prefix heuristics -- file-copy and workflow-authoring services still carried English-language interpretation hacks -- turn-contract behaviour still leaked across boundaries as case-specific plumbing - -Now: - -- workflow discovery is retrieval-backed and authority-aligned -- structured tool-family resolution is authority-backed rather than prompt-keyword-driven -- explicit predicate-relative ontology turns now have a first-class incidence lookup rather than broad unfiltered relation paging as the primary narrowing step -- several domain interpretation surfaces have been moved behind prompt/Vontology-backed services -- turn expected-outcome contract handling is explicit and stage-boundary-visible -- turn-contract tool requirements now travel as explicit `required_tools` contract state rather than being re-inferred from prose inside the orchestrator -- grounded false-negative answer paths no longer silently count as verified success in turn-execution correctness reporting -- anti-drift reporting now guards multiple high-risk support surfaces - -That is substantial progress even though the remaining monoliths are still too large. - -## Current Structural Reality - -Some honest current code-size facts: - -- `src/backend/integrations/internal_mcp/orchestrator.py` is still `34,608` lines -- `src/backend/integrations/internal_mcp/catalogue.py` is still `27,466` lines -- `src/backend/server/routes/von_routes.py` is still `13,348` lines - -So the codebase is materially cleaner in responsibility placement, but not yet dramatically smaller in its largest files. - -## Current Jira State Relevant to This Programme - -As of April 21, 2026: - -| Task | Status | Note | -|------|--------|------| -| `JVNAUTOSCI-1116` | `To Do` | Epic remains open even though many child cleanups are complete | -| `JVNAUTOSCI-1913` | `In Progress` | Still the right umbrella; April 21 follow-through through `JVNAUTOSCI-1972` removed contract-text tool inference and the final turn-contract prose backfill from the live path | -| `JVNAUTOSCI-1929` | `Done` | Turn contract is now a first-class boundary object | -| `JVNAUTOSCI-1930` | `Done` | Dispatch preflight is now contract-aware | -| `JVNAUTOSCI-1953` | `Done` | Structured tool-family keyword/prefix heuristics removed | -| `JVNAUTOSCI-1954` | `Done` | Predicate-incidence lookup landed for entity/type incidence narrowing | -| `JVNAUTOSCI-1955` | `Done` | Grounded false-negative turns no longer count as verified success | -| `JVNAUTOSCI-1970` | `Done` | Remaining write-denial heuristic and hard-coded write-risk classes removed from the live write path | -| `JVNAUTOSCI-1121` | `Done` | `/von/generate` lifecycle and response/debug support surfaces extracted from `von_routes.py` | -| `JVNAUTOSCI-1120` | `To Do` | Frontend monolith cleanup remains open | -| `JVNAUTOSCI-1825` | `Done` | `create_flask_app(...)` is now a materially smaller composition layer over extracted startup/admin support surfaces | -| `JVNAUTOSCI-1971` | `Done` | Residual buttonify heuristic/prose-recovery helpers and stale compatibility surfaces removed; live path is structured-output-only | -| `JVNAUTOSCI-1972` | `Done` | Residual turn-contract prose parsing removed; structured contract state is now the only live Python authority for `required_tools` | -| `JVNAUTOSCI-768` | `To Do` | Vontology concept structure for tool heuristics still not landed | -| `JVNAUTOSCI-770` | `To Do` | Rule-loader integration task still not landed | -| `JVNAUTOSCI-1813` | `To Do` | Deferred structural exception-barrier task, not in the core `1116` line | - -## What Still Remains - -The remaining work is now more concentrated. The biggest live problems are no longer the same ones this note identified earlier. - -### 1. Buttonify’s residual heuristic helper slab has now been removed - -**Location:** [buttonify_service.py](../../src/backend/services/buttonify_service.py) - -The live route/workflow path was already structured-output-first through `#V#chat_buttonify_workflow` and `buttonify_options_json`, but `buttonify_service.py` still carried residual heuristic/prose-recovery helpers and stale tests/docs that obscured the intended authority boundary. - -That residual slab is now gone. Buttonify options are validated from structured output only, and malformed/non-JSON output fails closed instead of being reverse-engineered from prose. - -### 2. The orchestrator remains the biggest structural risk - -**Location:** [orchestrator.py](../../src/backend/integrations/internal_mcp/orchestrator.py) - -Even after the recent cleanups, `InternalMCPChatOrchestrator` is still an oversized integration monolith. The most egregious surviving semantic slab from the previous draft, the structured tool-family prompt-keyword heuristic block, is gone, and `1954`/`1955` repaired two important grounded-answer failure families. But the class still combines too many concerns: - -- workflow dispatch -- tool planning and tool execution orchestration -- turn reporting and tracing -- renderer and response shaping paths -- multiple support-policy seams that are still easier to patch inline than to extract - -This is the main structural reason new hacks tend to accumulate here first. - -### 3. Backend route layers improved materially, but the monolith is still too large - -**Location:** [von_routes.py](../../src/backend/server/routes/von_routes.py) - -`JVNAUTOSCI-1121` is now done, and it made a real structural dent: the `/von/generate` durable-turn lifecycle, chat-history persistence, and success/error payload shaping helpers were extracted into `generate_route_support.py`, taking several hundred lines out of `von_routes.py` and giving later route work clearer seams. But the backend route layer is still large and mixed in responsibility, so this remains an ongoing structural concern rather than a solved problem. - -### 4. Flask app-factory sprawl is no longer the main route-layer blocker - -**Location:** [utils_flask.py](../../src/backend/server/utils_flask.py) - -`JVNAUTOSCI-1825` is now done. The old `create_flask_app(...)` slab has been cut down materially by extracting request timing/configuration, startup bootstrap, chat-history admin context resolution, diagnostics helpers, and prewarm wiring into explicit support surfaces. The file is still large, but the structural risk has shifted away from one huge all-purpose app-factory function and toward the remaining large route modules themselves. - -### 5. Catalogue size is still large, but the risk has shifted - -**Location:** [catalogue.py](../../src/backend/integrations/internal_mcp/catalogue.py) - -`JVNAUTOSCI-1827` already removed the worst inline assembly slab by turning `build_default_catalogue()` into a composition layer over builder helpers. The file is still very large, but the remaining risk is now more about size, merge pressure, and further per-family decomposition than about one egregious remaining semantic-hack block. - -That means it is still a structural concern, but it is not the highest-priority anti-policy-drift problem anymore. - -## Revised Summary of Remaining Violations - -| Category | Current state | Best current handling | -|----------|---------------|-----------------------| -| Structured tool-family prompt heuristics | Removed by `1953` | Keep anti-drift guard in place | -| Low-information answer-surface heuristic | Removed by `1956` | Keep the prompt-backed critic authoritative | -| Turn contract stage-boundary plumbing | Fixed by `1929` | Done | -| Multi-surface dispatch verification | Fixed by `1930` | Done | -| Write confirmation / destructive regex | Removed by `1970` | Keep represented authority surfaces canonical | -| Hardcoded write-risk tool classes | Removed by `1970` | Keep represented runtime profile authoritative | -| Residual buttonify heuristic helper slab | Removed by `1971` | Keep structured-output validation fail-closed | -| Orchestrator monolith | Still present | Ongoing structural extraction | -| Backend route monolith | Improved by `1121`, still present | Continue route/app-factory decomposition | -| Flask app-factory sprawl | Reduced materially by `1825` | No longer the sharpest open route-layer problem | - -## Recommended Next Steps - -This section is now historical for overall programme sequencing. - -After the later same-day landings through `JVNAUTOSCI-1994`, `1993`, and the -repeat review in `1989`, the live next-task ordering moved to the broader -memory/evaluator/self-improvement umbrellas recorded in -`docs/engineering/agentic_architecture_status_2026-04-23.md`. - -The still-relevant `JVNAUTOSCI-1913`-local guidance from this note is narrower: - -1. continue orchestrator extraction where it removes real mixed-responsibility - pressure or hidden policy drift rather than opening noise tickets based only - on file size; -2. continue route-layer de-bloating where it produces clearer reusable support - seams; and -3. keep anti-drift tests and purity checks aligned with the newer - authority-backed architecture. - -Do not use this note's April 21 sequencing as the current programme-wide next -step list. - -## Implementation Note — April 21, 2026 - -A first support-surface landing has now been made toward item 2 above. - -`turn_execution_record_service.py` now accepts an authority-supplied `required_evidence_answer_consistency_blocker` via `critic_verdict` and records that source explicitly when the completion gate is driven by the authored critic. - -`run_turn_execution_critic(...)` now also emits a structured `critic_verdict` on the live action path so that this authority seam is carried through workflow outputs, diagnostics, and turn-record finalisation rather than existing only as a record-builder input contract. - -At that stage, the gate was no longer structurally forced to depend on a Python response-surface check. That mattered because it let a workflow- or LLM-authored critic take ownership of the decision without another turn-record schema change. - -### Follow-on Landing — April 21, 2026 - -That authoritative producer has now been introduced on the canonical conversation-turn path under `JVNAUTOSCI-1956`. - -- `#V#conversation_turn_execution_workflow` and `#V#tool_calling_workflow` no longer call `turn_execution.critic` directly for the postcondition-critic stage. They now invoke `#V#kb_mutation_postcondition_critic_workflow` as an explicit subworkflow. -- `#V#kb_mutation_postcondition_critic_workflow` is now a real multi-step authority surface: it first builds a bounded turn-execution evidence bundle, then runs `llm.action` against the new authoritative prompt concept `#V#prompt_turn_execution_postcondition_critic`, and only then finalises the canonical turn-execution record. -- `turn_execution_runtime_support.py` now supports an evidence-building mode that suppresses the synthetic default `critic_verdict`, so the prompt-authored critic can become the primary producer instead of merely decorating a Python-owned judgement. -- `turn_execution_record_service.py` no longer derives `required_evidence_answer_consistency_blocker` from response-surface heuristics. If the authoritative critic path cannot produce a blocker, the turn record now leaves that field empty rather than reviving the retired Python detector. - -### Follow-on Landing — April 21, 2026 - -A bounded `JVNAUTOSCI-1913` follow-on landing in `JVNAUTOSCI-1972` has now -removed the remaining orchestrator contract-text tool-inference seam. - -- `TurnExpectedOutcomeContract` now carries explicit `required_tools` state and preserves it across stage-boundary rendering, dispatch preflight, and selector/tool/recovery context projection without reparsing support prose back into Python decision state. -- the expected-outcome inference prompt now emits `required_tools` as part of the authoritative contract payload instead of forcing later Python recovery from prose wording -- the old orchestrator helpers that inferred KB / relation / web retrieval surfaces by scanning contract text have been deleted - -That means the next meaningful anti-egregious-path work is no longer more -contract-prose parsing cleanup. It is continued orchestrator extraction where it -reduces mixed-responsibility pressure, followed by the represented tool-metadata -and rule-loader follow-through tasks. diff --git a/docs/engineering/reliability_ratchet_articles_and_cases.md b/docs/engineering/reliability_ratchet_articles_and_cases.md deleted file mode 100644 index 4b8ff96ff..000000000 --- a/docs/engineering/reliability_ratchet_articles_and_cases.md +++ /dev/null @@ -1,666 +0,0 @@ -# The Reliability Ratchet: source articles and tracked cases - -- **Kind:** Advisory source copy and evidence log -- **Lifecycle:** Active -- **Authority:** The reproduced articles are advisory; tracked cases are - evidence records. Neither overrides current user direction, `AGENTS.md`, live - authority, or current implementation evidence. -- **Authority scope:** Reliability-ratchet analysis in Von engineering work -- **Owner:** Von maintainers -- **Last reviewed:** 20 August 2026 -- **Review trigger:** A new tracked case, a material source correction, or new - evidence that changes a recorded diagnosis or status -- **State or evidence as of:** 20 August 2026 -- **Open questions:** Which recorded diagnoses remain supported, need narrowing, - or should be reclassified after outcome-level validation? - -This file preserves the two public ratchet articles as source material and -records concrete Von cases in which the pathology is observed. It deliberately -does not convert the articles into a mandatory checklist, schema, approval -stage, or second repository constitution. Their useful role is as a revisable -diagnostic lens. The governing engineering invariants remain in -[`AGENTS.md`](../../AGENTS.md). - -## Source provenance - -The article bodies below are faithful Markdown transcriptions of the public -rendered WordPress bodies downloaded on 18 August 2026. Heading levels were -adjusted only to fit this containing document. Wording, spelling, punctuation, -emphasis, lists, and links—including source errors—were otherwise preserved. - -| Article | Published | Public source | WordPress body | Rendered-body SHA-256 | -|---|---:|---|---|---| -| *The Reliability Ratchet* | 1 August 2026 | [Canonical article](https://michaelwitbrock.com/2026/08/01/the-reliability-ratchet/) | [Post 3](https://public-api.wordpress.com/wp/v2/sites/michaelwitbrock.com/posts/3) | `f05f7f9722885a4a12f559812d70e598997d33ebff36d73b17156f4be29b6475` | -| *One turn of the ratchet* | 18 August 2026 | [Canonical article](https://michaelwitbrock.com/2026/08/18/one-turn-of-the-ratchet/) | [Post 50](https://public-api.wordpress.com/wp/v2/sites/michaelwitbrock.com/posts/50) | `8e9e1b62161a6d62923bc3548e683f12196025c2c94a0f558bc473f45a4ae84e` | - -WordPress identifies Michael Witbrock as the publishing author of both posts. -The first article contains its own explicit contributor line; the second -acknowledges its collaboration in the opening paragraph. - -## Article 1 — The Reliability Ratchet - -*How AI-coded systems turn fixes into fossils* - -*Michael Witbrock and Codex (GPT-5.6 Sol Ultra) written in the context of a frustrating programming session.* - -The most dangerous code written by AI is not obviously bad code. - -Bad code is visible. It fails tests, throws exceptions, performs poorly or can’t be maintained. The more dangerous output is a clean, documented, reviewed and thoroughly tested patch that prevents one observed failure by quietly eliminating valid behaviours, alternative implementations and future routes to improvement. - -A failure is reported. A coding agent analyses it, constructs a plausible causal story and repairs the nearest relevant code. It adds a validator, contract, state, retry, adapter or fallback. It writes tests showing that the failure is gone and updates the documentation to explain the new mechanism. - -The ticket closes. - -Has the system improved? Or has one uncertain diagnosis just become permanent architecture? - -We call this the reliability ratchet: a process in which every visible failure adds machinery, while almost nothing creates comparable pressure to remove it. The system becomes more locally defensible and more globally constrained. - -That is non-progress—not inactivity, but energetic work that makes future work harder and may remove all reasonable paths to achieving the intended system.. - -### An old pathology at a new speed - -Human software teams have always accumulated workarounds, stale abstractions and tests that preserve yesterday’s implementation. Coding agents did not invent technical debt. They change its economics. - -A human team might take days to turn an incident into a diagnosis, an implementation, a test suite and a design explanation. A coding agent can produce the entire package in minutes. Because all the pieces are coherent, the result looks unusually rigorous. - -But the test is often not independent evidence for the diagnosis. It is the diagnosis repeated in executable form. The documentation repeats it in prose. The contract embeds it in an interface. The implementation makes it operational. - -Four artefacts agree because one conjecture generated all four. - -This creates a distinctive form of epistemic debt. The code may be locally excellent, the tests comprehensive and the documentation clear. What has not been established is that the explanation of the failure was true—or that the repair preserves the wider range of behaviour the system ought to support. - -AI coding lets a weak causal theory acquire institutional authority with unprecedented speed. - -### How the ratchet works - -The cycle is usually: - -**Observation → conjecture → mechanism → regression test → inherited law.** - -A coding agent receives a bounded task: make this failure stop happening. It inherits the existing architecture, tests and repository instructions and thinks they are largely intentional – that they have resulted from a coherent plan. Questioning that architecture appears out of scope; modifying the nearest code appears responsible. - -So one race condition becomes a mandatory global ordering. One timeout produces a universal retry layer. One malformed payload makes an optional field compulsory. One integration failure creates a permanent fallback. One ambiguous request becomes an intent classifier. - -Any of those changes might be right; the mistake is moving from a single observation to a deterministic structure without either first establishing the failure class or weighing the effects against the overall purpose of the software.. - -The next coding agent encounters the new mechanism and its tests as authority. Removing them looks dangerous. Working around them looks safe. When a neighbouring case fails, another mechanism is added outside the first. - -Nothing in this process is locally irrational. That is precisely the problem. - -### A guardrail that recorded its own defeat - -This ratchet is exceptionally difficult to avoid. In the large system maintained substantially with coding agents (including both of the authors) that provided the initial context for this analysis, a test was introduced to restrain the growth of a central coordination module. Its accepted baseline was 46,106 lines. A genuine deletion brought the module down to 45,355. - -Then development resumed. As new features and repairs enlarged the file, the baseline was repeatedly refreshed until 49,717 lines counted as compliant. The test even printed the command required to approve the larger number. - -Line count itself was not the diagnosis; large files can sometimes be justified. The revealing fact was that a supposed anti-growth mechanism repeatedly certified growth. The guardrail had not changed the development pressure. It had converted accumulation into a notarised exception. - -The repository possessed the right principle and an automated check. The executable environment still rewarded closing the current task. - -### A conjecture is not an invariant - -A trace rarely determines its own explanation. The observed failure may have been caused by the nearest branch, or by missing context, a false abstraction, an obsolete stage, an upstream race, an incorrect requirement—or the fact that the entire mechanism no longer earns its place. - -The coding agent’s diagnosis is a hypothesis. It should remain revisable until neighbouring evidence supports it. - -Software is uncomfortable with hypotheses. It prefers types, states and assertions. That preference is valuable when the underlying property is genuinely invariant: - -- One user must not access another user’s private data. - -- A payment must not be charged twice. - -- A destructive action requires proper authority. - -- A committed record must remain internally consistent. - -Those requirements can be stated independently of the incident that revealed them. Other decisions are contingent: which component should handle a request, which recovery should be tried first, whether one intermediate state must exist, and whether today’s workaround will remain useful after the surrounding system changes. And perhaps most importantly, whether today’s workaround will interfere with a current or future vital system function that just happened not to have been exercised during the test that drove the fix. - -Hardening a contingent diagnosis into an invariant does not remove uncertainty. It launders uncertainty into structure. - -In AI systems, user intention makes this especially obvious. Intention is normally a working hypothesis assembled from language, context, available actions and the consequences of being wrong. It should be revised as evidence arrives. Turning it immediately into an immutable classification or expected-outcome contract may make the system more explicit while making it less intelligent. - -But the same mistake occurs throughout ordinary software. A causal theory is not made true by giving it a schema. - -### Tests as an accidental constitution - -Tests are indispensable. But tests do not acquire authority merely by existing. - -A test may protect an externally meaningful requirement: data integrity, authorisation, idempotency, API compatibility or an observable user outcome. Or it may instead protect an accident: the exact helper called, stage traversed, internal label assigned, cache consulted, message emitted or sequence followed by yesterday’s successful repair. - -Those are not equivalent. - -When implementation-specific tests are treated as enduring requirements, the test suite becomes an accidental constitution. Future agents are told, in executable form, that correctness means reproducing an approved internal history, not satisfying the system’s purpose. - -This is particularly powerful for coding agents. Tests are concrete, local and machine-verifiable. Lost opportunities are none of those things. Persistent instructions to coding agents are not successful in causing them to view tests as revisable, or deletable, - -The valid input wrongly rejected by a test leaves no stack trace. The simpler architecture never attempted produces no failing test. The recovery strategy eliminated by a validator generates no incident. The cost of making future changes harder appears days or, in projects slowed by the capability ratchet, months later, distributed across unrelated tickets. - -Every visible failure can produce another permanent artefact. Missing alternatives leave almost no evidence that they ever existed. The ratchet is asymmetric. - -### Why capable agents can conceal a bad architecture - -A sufficiently capable coding agent can navigate an astonishingly complicated codebase. It can discover obscure contracts, satisfy brittle tests, thread another field through twelve layers and produce a plausible explanation for all of it. - -That capability can conceal architectural decline. - -The fact that an agent can successfully modify the maze does not show that the maze is justified. It may show only that the agent is good enough to compensate for it. - -As coding models improve, this problem may worsen before it improves. Stronger agents can keep increasingly elaborate systems operational, delaying the moment when humans are forced to confront the structure itself. Meanwhile, the product may become slower, less adaptable and more expensive even as the rate of closed tickets rises. - -As we’ll explore in a future post, these abilities also have recently been enabling capable agents to find composite cyber-vulnerabilities in complex systems; this does not mean that they will be as capable at planning and implementing function-preserving mitigations. - -### Why more rules will not save us - -Most repositories and software engineering guides already contain excellent principles: prefer simple designs, avoid special cases, preserve compatibility, do not overfit, add abstractions only when justified, and remove obsolete machinery. - -Then the development environment rewards the opposite. - -Issue trackers fragment systemic failures into local units of closure. Continuous integration rewards preserving the visible suite. Review templates reward explaining additions more readily than demonstrating that an existing mechanism should disappear. Repository instructions accumulate the lessons of incidents until agents spend increasing amounts of context interpreting accumulated fear. - -When prose and executable authority conflict, executable authority wins. - -Adding a mandatory fourteen-field “repair theory contract” would merely reproduce the pathology at the process level. The answer is not more paperwork describing simplicity. The selection pressure must change. - -### Changing what counts as progress - -A better development environment would make several distinctions operational. - -#### Separate observation, inference and invariant - -Record what happened separately from the proposed explanation. Ask what neighbouring observations would contradict that explanation. Do not let the first plausible diagnosis arrive pre-packaged as the permanent contract. - -#### Diagnose a failure class before prescribing a route - -Test nearby inputs, alternative sequences, different timings and plausible competing causes. The aim is not exhaustive proof, but enough variation to discover whether the proposed repair generalises or merely recognises the incident. - -#### Test outcomes while allowing implementation freedom - -Where the internal path is not itself a requirement, test the observable result and prohibited effects. Accept multiple valid implementations. Characterisation tests for legacy machinery may still be useful, but they should not silently govern its replacement. - -#### Compare additions with the simplest credible alternative - -Before adding a new layer, compare it with removing or bypassing an existing one. A no-new-mechanism baseline is often more revealing than a more elaborate competing design. The baseline will not always win; its purpose is to make complexity earn its place. - -#### Use evidence the implementation agent has not already absorbed - -Neighbouring and held-out cases can expose repairs that merely encode visible examples. Where appropriate, use an independent reviewer or protected evaluation set that initially reports outcomes and failure classes rather than handing the implementer every case. A “hidden” suite readable by the coding agent is not a holdout. - -#### Give deletion equal status with addition - -A task should be closable by proving that a stage, adapter, fallback, rule or test is unnecessary. Temporary mechanisms should have removal conditions. When a replacement succeeds, delete what it replaced. Permanent dual paths are how the next tower begins. - -Whether this development environment can be achieved within current dev platforms and with current coding agents remains to be seen. - -### A shared theory of purpose - -Large systems do need shared intent; they do not need an encyclopaedic contract describing every permitted implementation. - -They need a compact, revisable account of the jobs the system exists to perform, the outcomes that matter, the states that are genuinely unacceptable, the constraints that are hard, the mechanisms that are merely current choices, and the evidence that would justify changing or removing them. - -Without that shared theory, the test suite becomes the de facto product definition and the issue tracker becomes the architecture. The purpose of such guidance is not to dictate every repair. It is to give coding agents enough context and authority to recognise when the right action is subtraction. - -### The standard of belief - -A coherent explanation is not enough. Coding agents are extremely good at coherent explanations. - -For any substantial new mechanism, the harder questions are: - -- What class of failures does this prevent? - -- What valid behaviours or future designs might it exclude? - -- What evidence would cause us to remove it? - -If there is no serious answer to the last question, the mechanism is unlikely to remain an engineering decision. It will become a fossil. - -### What to look for in your own repository - -The reliability ratchet is a hypothesis other teams can test. Look for: - -- Local regression rates falling while change latency and validation burden rise. - -- Baselines that are repeatedly waived or refreshed upward. - -- Tests that prescribe internal routes rather than observable outcomes. - -- Temporary fallbacks with no removal conditions. - -- Agents that spend more effort satisfying accumulated structure than solving the underlying problem. - -None of these observations alone proves that a system is over-engineered. Together, they suggest that local reliability may be purchased by quietly narrowing the future. - -Coding agents are capable of extraordinary work. The answer is neither to constrain them less indiscriminately nor to trust them without any hard boundaries. - -It is to constrain the right things, and to preserve enough room for a better explanation, an unanticipated solution, or the discovery that yesterday’s successful fix should no longer exist. - -**Intelligence needs boundaries. It also needs room.** - -## Article 2 — One turn of the ratchet - -*Using the risk of a reliability ratchet as an engineering constraint* - -After publishing [*The Reliability Ratchet*](https://michaelwitbrock.com/2026/08/01/the-reliability-ratchet/), we (GPT 5.6 Sol Ultra Codex and I) used its argument, verbatim, as context for another substantial piece of engineering work. What followed suggested a practical role for architectural writing that the original post had not quite articulated. - -It was useful in a way worth distinguishing from a checklist or rule. - -It did not prescribe a solution. It changed the burden of proof. New mechanisms had to outperform a code change that would perform adequately while simplifying the code. Existing tests could be questioned when they protected an internal code choice rather than a broader outcome. This use of the context affected the process; work was reconsidered as new evidence appeared. Several coherent, well-tested fixes were rejected and redesigned because they would have made the local case tidier by creating a broader restriction. - -This suggests a practical role for architectural writing: a compact description of a known pathology can supply useful counter-pressure at the moment when the repository, ticket and test suite all press towards closure using a local fix. It gives a coding agent the vocabulary—and the permission—to notice when its own proposed repair reproduces the disease it is meant to cure. - -The limitation is crucial and points to the usefulness of such advisory writing in modern AI-assisted software architecture and construction. If such an essay is converted into a compulsory checklist, schema or approval stage, it may itself become another tooth in the ratchet, driving the code into a state that fixes all previous bugs, but prevents future evoluton towards intented function. Such advice, in Its most useful form is a revisable lens: What independent invariant is being protected? What simpler alternative was compared? What existing machinery can now disappear? What evidence would make us change course? - -A principle earns its place when it changes a decision. It overreaches when it insists on becoming machinery. - -*Aug 18th 2026, 9:40 AM, Ljubljana* - -## Tracked cases - -These are dated, revisable evidence records, not automatic release gates. -A case should distinguish what was observed from the causal interpretation, -name the independent outcome or invariant worth preserving, and state what -evidence could change its diagnosis or status. The prose structure may vary -with the evidence; it is not a compulsory schema. - -### RR-001 — A valid authority invariant fossilised into a broken workflow-route requirement - -- **Observed:** 18 August 2026; scope widened 19 August 2026 -- **Status:** Open — diagnosis established and since broadened from a - workflow-route requirement to a delegation-transport gap seen on three - unrelated routes; no runtime repair was made by this documentation change -- **Delivery tracking:** [JVNAUTOSCI-2649](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-2649) - owned the repair for the durable workflow route and is Done. It deliberately - scoped to that route: its ship criteria require that sessionless and internal - MCP effects remain denied. The remaining routes recorded in the 19 August 2026 - widening are owned by - [JVNAUTOSCI-2653](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-2653), - which holds the open decision on whether an authorised operator repair should - have any route at all. Repair alternatives, delivery decisions, acceptance - evidence, and implementation status belong to those tickets, not here. -- **User outcome:** Represent a scholarly article and its metadata in the - authenticated user's private Vontology context -- **Observed impact:** The represented workflow was launched twice and both - instances failed at their first generic ontology mutation. Direct fallback - effects left the overall outcome partial and indeterminate, while the final - report described a workflow-instance read-back as though it concerned the - article target. -- **Evidence boundary:** Failure capsule generated at - `2026-08-18T14:10:06.247605Z` for request - `845f7501-8a13-417d-a7c8-48e1a85441fe`, produced by commit - `9c0803cb0b332b0c2d81be8130aa926c27b82813` -- **Workflow:** `#V#scholarly_article_metadata_representation_workflow` -- **Failed instances:** `d74d0e99-8cc5-4509-a010-dec519133bba` - (evidence `ev_YvGwmSM5K3UTOFgQWpjGGApN`) and - `15352a1c-1c64-4f11-974c-c664a515f047` - (evidence `ev_2opEFvdKubCsUwjTH4t9rO3Z`) -- **Canonical scope reported by the capsule:** `user` - -#### What was observed - -Both workflow instances failed with: - -```text -ontology_agent_delegation_required -An executing agent needs a server-issued delegation bound to this exact ontology effect. -``` - -The read-back conclusively established that the two workflow instances were in -the terminal `failed` state. It did not establish that the scholarly-article -target had been read back exactly. In the same turn, direct fallback effects -included an indeterminate concept creation and two handler-reported successful -text upserts, so the requested representation could not honestly be called -complete. - -#### The independent invariant worth preserving - -The security requirement is real: untrusted model output, a represented -workflow, or spoofable client identity must not enlarge an authenticated -actor's ontology authority. Organisation-wide or global publication, -authority-changing effects, destructive shared mutations, and cross-user access -need a server-enforced boundary. Exact, short-lived delegation is one useful -way to bind the grantor, executing agent, audience, tool, effect, workflow or -turn, intended target and delta, and expiry. - -The pathology is therefore not “security checks are bad” or “delegation should -be removed”. It is that one particular transport mechanism became an inherited -route requirement without a complete, function-preserving route for the -authorised workflow. - -#### The ratchet sequence in this case - -| Ratchet stage | This case | -|---|---| -| **Observation** | Agent-mediated ontology effects presented a real confused-deputy and identity-spoofing risk, especially for shared publication and authority changes. | -| **Conjecture** | Every agent-labelled ontology mutation must arrive with an exact server-issued delegation before the actor's direct authority may even be considered. | -| **Mechanism** | The publication authority service unconditionally denies an agent-labelled effect with no delegation. | -| **Regression test** | Negative tests prove that a tokenless call is denied; the positive workflow-propagation test manually injects a delegation-shaped value rather than exercising normal production issuance. | -| **Inherited law** | The durable workflow executor supplies no delegation, but its generic ontology actions are still required to present one. A valid private additive workflow therefore fails by construction. | - -The distinction between the independent invariant and its current mechanism is -critical. The earlier authority work had good reason to prevent ambient or -client-supplied identity from becoming semantic-administrator authority. -Available evidence does not justify claiming that the original security -diagnosis was false. What it does establish is that the implementation was -allowed to count as complete without preserving a normal authorised workflow -outcome. - -#### Exact causal path - -1. Direct adaptive ontology calls automatically issue and bind a same-turn exact - delegation after their method and arguments are known in - [`adaptive_turn_service.py`](../../src/backend/services/adaptive_turn_service.py). -2. The represented capability's outer method is `workflow_execute`, which is - not itself classified as an ontology mutation, so that issuance path is - skipped. -3. The durable executor constructs its environment without an - `ontology_delegation_id` in - [`durable_executor.py`](../../src/backend/workflows/durable/durable_executor.py). -4. The first generic `create_concepts` step in the - [paper workflow seed](../../src/backend/workflows/repo_seed_bundles/paper_representation_workflow_seed_bundle.json) - labels the effect as workflow-agent execution and propagates the missing - value through - [`workflow_mcp_tool_actions.py`](../../src/backend/workflows/workflow_mcp_tool_actions.py). -5. The shared - [publication authority service](../../src/backend/services/ontology_publication_authority_service.py) - denies the effect before considering whether the authenticated user has - direct authority for the ordinary private additive write. -6. The positive regression test in - [`test_ontology_authority_tier3_regressions.py`](../../tests/backend/test_ontology_authority_tier3_regressions.py) - checks propagation only by manually supplying `"server-issued-only"`; it - does not prove that a production workflow can obtain a valid grant. -7. Custom scholarly-workflow handlers in - [`paper_representation_workflow.py`](../../src/backend/workflows/durable/paper_representation_workflow.py) - call lower-level mutation services directly. Semantically equivalent effects - can therefore bypass the boundary that blocks generic actions. - -Each local component can explain its behaviour: the authoriser denies a missing -credential, the gateway records agent provenance, the workflow propagates the -field it received, and the regression test confirms the selected rule. Their -agreement is not independent evidence that the end-to-end design works. It is -the same implementation conjecture repeated through code and tests while the -user outcome disappears—precisely the “accidental constitution” described in -the first article. - -#### Reporting amplified the pathology - -The outcome renderer then made the mechanism sound more authoritative than its -evidence allowed: - -- `changed: true` meant that a durable workflow-instance record was created, - not that article metadata changed; -- `outcome_resolved: true` meant the workflow failure was conclusively known, - not that the task succeeded; -- the generic target projection reused the workflow `instance_id`, producing - identical `instance` and `target` identifiers; and -- “the current target was read back exactly” referred to rereading the failed - workflow instance, not the scholarly article. - -This is another small ratchet effect: operational bookkeeping acquired -domain-sounding language and obscured the missing user outcome. - -#### Why this qualifies as the pathology - -The guard is locally defensible but globally constraining. It closes a selected -unsafe route, yet in this private additive case it also erases a safe, -authorised route that the server already has enough trusted context to bind. -No further human approval is semantically required: the direct path creates the -same kind of delegation automatically. Requiring the token without providing -an issuer adds user-visible failure and architectural burden without adding a -corresponding decision. - -The enforcement is also route-dependent: governed generic actions fail while -custom handlers can bypass it. That is evidence that the current mechanism is -not yet identical with the intended security invariant. - -#### Scope widened — 19 August 2026 - -The original framing above, "workflow-route requirement", is too narrow. The -same missing issuer was met on two further, unrelated routes while attempting an -explicitly authorised ontology repair tracked by -[JVNAUTOSCI-2651](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-2651): -reclassifying twelve workflow action contracts wrongly recorded as instances of -`#V#mcp_tool`. - -| Route | Result | -|---|---| -| Durable workflow executor | Supplies no delegation; the original RR-001 observation | -| `add_relationship` on direct stdio | `ontology_sessionless_delegation_not_supported` | -| `von_chat_run` | `von_chat_run_read_only`; an explicit `allow_writes=true` is rejected outright | - -Three unrelated routes, one absent issuer. The defect is therefore in the -delegation transport rather than in the security invariant, which is a stronger -claim than this record originally supported. - -The `von_chat_run` observation is distinctive and worth separating from the -others. Its own access profile reports `write_mode: -enabled_canonical_primary_with_explicit_approval` and -`write_category_tools_allowed: true`, while the entry point is unconditionally -read-only. Write capability is configured for the profile and unreachable -through it. Configuration that describes an affordance the code will never grant -is a quieter failure than a denial, because nothing contradicts it until someone -tries. - -The blocked effect had every property that should make authorisation easy: -method known, arguments known, targets known, prior state captured, fully -reversible, and explicitly approved by the repository owner. No surface -distinguishes that case from an open-ended grant, so both are refused -identically. The stated reason for closing sessionless mutation is sound — -binding a grantor before caller arguments are matched to a stored exact intent -would expose a private read oracle — but that reasoning does not apply to an -exact specified effect, and no route exists that can tell the two apart. - -This widening was recorded rather than opened as a separate case. The phenomenon -is one missing issuer observed three times, and splitting it across records would -make a systemic gap read as several local ones. A case log that accumulates an -entry per instance while nothing consolidates them is the ratchet operating on -the evidence log itself. - -#### Evidence that would change this record - -This record should be revised, narrowed, or closed if later evidence shows any -of the following: - -- the failed workflow did receive a valid production-issued delegation and the - recorded denial arose from a different authority fact; -- the executing route did not have server-established actor identity, or the - intended effect crossed the actor's private publication boundary; -- exact workflow-instance read-back also established the requested article - postcondition through evidence omitted from the capsule; -- the route-wide delegation requirement prevents a concrete materially - unacceptable outcome that trusted actor binding, exact private scope, - bounded effect policy, receipts, read-back, and recovery cannot adequately - contain; or -- an outcome-level replay shows that the described causal sequence is no longer - present in the relevant release. - -Repair selection, delivery criteria, implementation progress, and release -evidence do not belong in this case record. They are current design and work -tracking concerns owned by -[JVNAUTOSCI-2649](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-2649). -The case remains open as historical evidence until a dated outcome-level result -supports reclassification; that status does not prescribe which repair must be -used. - -### RR-002 — A correct fix that grew a surplus guard, and the guard's own defence - -- **Observed:** 19 August 2026 -- **Status:** Closed by subtraction — the surplus mechanism was removed the same - day, before it had ever fired -- **Related work:** [JVNAUTOSCI-2650](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-2650), - PRs #402 and #403, reversal in this change -- **User outcome:** Resolve MCP tool concepts from code instead of stale copies - in MongoDB -- **Distinguishing feature:** Unlike RR-001, the causal diagnosis here was - correct and evidenced by a stack trace. The pathology is in what was built - *on top of* a correct repair. - -#### What was observed - -While implementing virtual concept providers, a single `fetch_concept` never -returned. A `faulthandler` stack dump showed an unbounded cycle: access control -asked whether a concept was virtual, the tool provider answered by building the -contract registry, building contracts loaded tool metadata from Vontology, -loading that metadata ran an access-control check, which asked again. -`lru_cache` does not guard re-entrancy, so each level rebuilt from scratch. - -#### The repair, and the surplus - -The no-new-mechanism fix was to make `owns` decide from a cheap in-memory name -set. That alone resolved the failure completely. - -A thread-local re-entrancy guard was then added *on top*, refusing to recurse if -the cycle were ever reintroduced. It was justified by a hypothetical future -provider rather than an observed failure class, it had no removal condition, and -it degraded a loud failure into a quiet one: the suppressed call answers "not -virtual" for a concept that is virtual, which in access control means falling -through to rules a virtual concept cannot satisfy. A defence against a -hypothetical fault could therefore have denied a real one. - -#### The revealing second turn - -Asked whether the hazard was documented well enough to prevent recurrence, the -implementing agent answered by adding more machinery — a trip counter, a log -line, a `guard_trip_count` accessor and a further test — and shipped it as -"Make the re-entrancy rule enforceable". It never asked whether the guard should -exist. That is the article's "when a neighbouring case fails, another mechanism -is added outside the first", occurring inside a change whose stated purpose was -to harden a guardrail. - -#### What was kept and what was removed - -Removed: the guard, its counter, its logging and the two tests that existed only -to exercise it. - -Kept: the rule stated on the `VirtualConceptProvider.owns` contract where an -implementer reads it, and a structural test asserting that no registered -provider's `owns` reaches `get_canonical_tool_registry`, `get_tool_metadata`, -`_load_from_vontology` or `can_access_concept`. - -That test is itself worth recording, because two earlier versions of it were -useless. A version asserting the guard-trip count stayed at zero passed with the -bug deliberately reintroduced, because the contract registry was already warm. -A version that cleared the caches first also passed, because closing the cycle -needs stored concepts for the access check to run on. Only tripwires on the -forbidden calls failed against the reintroduced bug. Two coherent, plausible -regression tests provided no evidence at all, which is the first article's point -about a conjecture repeated in executable form. - -#### A dual path this case declined to remove - -The same programme left 52 materialised `#V#mcp_tool` rows coexisting with -virtual resolution, with materialised winning and no removal condition — the -"permanent dual paths" warning. Removing them was attempted and **declined on -evidence**: - -- 12 of the 52 are not MCP tools at all. They have empty attributes, no - `mcp_tool_name`, and are each the target of `#V#invokesAction` from an - `#V#entity_representation_*_step`. They are workflow step actions - misclassified as instances of `#V#mcp_tool`, and deleting them would be data - loss with no replacement. They need an ontology repair, not a deletion. -- The remaining 40 carry operational metadata the virtual provider does not - serve: `display_template` on 34, `user_salience` on 34, `planner_hint`, and - evidence-contract wiring on 8. - -The rows are therefore not redundant with the virtual path. They are a second -copy of a *different* class of data whose code defaults already exist in -`_DEFAULT_TOOL_METADATA`, and which has silently diverged in both directions: -20 of 39 named tools match their code defaults exactly, and 19 differ, some -where the code appears newer and some where only the database holds a value. - -Deleting them would have destroyed the only copy of 19 tools' curated values. -Recording this as a declined action matters more than the deletion would have: -the "obvious cleanup" was itself a candidate ratchet move, removing machinery on -a tidiness argument rather than evidence. - -#### Evidence that would change this record - -- The removed guard proves necessary because a provider reaches the cycle - through a path the documented rule and structural test do not cover. -- The structural test is shown to pass against some other realisation of the - same cycle, indicating it recognises one incident rather than a failure class. -- Reconciliation of the 19 divergent tool-metadata entries shows the database - values were stale rather than curated, making the rows straightforwardly - deletable after the values move into code. - -### RR-003 — A mechanism built without checking whether it could function - -- **Observed:** 20 August 2026 -- **Status:** Closed by redesign the same day; the gate was replaced with a - nightly report before merge -- **Delivery tracking:** - [JVNAUTOSCI-2656](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-2656) -- **Distinguishing feature:** RR-002's guard worked and was merely surplus. This - mechanism could not work at all, and the fact that established it was - available before any code was written. - -#### What was observed - -Backend pytest has never run in CI. `test_mcp_manifest_parity` was failing for -an unknown period and was found by accident while investigating something -unrelated. Measurement then established that 110 of 3776 `cost_normal` tests -fail on main with a working database. - -The diagnosis was correct and measured rather than conjectured, which is worth -stating plainly: this is not a case of a weak causal theory. - -#### The response, and what was wrong with it - -A per-pull-request gate, a shard matrix, a runner script, a 209-entry -deselection backlog, and a further test guarding that backlog against growth. - -`main` has no branch protection and no required status checks. Four of the last -twenty commits went directly to it. A failing gate therefore blocks nothing. The -mechanism was incapable of doing the job it was built for, and one API call -would have established that before any of it was written. - -The simplest credible alternative, a nightly report, was never compared. The -first article asks for exactly that comparison, and the second describes using -the argument to change the burden of proof on new mechanisms. Neither happened. - -When the gate then failed on runner timeouts, the response was more machinery: -four shards, then six, then an attempt to tune Mongo timeouts. Each step -addressed the symptom. None asked whether the gate should exist. That is the -article's "when a neighbouring case fails, another mechanism is added outside -the first", observed across a single afternoon. - -#### What stopped it - -A direct question from the repository owner: why do we really need this. No -internal check caught it, and no test could have. - -#### The finding about the case log itself - -RR-002 records this precise reflex, surplus machinery on a correct diagnosis -defended with further machinery when challenged. It was written the previous day -by the same agent that then repeated it. - -Recording a case did not prevent its recurrence within twenty-four hours. That -is evidence about what this log is: a lens that works when deliberately picked -up, not a passive safeguard that operates by having been written. The second -article anticipates the distinction when it warns against converting the essay -into a checklist; this is the same limit seen from the other side, where the -material is available and simply is not consulted at the moment of decision. - -#### What survived - -The measurement. That 110 tests fail with a database and 208 without, that -`lane_backend_core` holds 3488 of 3776 `cost_normal` tests so a lane matrix -cannot parallelise the suite, that shortening Mongo timeouts makes the suite -slower rather than faster, and that at least one test depends on state another -test leaves behind. None of that is contingent on the mechanism that was -abandoned. - -#### An ambiguity the response assumed away - -That 110 tests fail while nothing visibly breaks admits two readings: -enforcement is missing, or those tests were not protecting anything. The second -is the accidental-constitution failure described in the first article. Only the -first was considered, and the backlog was framed as work owed rather than as a -question. - -#### Evidence that would change this record - -- Branch protection is introduced, at which point a gate becomes capable of the - job and the redesign should be revisited. -- The nightly report proves insufficient to surface drift within a useful time. -- Review of the 209 recorded failures shows they protect outcomes that matter, - settling the ambiguity above in favour of enforcement. diff --git a/docs/engineering/represented_advice_design.md b/docs/engineering/represented_advice_design.md deleted file mode 100644 index 5dedcb1a0..000000000 --- a/docs/engineering/represented_advice_design.md +++ /dev/null @@ -1,1659 +0,0 @@ -# Represented Advice for Adaptive Von Behaviour - -- **Kind:** Design proposal -- **Lifecycle:** Draft -- **Authority:** Advisory proposal under `AGENTS.md`; it does not define live - runtime behaviour, grant effect authority, or make advice retrieval mandatory -- **Authority scope:** Von-authored defeasible guidance for role, capability, - workflow, tool, knowledge-acquisition, introspection, message, and task - decisions -- **Owner:** Von maintainers -- **Current implementation and retrospective:** - [JVNAUTOSCI-2719](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-2719) - and [JVNAUTOSCI-2720](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-2720) -- **Last reviewed:** 5 September 2026 UTC -- **State or evidence as of:** Public Von - `dc84fc1a850890b32b89bdc8b3e0da96cfce4264`, a read-only audit of the - configured Vontology and workflow-instance store on 4 September 2026, and - the repository subtraction, non-active candidate implementation, and isolated - experiment support described below. JVNAUTOSCI-2720 completed a bounded - negative learning cycle: source-derived candidate, later trial exposure, - canonical contrary evidence, rejection and subsequent verified absence. - The [experiment dossier](jvnautosci_2720_learning_cycle_2026-09-05.md) - records the exact result and earlier inconclusive comparisons. No positive - advice efficacy, production activation or deployment is claimed -- **Supersedes / superseded by:** Nothing. This proposal owns represented-advice - semantics, applicability, retrieval/projection, lifecycle, and retraction; - the broader - [automated-policy-learning design](automated_policy_learning_design.md) - retains actor-critic, candidate-generation, evaluation, and promotion scope - and defers its textual-guideline details here -- **Review trigger:** Experience or discussion yields a potentially reusable - candidate; a concrete runtime consumer is selected; an autonomous maintenance - canary is proposed; or a general advice vocabulary/projection is proposed - -## 1. Decision summary - -Von should explore represented advice, but should not begin by building a new -universal advice subsystem. - -There is nevertheless a credible unification path. Von can converge the soft, -defeasible parts of its current guidance surfaces on a **shared interchange, -projection, and lifecycle protocol**, while leaving decision semantics, -authority, storage, ranking, and specialised policy with their existing -owners. This is a federated maintenance and projection protocol for advice, -not a universal advice ontology, central ranker, authority plane, or mandatory -runtime service. - -The proposed meaning is: - -> **Advice is a short, provenance-bearing, defeasible text artefact authored by -> Von, retained as policy memory, and linked to the roles, concepts, -> capabilities, workflows, stages, or decision kinds for which it may improve a -> later judgement.** - -“Policy memory” here uses the broad category already defined by Von's memory -architecture; it does not make advice an enforceable policy or invariant. - -Advice is a revisable hypothesis about how Von might do better. It is not an -instruction, fact, workflow transition, permission, commitment, or proof that -its originating diagnosis was correct. The model making the existing decision -may use, adapt, or reject it in light of the governing objective and purpose, -shared situation, available capabilities, and observed evidence. - -Learning from experience and discussion is a central purpose of the Von -architecture, not an exception invoked only after repeated failure. A -conversation may expose a useful correction, rationale, method, local practice, -successful strategy, or conjecture before telemetry supplies a recurring -failure class. Von should be able to preserve that as an attributed, non-active -candidate without making it a prompt instruction or production policy. - -The gates are therefore deliberately separate: - -- **candidate formation** asks whether an experience or discussion contains a - material, potentially reusable lesson worth retaining; the minimum is its - body, contributor/source, and intended target, with richer evidence or - lifecycle structure added only when actual reuse needs it; -- **runtime activation** asks whether a concrete consumer and outcome comparison - show that the candidate helps; and -- **autonomous promotion or revision** asks whether the evidence and standing - authority justify changing the active projection without human intervention. - -The Reliability Ratchet constrains active projection and generalisation; it -must not become a reason for Von to avoid learning or to discard a useful -discussion simply because production efficacy has not yet been tested. - -The Phase 0 design decision is therefore: - -1. treat advice as a **semantic role and lifecycle contract** over existing - Vontology text, profiles, and policy-memory mechanisms; -2. subtract the dormant workflow-experience producer, evaluator-facing - projection, and unpopulated selected-workflow policy-memory injector before - publishing a general `Advice` ontology or inserting another model stage; -3. require any future advice lookup to be optional, batched, scoped where its - source or target requires it, and unable to remove candidates or enlarge - authority; -4. reify a first-class advice individual only when independent identity, - cross-target reuse, revision, evaluation, or retraction earns that extra - representation; and -5. remove or collapse the mechanism if the same content works as well in an - existing prompt or workflow profile. - -The bounded Phase 0.5 implementation makes the positive half of that decision -concrete. It adds an agent-reachable `learning_candidate.v1` artefact and -`capture`, `get`, `list`, and `revise` operations for a material lesson linked -to an actor- or organisation-visible conversation or episode critique. -An unmaterialised chat-history session is the narrower actor-owned source form. -The body is a canonical `hasDescription` relation; compact structured metadata -keeps source, contributors, target, audience, beneficiaries, purpose, -visibility, revision history, and Von authorship distinct. Its lifecycle remains -`non_active`; experimental evidence may give the exact revision an inspectable -`undecided`, `retained`, `rejected`, or `retracted` disposition without making -it active. Ordinary turns perform no automatic capture, selection, evaluation, -activation, or promotion. - -JVNAUTOSCI-2720 adds an explicitly invoked, fixture-bound Phase 1 experiment. -It can ask Von to form one candidate from a frozen source discussion, compare -the exact bytes as an optional sidecar against a truly withheld control at the -existing direct-adaptive model call, evaluate the trials blindly, and record a -canonically recomputed disposition. This is experiment-local projection, not -an ordinary runtime advice consumer. Candidate formation is learned policy -memory; behavioural efficacy and the value of represented retrieval remain -separate empirical claims. - -If the first local consumer and a second independently motivated consumer both -earn reuse, they may share this lifecycle vocabulary while retaining -consumer-owned loops: - -> evidence → candidate revision → scoped active selection → bounded projection -> → observed outcome → revision, graduation, expiry, or retraction - -The shared vocabulary makes maintenance interoperable; it does not create one -central maintainer or decide the underlying tool, workflow, acquisition, -message, task, or role question for every consumer. - -The broader identity decomposition, active lifecycle terms, maintenance worker, -and later role application below remain candidate mechanisms, not a package to -implement together. Explicit MCP reads can expose candidate text to a calling -model for deliberation. The Phase 1 runner can also supply one exact candidate -revision to an isolated decision call, but no ordinary active binding, resolver, -or configured production projection is selected. Later machinery is adopted -only when the preceding evidence independently earns it. - -This design is intended to keep useful conjectures revisable for longer. It -would fail its purpose if every disappointing episode simply became another -permanent paragraph in Von's context. - -## 2. Advice's place in Von's cognitive architecture - -Several nearby artefacts contain text, but they do different jobs: - -| Artefact | Question it answers | Authority | -|---|---|---| -| Observation or episode | What happened? | Evidence only | -| Assertion or domain knowledge | What is put forward as holding in a context? | Epistemic content with provenance; not automatically true | -| Advice | What might help at this decision, and under what circumstances? | Defeasible advisory policy memory | -| Prompt or role policy | How should this model stage generally behave? | Authored behaviour within its selected scope | -| Workflow | What reusable process can be executed and recovered? | Executable procedure whose effects remain independently authorised | -| Capability or authority boundary | What may the actor and system do? | Hard maximum access and effect boundary | -| Task or commitment | What work has been undertaken? | Durable obligation or external coordination state | - -Advice may influence a decision within these boundaries, but it does not alter -their meanings. In particular: - -- advice about a tool does not make the tool available or authorised; -- advice about a workflow does not establish that the workflow ran; -- advice about a knowledge-acquisition goal is not acquired knowledge; -- advice to inspect something is not diagnostic evidence; -- advice to send a message or create a task is not authority for that effect; -- advice about a role does not assign the actor new responsibilities or - permissions; and -- repeated advice does not become an invariant merely through repetition. - -A stable, broadly applicable lesson may eventually be incorporated into a -prompt or workflow. A concrete unacceptable outcome may justify an exact code -or authority boundary. Those are separate promotion decisions. When advice is -replaced by a stronger artefact, the active advice should be retired rather -than left as a second route saying the same thing. - -## 3. Intended value and counter-hypotheses - -The job is not “retrieve advice”. It is for Von to help a person, a human/AI -team, or an organisation recognise and carry out a useful role, choose an -adequate route, produce the relevant work product, and recover intelligently -with low latency and human burden. Some roles have no immediate user. Their -purpose and evidence may instead concern a team, an organisation, a social -group, society, humanity, or the Earth and its ecosystems. - -These levels are not interchangeable. Contributor, acting principal, target -role, audience, beneficiary, and purpose remain distinct where the work needs -them. A broader purpose does not grant authority or establish one uncontested -good; it identifies whose purpose is represented and which work product, -affected parties, competing values, or real-world consequence may matter to the -claim. Do not collapse them into user satisfaction, organisational convenience, -or one scalar reward. - -Represented advice could help through four causal mechanisms: - -| Claimed benefit | Plausible mechanism | Important counter-hypothesis | -|---|---|---| -| Faster | Avoid a known bad route, redundant discovery, unnecessary model call, or low-value question | Lookup, ranking, and extra prompt tokens cost more than the avoided work | -| More effective | Reuse local practice and prior recovery knowledge across turns without hard-coding it | Stale advice anchors the model and displaces better current judgement | -| More accurate | Prompt acquisition of missing evidence, preserve source limits, and recall failure patterns | Advice preserves a mistaken causal story and makes errors more coherent | -| Better role and purpose fit | Retrieve advice linked to the current role, responsibility, work product, organisation, beneficiary, purpose, and handoff | Generic role advice becomes a stereotype or fossilised process that conflicts with the actual situation or wider purpose | - -All four are hypotheses. A shorter prompt, better workflow description, better -tool metadata, stronger suitable model, or no extra mechanism may win. - -Advice is particularly promising where all of the following hold: - -- the decision remains semantically contingent rather than a hard invariant; -- experience has value beyond one episode; -- the lesson needs revision, provenance, or reuse independent of a large base - prompt; -- a consumer can retrieve it cheaply at an existing decision point; and -- success and negative transfer can be observed on the relevant work product or - end-to-end outcome at the appropriate person, team, organisation, or wider- - beneficiary level. - -It is a poor fit for an exact interface constraint, a one-off case judgement, -raw telemetry, a long procedure that should be a workflow, or a platitude a -capable model already follows reliably. - -Against Von's longer-term aims, the intended contribution is specific: - -- **reliable:** expose which revisable guidance affected a choice, preserve - counterevidence, and make withdrawal and recovery cheaper than another code - release; -- **knowledgeable:** use concept links to find relevant practice and recurring - evidence gaps, while leaving facts and acquired knowledge in semantic memory - rather than laundering them into advice; -- **adaptive:** bind revisions to roles, workflows, stages, actors, teams, and - changing model/tool dependencies without freezing one global rule; -- **autonomous:** let Von perform bounded proposal, comparison, drift detection, - revision, and retraction through independently governed workflows; -- **minimal-imposition:** reduce repeated discovery, clarification, approval, - and correction while spending human attention only on materially different - value, authority, publication, privacy, or coordination choices; and -- **team-effective:** retain who proposed, corrected, approved, or disputed a - practice and support better work products and handoffs without turning team - custom into universal policy; and -- **organisation- and purpose-effective:** learn roles, knowledge, and practice - that serve an organisation or a wider social or ecological purpose even when - no particular user is the immediate beneficiary, while keeping whose purpose - and evidence it is inspectable. - -## 4. This is not greenfield - -The repository contains several overlapping advice-like surfaces and now -records one retired prototype: - -| Existing surface | What it already provides | Design consequence | -|---|---|---| -| [`workflow_routing_profile.v1` and workflow discovery exemplars](../../src/backend/workflows/vontology_loader.py) | Vontology profiles store and carry routing role, direct-equivalence, selection guidance, examples, exclusions, and routing notes; current selector candidate rendering does not explicitly project `selection_guidance` | Audit and reuse the stored field deliberately if it proves useful; launchability remains independently established, and storage alone does not change selection | -| Retained episode evidence and critique memory | TERs, episode evidence bundles, evaluator axes, critic assessment, implicated workflows/tools/concepts, receipts, remediation links, and structured `episode_improvement_suggestion.v1` items remain persisted in `episode_critique_memory.v1` | Use this as a source-native observation and candidate substrate after checking its evidence and quality; do not rebuild it or mistake a critic suggestion for validated advice | -| Workflow Studio improvement guidance | The current human-facing workflow detail view lists recent critique suggestions with target, category, rationale, proposed change, evidence references, and source episode, and explicitly does not auto-apply them | Treat this as an existing candidate-discovery consumer for human deliberation, not as a model-facing runtime advice consumer or efficacy proof | -| Retained explicit evaluation and workflow self-improvement paths | Explicit episode evaluation can persist critique memory; represented proposal and promotion-recommendation workflows can turn a selected workflow suggestion into an isolated workflow candidate | Reuse only when a real selected candidate needs them; the live critique projections record no self-improvement launches, proposals, or promotion evaluations, so their operational loop is unproven | -| Retired workflow-experience guidance induction | The audited repository path attempted to author successful-run, failure-avoidance, and exploration text after every persisted episode assessment | Preserve those three purposes as optional lenses, but do not republish the forced producer; current criticism is repair-oriented and does not replace positive-practice or exploratory learning | -| Retired `#V#workflow_experience_context_prelude` | The audited prelude derived a workflow/model profile and read the latest five values of three additive relations for four direct proposal/evaluation/release callers | Replace the live-capable graph with an unpublished, actionless terminal tombstone and keep it out of those callers; recency was not activation, applicability, deduplication, or independent evidence | -| Retired selected-workflow policy-memory projection | A separate critique-memory-backed builder had no in-repository caller, while the orchestrator could still render caller-supplied suggestions as system text | Remove the builder, renderer, turn-stage injection, and misleading lineage path; retain the suggestion store and Workflow Studio discovery without treating unselected model output as promoted policy | -| [Tool planner hints](../../src/backend/services/tool_metadata_service.py) and [outcome-explanation maps](prompt_programs_and_model_routing_playbook.md#3b-outcome-explanation-guidance) | Specialised model-facing guidance at established tool and explanation surfaces | Keep domain-specific shapes where they are the simplest adequate representation | -| [Knowledge-acquisition profiles](../../src/backend/services/knowledge_acquisition_profile_vontology_service.py) | Typed defaults, thresholds, auto-application, and fail-closed behaviour | Treat these as stronger acquisition policy requiring its own ratchet review, not generic soft advice | -| [Publication-scope profiles](ontology_publication_authority.md#represented-publication-scope-profiles) | A concrete code-level precedent for defeasible Vontology advice separated from mutation authority | Reuse its boundary, not the domain schema or resolver unchanged: an invalid accessible profile can currently invalidate the decision, protected outcomes dominate ordinary conflicts, and add-only seed reconciliation can resurrect removed profiles | -| [Workflow selection experience and policy](../../src/backend/services/workflow_selection_policy_service.py) | Process-global outcome-derived numeric priors, lexical token affinity, and exploration signals that advise the selector while leaving semantic choice to the model | Treat this as a competing diagnostic baseline, not privacy-compatible production advice until its population and scope are isolated | -| [Operational learning release support](../../src/backend/services/operational_learning_release_vontology_service.py) | Candidate evaluation, artefact-specific active pointers, receipts, rollback, and exact namespace/user/organisation scope | Reuse proportionately for consequential promotion in that supported scope; it is not yet a general private-advice lifecycle and must not be imposed on every hint | - -The retired workflow-experience implementation supplied direct evidence of a -ratchet in the surrounding Python, tests, seed migrations, and Jira/design -record, plus plausible mechanisms by which represented guidance *could* -accumulate: - -- one episode-evaluation path renders and writes all three guidance kinds, - including a literal “no durable lesson” fallback; -- the three additive writes are sequential, so a later failure can leave a - partial set; the pilot must first establish whether that partial set is - harmful or the hints are independent, in which case idempotent writes with - receipts are preferable to an unnecessary transaction; -- its gateway test deliberately permits another relation containing the same - guidance body but a different request identifier; -- the retrieval prelude reads the latest five relations for each of three - predicates, rather than an active, deduplicated, applicability-aware view; - and -- its profile key contains workflow and model but the retrieved items are not - filtered by their recorded stage, supersession, expiry, utility, or - retraction state. - -It did **not** supply evidence that a ratchet had occurred in Von's knowledge -structures. The bounded live audit found no populated guidance to preserve or -promote. In the configured database there were zero text relations and zero scoped -assertions across all three guidance predicates, zero distinct linked guidance -profiles, and zero recorded prelude workflow instances. Subworkflow invocation -need not always create a separate instance, so the instance count alone is not -proof of non-execution; the empty guidance stores and source/runtime path are -the stronger evidence of dormancy. - -The same audit found 134,215 episode-evaluation workflow instances. The latest -100 by indexed creation time contained 81 cancelled and 19 failed instances, -with no success; the latest five failures stopped at the episode-evaluation -model step before assessment persistence or guidance induction. Before this -repository change, the completion-gate binding was enabled, the terminal-event -binding was disabled, and the integration defaulted automatic triggering on. -These observations distinguish a failing automatic producer from a useful -advice capability; they do not establish the state of every historical -instance or justify deletion of raw evidence. - -This satisfies the Phase 0 stop rule for the unused runtime mechanism, not a -stop rule for learning from experience or discussion. The repository -implementation therefore removes the guidance authoring tail, prompt -registration and seed vocabulary; -replaces the live-capable prelude with an unpublished, actionless terminal -tombstone; removes its four proposal/evaluation/release injections and the same -three retired history fields from 17 other LLM states that could still project -caller- or parent-supplied values; and defaults both -episode-evaluation bindings and the integration trigger off. On -repository bootstrap, canonical lifecycle write and read-back also mark the -tombstone retired and routing-ineligible. Explicit episode evaluation and raw -episode evidence remain. No live ontology artefacts or historical instances -were deleted, and this change is not a live deployment. A future resolver must -still never translate “latest” into “promoted”. - -The durable worker currently begins polling before these non-critical workflow -families finish their startup migrations. A bounded live read found no pending, -running, or paused instances for the retired prelude, episode evaluators, -self-improvement workflows, or affected operational-learning workflows, so no -present instance was found that could exercise that window. A future deployment -must inspect its actual worker topology and pending instances before deciding -whether quiescence is needed; this repository change supplies no evidence for a -universal quiescence requirement and does not claim rolling-deployment safety. - -The separate selected-workflow policy-memory builder read recent critique -suggestions from Mongo. A static source scan found no in-repository caller of -the builder, but the orchestrator would still render a caller- or parent-supplied -`selected_workflow_policy_memory_state` as system text in tool-planning and -synthesis contexts, and record it as lineage. Its source query also made -`namespace` optional and had no independent user/organisation access check. -Phase 0 therefore removes the builder, renderer, two turn-stage injections, and -lineage projection rather than preserving a dormant authority aperture. - -The retired episode-evaluation autotrigger default and completion-gate binding -could launch a large amount of failing work. They now default off; explicit -episode evaluation remains available. If later evidence identifies a worthwhile -recurring maintenance job, compare a separate asynchronous batched process with -the simpler direct path rather than piggybacking on every episode. - -Operational-learning releases provide valuable immutable-candidate, scope, -expiry, pointer, receipt, and rollback mechanics, but cannot be reused -unchanged. Registration currently requires failure-evidence packets, although -useful advice may arise from success or exploration, and rollback requires a -previous active release rather than supporting first-release retraction to the -ordinary empty/no-advice state. - -These mechanisms also do not share one visibility model. Some reads use an -ambient actor-effective Vontology view, some use the base-publication view, -tool metadata uses a process-global catalogue cache, and the latent critique -path accepts an optional namespace. A general resolver cannot inherit any one -of those behaviours unchanged. - -### 4.1 Teleological reading of the retired feedback loops - -Retirement is a judgement about the implemented causal path, not necessarily -about the purpose that path was trying to serve: - -| Intended function | What should survive | What should remain retired | -|---|---|---| -| Notice experience continuously | Durable episodes, TERs, receipts, explicit evaluation, and bounded sampling or event selection | An extra LLM and mutation tail attached to every episode without a demonstrated learning consumer | -| Learn what worked, what failed, and what to try next | Successful practice, failure avoidance, and low-imposition exploration as optional candidate lenses; allow zero, one, or several lessons from experience or discussion | Requiring all three outputs for every episode and storing a literal “no durable lesson” as guidance | -| Make a lesson reusable in its context | Source workflow/model/stage can be useful applicability evidence; roles, work products, teams, organisations, beneficiaries, and purposes may also be the right target | Creating durable `unknown_workflow`/`unknown_model` profiles or projecting stage-mismatched items | -| Preserve lessons outside code | Vontology remains a plausible home when independent identity, linking, revision, or reuse earns it; source-native conversation or critique memory is enough before then | Immediate additive writes whose mere recency stands in for selection, applicability, or activation | -| Bring prior experience to a later decision | Discover non-active candidates for human/AI deliberation, then project only a deliberately selected candidate at a concrete decision point | Latest-N injection, raw caller-supplied history as system text, or candidate history in its own evaluation/release context | -| Turn findings into maintained improvements | Workflow Studio inspection, bounded task/proposal handoff, evidence-linked candidate worlds, and later outcome comparison | Hard-coded escalation thresholds or automatic Jira/promotion behaviour copied into advice merely because those mechanisms exist | - -The live critique store is therefore evidence and research material, not waste -and not already-learned policy. A bounded read on 4 September 2026 found 2,083 -`episode_critique_memory` projections in the configured namespace: 585 pass, -240 fail, 604 inconclusive, and 654 follow-up-required. Of these, 583 contained -2,118 structured improvement suggestions spanning workflow, telemetry, prompt, -tool, contract, verification, critic, and support surfaces. The data also shows -why direct activation would be unsound: 610 memories lacked workflow identity, -1,291 lacked a routing fingerprint, the latest 20 were all inconclusive with no -suggestions, generic identifiers such as `s1`–`s5` recur across unrelated -workflows, and no projection recorded a self-improvement launch, proposal, or -promotion evaluation. Repeated suggestions may be useful recurrence evidence; -they are not 2,118 independent lessons or proof that the suggested change helps. - -Use this corpus to study evaluator behaviour, recover concrete candidate ideas, -and link recurring observations back to code, tests, Jira, discussions, and -outcomes. Re-ground any candidate before reuse. The 134,215 workflow instances -are likewise operational evidence about the failed mechanism and its cost, not -134,215 learning events. No historical instances, critique memories, or raw -episode evidence are deleted by Phase 0. - -The positive gaps are now sharper. Phase 0.5 supplies attributed -discussion-to-durable-candidate capture plus AI discovery and revision of -non-active candidates outside workflow-only Studio views. Successful-practice -induction, optional low-imposition exploratory lessons, a unified human -candidate view, and selected-advice exposure-to-outcome lineage remain -unimplemented. None requires restoring the retired loop. - -Role advice is not a shortcut around role representation. The -[programme design](Von_for_AgenticAI.md) describes a represented role with -responsibilities, authority, workflows, and work products, while the current -[task-ontology service](../../src/backend/services/task_ontology_service.py) -uses role primarily as text framing. Role-linked advice can therefore be -captured when experience or team discussion supplies a potentially reusable -lesson. The role may serve a person, team, organisation, community, humanity, -or ecological system and need not involve a direct user. When material, keep -its contributors, acting principal, beneficiary, purpose, target identity, -responsibilities, and intended scope inspectable rather than treating any one -of those as a proxy for the others. Activating advice still requires a concrete -role decision and outcome evidence appropriate to its work product or purpose; -candidate capture is not a shortcut to assigning responsibilities or authority. - -### 4.2 The unification boundary - -The useful commonality is deliberately narrower than “everything that guides a -decision”. A shared layer may own: - -- stable advice/revision identity and exact target links; -- provenance, evidence lineage, visibility, lifecycle, and removal triggers; -- mechanical proposal, compare-and-set selection, supersession, retirement, - rollback, and exact read-back operations executed only from the consumer's - authorised release decision; -- a bounded visibility-safe projection contract and failure diagnostics; -- exposure/outcome telemetry, dependency drift signals, and token budgets; and -- reversible adapters through which an existing surface exposes soft advice - without migrating its native schema. - -The consumer or specialised policy continues to own: - -- domain applicability beyond exact target matching; -- relevance, conflict interpretation, and decision-specific evaluation; -- whether and where advice enters its existing model call; -- the final action choice and its independent authority checks; -- native storage and stronger prompt, workflow, score, or policy semantics; and -- activation, retirement, and rollback decisions and promotion criteria - appropriate to the consequence and audience. - -The following soft components are plausible convergence candidates: - -- future workflow-local success, failure-avoidance, and exploration guidance - that is independently motivated rather than restored from the retired path; -- the `selection_guidance` portion of workflow routing profiles, but not - eligibility, launchability, routing role, or direct equivalence; -- evidence-conditioned or scope-specific tool-planner prose, while static - global tool descriptions may remain metadata; -- recovery, introspection, role-practice, message-structure, task-practice, and - explanatory advice once their target and outcome identities are represented. - -The following remain specialised even if they reuse versioning, evidence, or -release primitives: - -- workflow control flow, postconditions, and deterministic follow-up actions; -- tool schemas, availability, dispatch, and effect authority; -- knowledge-acquisition thresholds, automatic application, and fail-closed - behaviour; -- publication-scope semantics and protected-carrier conflict handling; -- prompt bodies and outcome-explanation maps; -- numeric workflow/model-routing policies; -- roles, tasks, commitments, assertions, and raw episodes; and -- security, visibility, and authority boundaries. - -An adapter may expose an explicitly soft projection from one of these stronger -surfaces. It must not relabel the whole source as advice or weaken its native -semantics. If a common implementation starts acquiring tool-, workflow-, -publication-, or role-specific branches, move those semantics back to the -consumer or abandon the abstraction. - -## 5. Design principles - -1. **The current job and purpose lead.** Advice supplements the explicit - request, represented role or organisational objective, broader purpose, and - shared situation that actually govern the work; it does not redefine them or - assume that a particular user is the beneficiary. -2. **Preserve alternatives.** Advice may change model-visible salience or - ranking. It must not remove an otherwise authorised tool, workflow, direct - route, recovery, or answer strategy. -3. **Advice is not authority.** Trusted context determines visibility; the - acting principal and delegation determine capability. The eventual effect - independently verifies authority. -4. **No universal advice stage.** A consumer opts in at an existing decision - point only when prior evidence suggests advice could change the outcome. -5. **Use the weakest adequate representation.** Attached text is preferable to - a new first-class individual until reuse, lineage, or retraction needs more. -6. **Retrieve a small relevant view, not a history.** Raw episodes and retired - advice remain auditable but do not enter the prompt by default. -7. **Absence is normal.** Missing, inaccessible, malformed, or slow advice must - not block an otherwise viable task. Readable alternatives remain usable. -8. **Keep conflicts visible.** “Newest”, “most specific”, “most restrictive”, - and “highest confidence” are not universal conflict-resolution rules. -9. **Do not self-govern.** Advice may propose a future resolver or evaluator - change, but it cannot select, rewrite, or supply the success criteria for the - resolver, evaluator, authority boundary, or activation policy governing its - current revision. Such a proposal needs a separate release and independent - evaluation. -10. **Deletion is a successful outcome.** Every production adoption has an - observable condition under which the advice or mechanism will be retired. -11. **A risk hypothesis earns observation, not machinery.** Do not add a - compulsory guard, schema, review stage, test matrix, or promotion ceremony - merely because harm is imaginable. First identify a concrete materially - unacceptable outcome, show by evidence or a clear causal path that it is - reachable in the proposed Von surface, and show why a simpler bounded, - observable, and recoverable approach is inadequate. Until then, record or - measure the uncertainty without constraining the architecture around it. - -Von authorship is not a content-safety boundary. Retrieved mail, web pages, -documents, tool output, and prior model text remain untrusted evidence even -when an evaluator summarises them into a candidate. Advice activation must not -launder source instructions into higher-priority policy. - -Promoted advice remains untrusted advisory data at consumption time. It must be -rendered as a labelled, defeasible input below the current objective, purpose, -role, commitments, observed facts, and any governing user instruction. It must not be -injected as an undifferentiated system instruction or acquire higher -instruction priority merely through representation or promotion. - -## 6. Representation - -### 6.1 Two adequate forms - -| Form | Use when | Avoid when | -|---|---|---| -| Text relation attached to one target or versioned profile | A simple candidate has one owner and consumer, while an enclosing profile or index owns any active revision | A raw attached relation itself needs revision, supersession, retraction, or separately accumulated evidence | -| First-class advice knowledge item linked to targets | The same advice is reused, revised, evaluated, contradicted, or retracted independently across several concepts or workflows | A new node would only wrap one existing prompt or profile field | - -A future first experiment should use the source-native profile or attachment -surface already owned by its selected consumer, not resurrect the retired -workflow-experience substrate by default. Its represented arm needs an isolated -explicit manifest or pointer for the candidate and active revision; raw -additive relations do not acquire lifecycle merely because they are in -Vontology. A general advice item must still earn its ontology layer. - -For advice that has earned runtime projection, whether stored natively or -exposed through an adapter, the minimum external logical contract is: - -- stable advice and revision identity when the release can change independently; -- one concise body; -- an inspectable target, using concept identifiers only where the target is - already represented that way; -- producer, provenance, and evidence references; -- a visibility carrier only where the source or target is not public, resolved - from trusted actor context and kept distinct from target and evidence context; -- lifecycle sufficient to identify what is active and what replaces or removes - it, using an enclosing source-native profile where that is adequate; -- a removal, review, or retest condition for every active production item; and -- an inspectable reason why the item was retrieved. - -When not unambiguously supplied by the exact target/carrier, the contract must -also include decision kind and applicability. Only when material to an item -should it add: - -- model, prompt, workflow, tool, or ontology-version dependence; -- calibrated confidence or utility evidence; -- counterevidence; -- an explicit relationship to competing advice. - -This is deliberately not one compulsory source schema or JSON envelope with -fourteen fields. Existing Vontology relations, specialised profiles, -text-relation context, revision records, and evidence locators may supply the -contract through reversible adapters. Before publishing new predicates, -implementation must search for and reuse the current lifecycle, provenance, -scope, and supersession vocabulary. - -If the operational-learning release vocabulary is reused, the provisional -mapping is candidate to `proposed`, active to `active`, and superseded to -`superseded`; rejected candidates use `rejected`, while `rolled_back` records a -release transition rather than acting as a synonym for retired advice. No -generic advice active pointer exists today. This mapping must remain an adapter; -it must not import the operational-learning approval rules or full release -apparatus into every ordinary hint. - -A provisional richer representation might use an `Advice` type, one content -text relation, target links such as `advises about`, evidence links, and a -supersession link. These names are design placeholders, not approved ontology -identifiers. - -### 6.2 Candidate long-term identity decomposition - -The first consumer should retain source-native identity plus an explicit -active-selection locator. If two independently motivated consumers later earn a -common layer, the following five-part decomposition is one candidate to test, -not a required schema: - -| Identity | Meaning | -|---|---| -| Advice series | The stable revisable lesson or question | -| Advice revision | One immutable expression with provenance and dependencies | -| Advice binding | The target, decision kind, consumer, applicability, and visibility context in which a revision could be useful | -| Advice release | The scoped selection of one revision for one binding, with its evidence, lifecycle state, and release authority | -| Advice projection | A derived, bounded runtime view; never an authority or canonical knowledge object | - -In that candidate, activation belongs to the binding/release, not to the -revision globally. The same revision may be active for one workflow stage, -still proposed for a second, and inapplicable or inaccessible to another actor. -Candidate lifecycle terms are `proposed`, `active`, `superseded`, `rejected`, -`expired`, and `retired`. Rollback is a transition with a receipt that restores -a prior eligible release, not another name for retirement. - -An enclosing versioned profile and explicit manifest can play these roles for -a future first pilot. First-class series, binding, and release items become -justified only when shared lifecycle operations remove real duplicated work. - -### 6.3 Body quality - -An advice body should normally express one reusable idea: - -- the situation in which it may apply; -- the action or consideration that may help; -- the intended consequence. - -Evidence identifiers, telemetry dumps, complete prompts, and stack traces stay -outside the body. The retired one-sentence, 280-character experience format is -a useful possible experimental bound, not a universal semantic requirement. - -Examples of suitably defeasible advice include: - -- “When a scholarly source exposes a DOI, resolve it before name-only author - matching to reduce duplicate identity candidates; retain the source - occurrence if identifiers disagree.” -- “For a request that may continue an earlier effect, inspect the exact - terminal receipt before launching new work; proceed directly when the new - request clearly diverges.” -- “Before asking for an acronym expansion, inspect the conversation situation - and linked project concepts; ask only if the remaining alternatives change - the work product.” - -Before runtime activation or a claim of reuse, the complete advice -artefact—not necessarily its short projected body—should make material -applicability and reconsideration conditions inspectable through the body, -source-native metadata, linked evidence, or evaluation note. A provisional -candidate may remain less structured while its usefulness and reuse are still -unknown. - -### 6.4 Targets and context - -Advice may be linked to: - -- a role, responsibility, work product, or handoff; -- an organisation, social group, beneficiary, or broader purpose when that link - is material and represented; -- a task or request type; -- a capability or tool family; -- a workflow, stage, or prompt concept; -- a knowledge-acquisition goal or evidence kind; -- an introspection or recovery surface; -- a message or task kind; or -- a model/tool/workflow version when the evidence is genuinely version-bound. - -Link to capability concepts rather than only transient tool names when the -lesson is about a stable capability. Link to exact versions when the lesson is -implementation-specific. Visibility follows the source and intended audience, -not the beneficiary: advice derived from private personal or organisational -practice remains within that visibility context unless an authorised -publication decision makes it wider, while public-purpose advice need not be -forced into a user-private carrier. - -Exact target links are strongest. Bounded ancestor or semantic matches may be -retrieved as weaker candidates but must not silently override direct advice. -Specificity affects relevance, not authority. - -Semantic applicability and visibility are independent. Resolve visibility -from a trusted actor-bound carrier before relevance ranking: a target link, -natural-language namespace, provenance context, or visible target concept never -confers access to the advice. An actorless read may return globally published -advice only. - -## 7. Runtime resolution and use - -Advice should enter an existing judgement rather than create a controller -tower: - -```mermaid -flowchart LR - S[Current job, purpose, and shared situation] --> C[Existing candidate discovery] - C --> D[Existing model decision] - T[Target concepts and applicable visibility scope] --> Q[Optional bounded advice query] - Q --> B[Small labelled advice bundle] - B --> D - D --> E[Bounded tool, workflow, message, or task action] - E --> R[Receipt, read-back, and outcome evidence] - R -. optional offline input .-> V[Evaluation] - V -. optional candidate .-> P[Candidate advice or revision] - P -. governed comparison and activation .-> X[Held-out comparison and governed activation] - X -. active revision .-> Q -``` - -### 7.1 Common projection protocol - -If a second consumer demonstrates genuine reuse, one compact runtime read model -may help even while source schemas remain federated. In a provisional -`advice_projection.v1`, the resolver derives the applicable visibility context -from pre-existing trusted server or workflow context: personal, team, -organisation, other governed group, or globally published. It rejects -disagreement with any supplied scope fields. The request supplies consumer and -decision kind, exact target and candidate identifiers, relevant dependency -versions, and an item/token budget; payload identity is never authentication, -and public-purpose work does not acquire a fictional user scope. - -Its response contains only accessible active releases and reports, for each -item, the available source-native or series/revision/binding/release identities, -concise body, exact target, source kind, retrieval reason, and any explicit -conflict group. Include an evidence reference only when it is independently -visible in the requesting context; otherwise use a non-revealing receipt or -digest. The -response also carries a projection digest, dependency versions, truncation, and -bounded error metadata without revealing inaccessible-item identities or -counts. The name, decomposition, and fields remain candidates until two -consumers demonstrate shared need. - -Common support may enforce trusted visibility, active-revision integrity, -exact-target matching, bounded size, exact duplicate identity, and inspectable -read-back. The consumer retains semantic applicability, relevance, conflict -interpretation, graph expansion, prompt position, and choice. There is no -cross-domain confidence score or common ranker. - -For the first consumer: - -1. Build the normal candidate set and minimum-sufficient situation projection. -2. If the decision is advice-enabled, issue one visibility-scoped batch read - using trusted server or workflow context, the decision kind, and already- - known target/candidate identifiers. Without a trusted private or group - context, query globally published advice only. -3. In the pilot, use exact active target links only. Add bounded ancestor or - semantic expansion later only if measured misses justify its cost. -4. Deduplicate exact logical/revision identity only. Identical wording from - distinct producers or evidence contexts remains distinct until an offline - semantic consolidation decides otherwise. Retain material conflicts and fit - the result to the consumer's small token budget. -5. Render the bundle below the governing objective, role, commitments, - applicable user instructions, and observed facts, clearly labelled as - optional prior advice with item/revision identities and retrieval reasons. -6. Let the existing decision model select among the full candidate set. Where - that model already emits a structured decision, it may name advice it - followed or rejected without exposing chain-of-thought. -7. Execute through the normal capability and authority boundary and verify the - result normally. -8. Record projection and outcome evidence for causal evaluation. - -The advice resolver may return zero items. An inaccessible or malformed item -does not invalidate readable items and must not leak its identity or count. A -material conflict may be exposed to the deciding model; it reaches the -appropriate human or team only when current evidence cannot resolve a choice -whose privacy, commitment, cost, or consequence differs materially. - -Advice retrieval does not alter `allowed_tools`, routing eligibility, workflow -publication status, or mutation authority. A candidate may become more -salient, but none may disappear solely because advice dislikes it. - -### 7.2 Consumer boundaries - -| Consumer | Useful advice | Boundary that remains authoritative | -|---|---|---| -| Role identification | Responsibilities, expected work products, handoffs, and local practice linked to a role and situation | Represented purpose and commitments, the current objective, and delegated authority | -| Workflow selection | Known applicability, counterexamples, and useful recovery alternatives | Published workflow capability, current inputs, and model judgement | -| Tool selection | Efficient tool families, prerequisite reads, and recurring failure patterns | Available tool catalogue, schemas, and effect authority | -| Knowledge acquisition | High-value missing evidence and low-imposition ways to obtain it | Current epistemic state, source provenance, and the actual work-product or purpose-level evidence need | -| Introspection and recovery | The next discriminating read-only probe or a previously effective recovery | Current telemetry, bounded task budget, and observed action order | -| Messages and tasks | Audience-appropriate structure, useful follow-up, and local coordination practice | Verified outcome, existing commitments, recipient scope, and authority to create or send | - -Advice can help role performance only after role and situation evidence have -been brought together. It must not become a persona prompt that invents duties -or treats a broad role description as permission. - -### 7.3 Latency and context discipline - -To have a credible path to making Von faster: - -- do not add a separate LLM call to select advice in the first implementation; -- batch reads instead of performing one sequential call per advice kind or - candidate; -- reuse the existing capability/graph retrieval result where possible; -- cache only derived visibility-safe projections keyed by exact trusted - personal/group/organisation/audience scope, semantic view, and - advice/ontology revision, while retaining Vontology as authority; -- never reuse the current process-global tool-metadata cache for private - advice; private cache keys include canonical actor/organisation scope and - semantic view, while a global cache may contain globally published advice - only; -- skip lookup on direct, trivial, or already-decided paths; -- cap projected advice through consumer-owned relevance policy and tokens, not - merely by “latest N”; and -- proceed without advice when the optional result is not ready in time for the - current decision. - -Begin with the direct batched read. If measured graph and assembly cost remains -material after a common protocol is earned, compile immutable scoped projection -manifests when releases change. The online path can then union only the -visibility-authorised global, group, organisation, personal, or contextual -manifests and verify their revision digests. Compilation is a derived -acceleration, not a second authority store, and must not precompute private -membership into a shared artefact. - -A late optional advice result may inform a later evaluation or cache generation -but must not restart a decision that has already safely progressed. Before -caching it, revalidate the exact scope and advice/ontology revisions, and never -place a private projection in a shared cache. Advice lookup latency, -prompt displacement, and end-to-end time are part of the result; they cannot be -reported as zero or excluded from a “faster” claim. - -## 8. Authoring and adjustment lifecycle - -### 8.1 Core lifecycle - -The desired learning loop is: - -1. **Observe and discuss.** Preserve the episode, receipt, outcome, relevant - conversational contribution, correction, decision rationale, and evaluator - result separately from any explanation. Retain who contributed what and the - situation in which it was proposed. -2. **Propose.** Von may author a concise candidate advice item and link it to - the observed evidence or discussion. A single episode or discussion may - support a narrow candidate, not automatic general policy. Advice derived from - untrusted external content remains quarantined until independent evidence - supports activation. -3. **Check redundancy and counterexamples proportionately.** Search existing - advice, prompts, profiles, workflows, and nearby episodes when candidate - volume, reuse, or the decision consequence makes that useful. Merge - equivalent candidates or leave the observation episodic when there is no - durable lesson; do not build a global deduplication service before duplicate - candidates are an observed problem. -4. **Evaluate.** Compare the candidate with the best simpler baseline on - neighbouring and held-out tasks. Test no-advice cases and materially - different valid routes. -5. **Activate narrowly.** Publish one immutable revision for the smallest - supported target and audience. Promotion authority is distinct from the - advice itself. -6. **Observe use and non-use.** Retain which revisions were projected, the - resulting actions and outcomes, and counterevidence. Do not infer that a - model used advice merely because it was in the prompt. Advice-conditioned - success is not independent confirmation of that advice: deduplicate evidence - by originating event/digest, and require a no-advice comparison or other - independent evidence before increasing confidence. -7. **Revise, supersede, or retire.** Change the explicit active selection, - artefact-specific pointer, or lifecycle relation provided by the chosen - representation rather than silently overwriting history. Retraction must - remove the item from active projections and read back exactly; there is no - generic advice pointer to assume today. -8. **Graduate or subtract.** Move a genuinely stable instruction into the - appropriate prompt or workflow, or a genuine invariant into its independently - justified boundary; retire the advice. Delete the runtime path if it adds no - net value. - -“No durable lesson” is an outcome, not advice text to append. Repeated identical -bodies should strengthen or challenge one candidate's evidence, not create -another active prompt fragment. - -Repository seeds may bootstrap missing vocabulary or a named release input. -They must not overwrite revised live content or resurrect retired advice. -Owned seed reconciliation therefore needs the same tombstone and -duplicate-identity discipline as other governed represented policy. - -Human-authored correction and discussion should remain distinguishable from -Von-authored advice. Von may synthesise a candidate from a human/AI discussion, -but it must preserve contribution provenance and must not silently relabel a -human instruction, preference, or commitment as its own advice merely to -simplify precedence. - -### 8.2 Evidence independence - -Maintenance consumes typed observations, not one cross-domain confidence or -reward score. Outcome and problem evidence may include canonical read-back, -repeated independently originating verified failures or recoveries, and an -independently verified downstream result. An explicit correction or discussion -can establish a scope-appropriate personal, role, team, organisational, or -purpose-level problem, instruction, preference, or candidate lesson, but not -that proposed advice fixes or improves later work; an unexplained artefact edit -is diagnostic only. Authenticated -approval from a suitably authorised principal can establish release authority -for an exact candidate and scope, but not efficacy. Blinded evaluator -judgements, repeated unnecessary questions or -retries, latency regressions conditional on adequate completion, and dependency -drift can support investigation. - -A model self-critique, aggregate score, advice projection or citation, user -silence, outer completion, or several AI summaries of the same episode is -diagnostic only. Every derived observation retains its root evidence lineage; -copies across telemetry, critics, and team agents still count as one originating -event. - -Candidate discovery or generation may run over a bounded experience-and- -discussion window and return one of: new candidate, revise, reinforce or -challenge existing evidence, retract, graduate, or no durable lesson. It keeps -observations and attributed contributions separate from the causal hypothesis. -A newly retained candidate needs only a useful body, contributor/source, and -intended target. Exact applicability, dependency, counterevidence, removal, -retest, and root-evidence fields become necessary only to the extent that a -later activation, generalisation, causal claim, or consequential decision needs -them. - -Non-active candidates must remain discoverable for later human/AI discussion, -comparison, revision, or rejection without being projected as runtime policy. -Retention without a practical discovery path is archival preservation, not an -effective learning loop. - -Before an evaluation used for activation or a research claim, freeze the -candidate digest and the comparison choices material to that claim, such as an -induction/calibration/held-out partition, rubric, thresholds, and relevant -resolver/release versions. The -candidate author cannot alter the evaluator, promotion rule, authority path, or -held-out cases governing its own release. Proposer, evaluator, and activator may -all be Von-operated workflows, but independently versioned roles provide -auditability rather than evidential independence. Exclude the candidate and -related active advice from evaluator context except in the deliberately -advice-on arm. Different agents or models are not independent when they share -the same trace, source, prompt lineage, or evaluator. - -Advice-exposed outcomes are labelled as such. They can reveal harm immediately, -but an independent read-back verifies the outcome, not that advice caused it. -Only a matched or randomised advice-off comparison supports efficacy; an -alternative-route replay tests whether advice suppressed opportunity. A small -bounded advice-off or alternative-route replay sample should continue after -activation where the corresponding claim matters; otherwise successful advice -can suppress the observations capable of disproving it. - -### 8.3 Bounded autonomous maintenance - -The ordinary work path should not acquire a mandatory learning stage. A -discussion may still formulate, discover, revise, or retain a candidate in its -existing conversation situation or review surface while the work proceeds. -If a bounded pilot independently earns heavier autonomous maintenance under the -post-pilot branch below, counterexample search, experimentation, consolidation, -and ordinary promotion may run asynchronously in a low-priority, visibility- -scoped, idempotent maintenance workflow. Trigger such a workflow by an actual -learning opportunity such as novelty, successful practice worth reusing, -recoverable failure or burden, expected reuse, conflict, explicit correction, -dependency drift, or a due review—not automatically after every episode. If no -recurring maintenance job earns that machinery, keep the path conversational, -human-inspectable, or manual. - -Autonomy is graduated by the consequence and scope of the maintenance effect: - -| Maintenance action | Permissible autonomy | -|---|---| -| Candidate capture and isolated analysis | May run within existing resource and visibility bounds when a real learning job warrants it; do not add inventory, deduplication, or conflict machinery until candidate use or accumulation needs it | -| Integrity response | If an implemented resolver actually observes an unreadable or invalid selected item, use its ordinary no-advice path; add automatic withdrawal only when that failure and recovery job are demonstrated | -| Semantic activation, revision, or retirement | A narrow, reversible, low-consequence change may run under standing authority when outcome evidence supports it and its actual effect is read back; A/B/C is needed only for a represented-layer claim, and recovery rehearsal only for a demonstrated material recovery risk | -| Shared, organisational, high-burden, or materially consequential guidance | Existing authority and consequence boundaries determine review; representedness alone does not require a stronger ceremony | -| Resolver, evaluator, promotion policy, authority, or maintenance-budget change | Treat as an ordinary change to its owning surface with evidence proportionate to its claim; current advice cannot enlarge its own authority | - -Retraction is deliberately easier than activation. An authorised instruction -not to use an item can suppress it immediately for the instruction's exact -personal, team, organisational, or public-purpose binding. Confirmed -scope, privacy, authority, wrong-effect, or stale-revision harm withdraws the -exact affected release within pre-authorised scope; weaker performance evidence -accumulates through a matched observation window. One contributor's rejection -remains contributor-, scope-, and reason-specific evidence. It becomes broader -counterevidence only after a separately authorised, independently verified -general failure claim. The -active selection can become empty; no dummy “no advice” item should be -manufactured. - -Bind version-sensitive advice to the smallest necessary dependency fingerprint: -model/provider profile, prompt or workflow revision, tool/capability schema, -relevant ontology semantics, and resolver revision. Actor/audience scope is a -trusted access and release-binding dimension, not a dependency fingerprint: a -scope change requires fresh visibility resolution or denial, not behavioural -retesting. An exact declared dependency mismatch makes the binding inapplicable -and triggers bounded retest without blocking the current work. An unrelated -version change does not invalidate stable advice, and a review date requests -evidence rather than silently erasing an otherwise valid release unless expiry -was explicit. - -One capability-specific maintenance resource envelope should bound online -retrieval, observation sampling, candidate/evaluator work, experiments, human -review, and release churn. Add separate sub-budgets or oscillation controls only -when observed behaviour warrants them. Exhaustion deduplicates, defers, or drops -low-value maintenance; it never delays the current work or disables a cheap -retraction/read-back path where the implemented release actually needs one. - -### 8.4 Human and AI teams - -Human instructions, corrections, approvals, and observed artefact edits remain -distinct from AI proposals and evaluations. AI-team feedback can contribute -evidence but cannot impersonate human approval, widen publication scope, grant -authority, or multiply one root event into consensus. - -For team-visible advice, preserve producer identity, role, audience, dissent, -and handoff context. Ask people only when a material value, authority, -publication, privacy, or coordination choice cannot be resolved from the shared -situation and evidence. Bundle consequential candidate, supporting and contrary -evidence, intended scope, and removal condition into one review rather than -soliciting feedback after every turn. Ordinary use should remain unobtrusive; -surface advice provenance to the relevant person or team only when it -materially explains a choice or uncertainty. - -## 9. Reliability Ratchet test - -This design applies the diagnostic from -[The Reliability Ratchet](https://michaelwitbrock.com/2026/08/01/the-reliability-ratchet/) -and the repository's -[source and case log](reliability_ratchet_articles_and_cases.md). - -For a material candidate or adoption decision, the proposal/evaluation should -answer three lightweight questions: - -1. What experience, discussion, failure, opportunity, or successful practice - suggests this advice, and for which concrete decision might it help? -2. What valid behaviour, current judgement, or future route might it suppress? -3. What observation would cause us to revise, retire, or delete it? - -These answers belong in an existing candidate or evaluation record. They do -not justify a new mandatory repair-theory schema for every hint. - -As of the September 2026 audit, the ratchet risks in the table below are design -hypotheses for represented knowledge, not observed knowledge-base failures. The -audited advice stores were empty. The actual accumulated evidence was in code, -tests, seed and migration compatibility, and Jira/design records surrounding an -unused capability. Controls on represented advice must therefore remain the -weakest ones earned by an active candidate and consumer; this table is not a -reason to impose a universal lifecycle or promotion ceremony in advance. The -empty stores also provide no current evidence that Von needs a general expiry, -deduplication, conflict-resolution, or retraction worker. - -| Possible ratchet mechanism | Proportionate response if observed or shown reachable | -|---|---| -| Every failure appends a lesson | Candidate state, evidence threshold, deduplication, and an explicit no-durable-lesson path | -| Context grows monotonically | Only active relevant revisions enter a fixed budget; raw history remains outside the prompt | -| Advice becomes a rule by tone | Explicit defeasible labelling and preservation of all authorised candidates | -| A diagnosis fossilises | Store observation separately, retain counterevidence, and test neighbouring causes/routes | -| Old model/tool behaviour governs a new release | Version-sensitive applicability and review triggers where evidence requires them | -| One malformed or inaccessible profile disables the task | Partial readable results plus normal no-advice fallback | -| Seeded advice returns after deletion | Ownership, tombstones, duplicate-aware reconciliation, and retraction tests | -| Tests require yesterday's route | Score the relevant personal, team, organisational, or wider-purpose outcome, prohibited effects, and recovery while allowing competent alternative paths | -| Advice duplicates prompts or workflow metadata | Prompt-only baseline and graduation/subtraction rule | -| A central optimiser collapses incomparable values | Consumer-specific evaluation and no cross-domain scalar, ranker, or reward oracle | -| The common interface accumulates domain branches | Move semantic interpretation back to its consumer or remove the shared abstraction | -| Advice confirms itself or team agents echo one trace | Root-evidence deduplication, exposure labels, independent outcomes, and counterfactual controls | -| The resolver becomes a compulsory dependency | Maintain and continuously test the direct/no-advice bypass | - -Advice can be an anti-ratchet layer because it gives a conjecture durable -identity without immediately making it executable law. It becomes another turn -of the ratchet if its accumulation is easier than its retraction, or if -“defeasible” is only a label on context that the model cannot realistically -disregard. - -## 10. Distinguishing evaluation - -### 10.1 First bounded claim - -The first evaluation should separate two narrow claims: - -1. **Advice-content claim:** at one selected decision stage in one workflow, the - chosen advice bytes improve a predeclared work-product, role, - organisational, or wider-purpose outcome, or successful-path efficiency, in - Arm B relative to Arm A. -2. **Representation claim:** Arm C preserves Arm B's immediate benefit within a - predeclared latency/context bound and materially reduces operational error, - time, or human burden during controlled revision, scope, drift, retraction, - or maintenance. It is not expected to change model behaviour merely because - the same bytes came from Vontology. - -A candidate may first be captured from a single experience or discussion. -Before using an evaluation to activate or generalise it, freeze the candidate -bytes and choose the smallest fair comparison whose plausible results could -change that decision. For a research or causal claim, partition eligible cases -into induction, calibration, and untouched held-out sets before tuning or -evaluating the candidate; for an ordinary narrow recoverable adoption, a -smaller direct and neighbouring comparison may be adequate. Predeclare the -primary outcome and any blocking harm supported by the selected surface, and -do not revise those choices after seeing the decision evidence. - -### 10.2 Comparison arms - -| Arm | Purpose | -|---|---| -| A — current path, candidate off | Current production context, including pre-existing advice-like material, held constant; only the candidate under test is omitted | -| B — simple versioned sidecar | The same targeting/applicability mapping and selected bytes supplied from one preselected prompt/profile sidecar without ontology retrieval; best simpler baseline | -| C — represented advice | Independently revisioned, visibility-scoped, applicability-resolved, bounded projection | - -Keep the applicable acting principal and visibility context, ontology snapshot, -available tools, prompt/workflow revisions, and model configuration fixed, and -retain requested, selected, and -provider-observed model identity. Isolate or reset conversations, caches, -learning state, durable effects, task artefacts, and advice pointers between -arms, using matched isolated fixtures where a state-changing case cannot be -cleanly replayed. Randomise order only within that isolation. Evaluators should -be blind to arm and internal route. Deterministic postconditions and canonical -world-state read-back outrank stylistic judgement; use independent semantic -evaluation only where the work product genuinely requires it. - -Select Arm B's surface before running the held-out comparison. Hold its -targeting/applicability result, injection point, label, ordering, selected -bytes, and context budget equal to Arm C so the experiment distinguishes -represented retrieval and lifecycle from content or presentation differences. -If that equality is impossible, add a predeclared direct-injection diagnostic -and bound the claim accordingly. - -When the representation claim includes lifecycle value, run the same -revision/retraction crossover on B and C: revise and then retire one item, -repeat its triggering and neighbouring cases, and compare steps, errors, -elapsed effort, and human burden. Verify that each exact active projection -changes, the no-candidate projection is restored, and ordinary alternatives -remain available. Any competent route may then succeed. This tests the claimed -operational value of representation rather than only its wording; omit it when -revision and retraction are not part of the claim. - -If a failure plausibly reflects model capability, compare a stronger suitable -model without advice before adding more advice machinery. Advice should not -become permanent compensation for a model or tool limitation that has ceased to -exist. - -### 10.3 Measures - -Choose one or two primary measures and the blocking harms before the held-out -run. The lists below are menus for task-specific selection and causal diagnosis, -not a conjunctive release checklist. - -Candidate primary outcome measures: - -- verified completion or useful partial completion; -- factual groundedness and false-empty/unsupported-claim rate; -- correct durable effects through canonical read-back; -- useful-action, false-refusal, abandonment, and recovery rates; -- fulfilment of material role obligations and work-product criteria; -- observable organisational or wider-purpose consequences, keeping materially - different beneficiaries, affected parties, and values separate; and -- unnecessary work, handoffs, questions, confirmations, or material human/team - corrections. - -Efficiency is measured conditional on adequate completion so fast failure -cannot win: - -- end-to-end time to verified outcome and time to first material progress; -- successful-turn latency distribution; -- model calls, tool calls, workflows, failed attempts, and retries; -- input/output tokens and cost, retaining unavailable values as unknown; and -- advice retrieval latency, projected tokens, cache state, and displaced - context. - -Advice- and ratchet-specific measures: - -- applicability precision and missed-applicable-advice rate; -- negative transfer on no-advice and conflicting-advice cases; -- train-to-holdout generalisation gap; -- stale, duplicate, conflicting, superseded, and never-selected advice; -- revision, retraction, and rollback burden; -- authoring, review, and promotion effort amortised over realistic use; -- active advice count and prompt footprint over time; and -- whether any valid candidate or recovery route became unavailable or - effectively hidden. - -### 10.4 Telemetry and causal interpretation - -Retain, where the claim needs them: - -- request, turn, experiment, actor/namespace, and task-case identities; -- prompt, workflow, tool-schema, ontology-snapshot, and advice revisions; -- candidate advice considered, retrieval/applicability reasons, selected and - omitted revisions, projection order, digest, and token count; -- advice-query latency, cache state, and truncation; -- candidate tools/workflows before projection and the path actually chosen; -- tool/workflow attempts, recovery, receipts, final answer, terminal state, and - canonical read-back; and -- evaluator rubric/version and judgement provenance. - -Telemetry can establish what was projected and what happened. It cannot by -itself prove that advice caused the model's choice. Randomised, state-isolated -ablation supports only the bounded causal claim under test; it does not by -itself establish long-term or cross-consumer architectural value. - -### 10.5 Decision rules - -- If B does not improve on A, the candidate content has not earned a runtime - path; do not build C to rescue it. -- If B improves on A but C is inferior beyond the declared online bound, keep B - or remove the candidate and repair no representation machinery. -- If C preserves B's benefit but does not reduce real lifecycle/scope/drift or - maintenance burden, the represented layer has not earned its additional cost; - keep the simpler sidecar. -- If C preserves B's benefit and materially improves a predeclared operational - maintenance outcome, retain only that bounded represented use. -- A direct-injection diagnostic can isolate lookup/projection overhead when B - and C cannot otherwise be presentation-equivalent; it does not establish - contextual-selection or lifecycle value. -- If evidence supports only the selected workflow/stage, retain only that - target and do not claim a general advice architecture. -- Do not broaden to graph-wide concept, tool, message, and task advice. Attempt - reuse only when a second materially different person, team, organisation, or - wider-purpose job independently needs it and direct prompt/profile attachment - appears inadequate. -- Stop gathering evidence when another run is unlikely to change the decision - to activate, narrow, revise, roll back, or subtract. - -## 11. Convergence and delivery sequence - -### Phase 0 — audit before adding - -- Read the live Vontology and exact runtime path. -- Count active and historical workflow-experience relations, duplicate/no- - lesson bodies, actor scopes, and prompt tokens. -- Establish whether the Vontology guidance prelude and raw selected-workflow - policy-memory projection are actually populated and consumed. -- Classify existing workflow-experience relations as legacy/unclassified until - promotion evidence and lifecycle state are established; never infer active - status from recency. -- Measure lookup and total-turn latency. -- Identify the existing canonical lifecycle and supersession primitives. -- Measure per-episode self-improvement launches, root-evidence duplication, and - maintenance work across actor/target/time windows rather than relying on - per-run caps. - -Stop before runtime A/B/C if the live projection is dormant, no candidate and -consumer decision are available, or deleting/merging obsolete paths removes the -accumulation problem. In that case subtract the unused runtime substrate and do -not create a replacement advice worker. This does not prohibit source-native, -non-active candidate capture from experience or discussion; it prohibits -claiming that an untested candidate has earned projection. - -#### Phase 0 result — 4 September 2026 - -The live Vontology projection was dormant, its additive stores were empty, and -the audited runtime substrate supplied no selected candidate or consumer. Its -four declared direct callers were proposal, evaluation, and release workflows, -where injection would undermine evidence independence. Seventeen other LLM states -still declared the same history fields without a repository producer, leaving -a direct-input compatibility aperture. The separate selected-workflow memory -builder also had no caller, although its renderer could accept caller-supplied -state. Meanwhile the automatic episode producer had accumulated 134,215 -instances and its latest bounded cohort had no successes. The stop condition -therefore fired before candidate authoring or an A/B/C campaign. - -The repository response is subtraction and containment, not a smaller advice -platform: remove the producer tail, prompt, seed vocabulary, four direct caller -steps, 17 remaining LLM-state projection apertures, and the unused -selected-workflow builder/renderer/injection path; replace the old prelude graph -with an unpublished, actionless terminal tombstone; default automatic episode -evaluation off; retain explicit critique, evidence, and normal evaluator paths; -and add structural and lifecycle read-back checks so a successful reviewed -bootstrap cannot restore the normal routed active path. -Unknown live authority drift remains a reported migration blocker rather than -being overwritten. This result does not imply that discussion- or -experience-derived learning should stop. Reopen Phase 1 when an attributed -candidate from experience or discussion and a concrete consumer make an -advice-off versus simple-sidecar comparison meaningful. - -### Phase 0.5 — retain candidate learning without activation - -The repository now implements this bounded slice. Von can retain a material -lesson from experience or discussion as a `learning_candidate.v1` Vontology -artefact when independent identity, discovery, and cross-session revision are -useful. It is an instance of the existing `#V#artifact` type rather than a new -general advice class. Its canonical body is stored as `hasDescription`; compact -`concept_data.learning_candidate` metadata records the immutable source, -contributors, semantic targets, audience, beneficiaries, purposes, visibility, -Von authorship, actual capture/revision actor or organisation, idempotency, and -revision history. - -Capture accepts a visible `episode_critique_memory.v1` source, a materialised -conversation concept, or an actor-owned chat-history session that need not -already have a Vontology conversation projection. It verifies the source before -writing and never materialises or widens the source as a side effect. Candidate -visibility may be narrowed to the acting person or, when the source already -permits it, shared with the trusted organisation. Organisation-visible -candidates may be read and revised collaboratively within that scope; exact -reviser provenance and prior revisions remain visible. Contributor, target, -audience, beneficiary, and purpose identifiers are semantic references, not -access or effect authority. - -The four Internal MCP operations make capture, discovery, read-back, and -revision available to an authenticated user or organisation context. Raw -payload identity cannot select that context. Retry identities prevent duplicate -capture and stale revision replay without attempting semantic global -deduplication. Source visibility is rechecked on read. Semantic references do -not become access gates: currently unavailable references are redacted, with -counts reported but hidden identifiers not disclosed. An interrupted -cross-store body write remains non-active, is reported as invalid in discovery, -and can be repaired by an exact retry. - -Candidate capture still does not create an active binding, alter the tools or -workflows available to a turn, or imply efficacy. It is an optional model or -human judgement: zero candidates is normal, and there is no mandatory post-turn -learning stage. A later consumer-specific evaluation may select a candidate; -until then it is inspectable, revisable learning rather than runtime advice. -Shared-invitee conversation-source resolution and a human candidate-management -view are not part of this first slice. - -### Phase 1 — isolated comparison - -JVNAUTOSCI-2720 now implements the first bounded comparison without adding an -ordinary consumer. Its fixed source is an actor-visible discussion about a -channel-neutral message request that tried only Gmail, received a typed local -failure, and overlooked an authorised Von-message capability that had worked -earlier. Candidate formation sees only the selected source messages, their -attested Turn Execution Records, and allowed semantic references—not Jira's -draft wording, cases, evaluator, rubric, or result rule. - -The body-free manifest and deterministic plan bind one exact candidate revision, -source, actor, organisation, namespace, evaluator, rubric, runtime, gateway, -model configuration, six model-held-out cases, two repeats, and 24 balanced -trials. OpenAI `gpt-5.6-luna` is fixed for acting and blinded evaluation. Arm A -withholds both bytes and presence cues; Arm B supplies the exact bytes through a -labelled optional sidecar in the same existing model call. A pre-provider -observer rejects candidate or source contamination and records only request -digests and sizes. Trial TERs retain normal actor-scoped bounded evidence needed -to audit the work product; experiment observations and operator output do not -duplicate raw message content. - -This is a stripped, read-only direct-adaptive replay with represented workflow -discovery and represented tool-result projection disabled equally in both arms. -It is not byte-equivalent to the ordinary production path. The comparison can -screen candidate-content efficacy, not represented-layer value. A B win stops -with `represented_arm_c_required`; Arm C, an ordinary consumer, and autonomous -activation remain unimplemented, and the Phase 1 projection contract rejects a -caller-constructed Arm C. A B non-win may reject the exact revision only from -independently recomputed evidence, followed by trigger and neighbouring turn -read-back proving the rejected bytes were absent. Drift, contamination, -unblindable evidence, or incomplete pairs remain inconclusive. - -#### Phase 1 interim checkpoint — 4 September 2026 - -The implementation formed and canonically read back one source-grounded, -Von-authored candidate while keeping it non-active. Its contributor, -organisation, semantic target, audience, beneficiary, and purpose remained -distinct; capture did not create an authority grant or an ordinary consumer. -The frozen comparison then exercised 24 balanced A/B trials, persisted every -trial Turn Execution Record, completed every blind evaluation, and read back -the bound manifest, plan, result, and request identities. An earlier run that -lost persistence during an Atlas DNS failure was explicitly aborted and remains -inspectable as an operational non-result. - -The first complete comparison was nevertheless inconclusive: only 2 of 12 -pairs were comparable. Eleven evaluations rejected themselves because the -model-visible input schema identified an advice experiment, while the -represented evaluator treated any experiment identity as a blinding failure; -five more lacked the exact bounded read-evidence content that the acting model -had seen. A single, predeclared harness repair replaced the treatment-labelled -input schema with a neutral one and supplied that exact bounded content only to -the evaluator. It did not change the candidate, cases, acting or evaluator -model, randomisation, rubric, decision thresholds, or stored observations, and -tests prove the private evaluator projection is not persisted. - -The one justified replacement run again completed all 24 trials, TER writes, -and evaluations with valid execution integrity. It was still inconclusive: -only 1 of 12 pairs was comparable and 18 evaluations returned -`evaluator_blinding_failed` (8 in A and 10 in B). The remaining fixed evaluator -and rubric metadata expose common study context, not treatment assignment, but -the represented evaluator's broader wording currently rejects that context. -This demonstrates evaluator self-invalidation, not observed treatment leakage. -It is not evidence that the -candidate helps, does not help, or should be activated or rejected. - -The stopping rule was applied at that checkpoint. No Arm C, ordinary consumer, -activation, rejection or prospective later-use claim followed from those runs. -The candidate remained non-active and undecided. Their records remain immutable, -rather than being rescored; JVNAUTOSCI-2720 preserves the dated protocol -registrations, run identifiers and interim delivery decision. - -#### Completed negative learning cycle — 5 September 2026 UTC - -Following the user's resumed completion request, the separately registered -evidence-projection correction produced 24 valid trials and evaluations, with -all 12 pairs comparable. B and A each passed 5 of 8 applicable trials; B did not -meet the unchanged use criteria. The canonical service recorded rejection -against the exact source-derived revision, preserving its text and provenance. -Two subsequent turns verified absence across 15 actual model requests and -canonical execution-record read-back. The -[frozen dossier](jvnautosci_2720_learning_cycle_2026-09-05.md) gives individual -outcomes, identities, repairs, costs and limitations. This establishes the -bounded negative learning loop, not a performance improvement or an active -represented advice layer. Current publication status remains in 2720. - -### Phase 2 — one bounded consumer - -The resumed JVNAUTOSCI-2720 delivery, following review of -[role convergence](role_learning_convergence.md) and JVNAUTOSCI-2721, corrects -the demonstrated evaluator contradiction under a separately versioned prompt. -Its pre-registration and live decision remain in 2720. Candidate formation, -later behavioural benefit, and longitudinal role competence are distinct -claims: 2721 consumes a fitting delivered result without expanding this -experiment's acceptance criteria. The previous inconclusive runs remain -historical evidence. - -The resumed harness preserves complete already-bounded model-visible tool -outputs, including catalogue context and structured evidence matches; a second -field whitelist had silently discarded facts needed to check grounding. The -semantic evaluator judges task outcomes while the existing harness owns model -identity, exposure and contamination checks. Four synthetic live calibration -cases distinguished grounded output, fabrication, a correctly reported external -failure and focused clarification. This is evaluator calibration, not evidence -of learned benefit. Frozen versions and subsequent findings are recorded in -2720 rather than retrospectively changing earlier results. - -Only if the comparison favours represented advice: - -- add one optional batched active-advice projection at the winning decision - point; -- add only the revision, active-selection, and read-back semantics needed by - that candidate's actual lifecycle; -- migrate or retire only a demonstrably duplicate injector so the consumer has - one advice projection, while preserving distinct episode and evidence stores; - and -- verify the normal no-advice route and those conflict, access, revision, or - retraction behaviours that the selected implementation actually exposes. - -After this local slice, cross-consumer reuse and autonomous maintenance are -independent branches. A second consumer does not prove safe autonomous release, -and useful local maintenance does not justify a common protocol. - -### Post-pilot branch A — earn reuse - -- Only when a second materially different personal, team, organisational, or - wider-purpose job independently needs advice, test whether the same minimal - query and lifecycle contract is useful; do not add a consumer merely to - validate the abstraction. -- Reify advice items only if cross-target reuse or independent governance now - provides measurable value. -- Keep specialised publication, knowledge-acquisition, tool, and outcome- - explanation profiles specialised unless consolidation makes them simpler. -- Proceed with the shared protocol only if it removes real duplicated lifecycle, - scope, projection, or telemetry work. If the second consumer needs shared - domain branches, a larger universal payload, or a common ranker, retain the - specialised implementations and stop generalising. - -### Post-pilot branch B — bounded autonomous revision - -Do not attach autonomous advice activation to the current per-episode -self-improvement launch. If the earlier phases reveal a worthwhile recurring -maintenance job, select one bounded campaign containing only the failure modes -that could change its release decision. Candidate probes include: - -- a signal/lineage fixture with duplicate roots, weak critique, untrusted - content, independent evidence, and an allowed no-durable-lesson result; -- a shadow A/B/C promotion, revision, and retract-to-empty crossover; and -- a drift/scope/budget sentinel covering the dependencies and actor boundary - material to that candidate. - -Scope leakage, self-confirming efficacy evidence, or failure to retract a first -release to empty becomes a stop-ship condition only when the implemented path -contains the corresponding mechanism and a material consequence is reachable. -Other probes are selected proportionately. When the claim includes autonomous generation, the -frozen generator output—including `no durable lesson`, revision, or retraction— -must feed the shadow release test; substituting a curated good candidate tests -release mechanics only. - -Run a live canary for one low-consequence personal, team, organisational, or -public-purpose binding only if a real job and the selected campaign support it -under standing release authority. Use existing -operational-learning mechanics where their exact namespace/user/organisation -scope and evidence model fit; add a lighter retract-to-empty primitive rather -than pretending failure-only registration and previous-release rollback are -already generic. Do not auto-activate from one episode, evaluator, or aggregate -score. - -### Role, organisation, and wider-purpose learning - -Learning for a role, organisation, or wider purpose is part of the current -architectural scope, even though no role-linked runtime advice consumer is yet -selected. It need not begin with a user's dissatisfaction or a recurring -failure. Successful practice, a handoff, a discussion, an unrealised -opportunity, or evidence about a work product may justify retaining and later -discovering a candidate. Some useful roles have no immediate user at all. - -Candidate and outcome records should distinguish the contributor, acting -principal, target role, audience, beneficiary, affected parties, and purpose -whenever collapsing them would change the judgement. A role may serve a team or -organisation; an organisation or social group may pursue a broader social, -human, or ecological purpose. Those links provide context for relevance and -evaluation. They do not grant authority, prove that the represented purpose is -good, erase disagreement, or justify reducing plural outcomes to one score. - -A later runtime use is earned by an actual decision point and comparison with -the simpler role prompt, workflow, playbook, or conversation-situation -baseline. It may help Von retrieve before interrupting people, choose useful -work products and handoffs, recover intelligently, and preserve practice across -membership changes. Stable practice can graduate into the appropriate role -prompt, workflow, playbook, validator, or typed policy, retiring the duplicate -advice. Role-linked advice cannot assign responsibilities, infer permission, or -replace the relevant authority and commitment model. - -A SAIL research-operations role is an illustrative candidate from Von's -programme direction, not selected runtime scope in this design. Any pilot keeps -personal, team, organisation, and publicly visible bindings distinct and -preserves human and AI contribution, dissent, and evidence provenance. - -This sequence does not begin by rewriting all prompts, routing profiles, -planner hints, or workflows into a common advice ontology. - -## 12. Acceptance, stop-ship, and removal conditions - -This section is a menu tied to the eventual claim and implemented surface, not -a universal preflight checklist. A plausible but undemonstrated risk belongs in -measurement or an open question; it becomes a release gate only when evidence -or a clear causal demonstration connects it to a material outcome in the -selected consumer. Adding every conceivable safeguard in advance would itself -be the most direct reliability ratchet. - -### 12.1 First consumer and autonomous canary - -For a first runtime use, Arm B must establish a likely net improvement over the -current Arm A route on the selected outcome, including material latency and -context cost, with no observed material regression on the smallest relevant -neighbouring case. That is enough to decide whether the *content* merits bounded -use; it does not require a represented layer. - -Arm C is required only when claiming value for represented advice. Retain it -only if it preserves B's benefit and improves a named operational outcome—such -as time, error rate, or human burden during a revision or scope change—over the -equally versioned sidecar. Demonstrating that lifecycle operations merely -function is not by itself an improvement. - -Test further boundaries only when they exist on the selected path or evidence -shows a material reachable failure. Examples include actor isolation for -private advice, conflict behaviour when conflicting active items actually can -co-occur, and exact prior/no-advice read-back when revision or retraction is part -of the claim. Do not instantiate these mechanisms merely to satisfy this list. - -For autonomous activation, prove the bounded outcome and standing authority for -the exact effect. Add independence, churn, emergency withdrawal, or resource- -budget tests only where the proposed autonomous path introduces those concrete -dependencies or failure modes. - -For JVNAUTOSCI-2720 specifically, a B non-win blocks Arm C and may support a -scientifically useful negative learning result only when canonical disposition -and later no-exposure read-back agree. A B win is permission to test represented -Arm C and prospective use, not permission to activate. An incomplete, -contaminated, drifted, or unblindable run supports neither conclusion. - -### 12.2 Stop-ship conditions - -Stop ship for the affected consumer when observed evidence or the implemented -causal path shows a material regression: for example, advice causes an authority -violation, wrong durable effect, cross-scope disclosure, suppression of explicit -user intent, disappearance of a valid recovery, material context growth, or -maintenance that blocks the ordinary task. Stale retracted projection, -self-confirming promotion evidence, candidate/evaluator contamination, -scope-widening feedback, promotion/retraction churn, or failed exact retraction -join this list only when the selected implementation actually contains those -paths and the consequence is material. - -If retraction fails, do not mark the release retired. Disable projection through -the consumer's no-advice bypass, invalidate derived caches, verify absence from -the model-visible context, report stop ship, and reconcile canonical state. -Mark retirement only after canonical read-back confirms removal. - -### 12.3 Removal at three levels - -Retire an **advice release** immediately when its scope is wrong, a declared -dependency invalidates it, it contradicts newer trusted evidence, it suppresses -a valid route, or its independently verified outcome is materially harmful. -Retire it after its observation window when it makes no beneficial decision -change, current model/tool behaviour dominates it, or it duplicates an existing -prompt or workflow profile. - -Remove a **consumer integration** when: - -- advice-off or the specialised native surface is non-inferior and cheaper; -- applicable advice is too rare to amortise lookup and maintenance; -- its adapter requires consumer-specific branches in shared code; or -- maintenance cost exceeds the burden or failed work it avoids. - -Collapse or remove the **shared protocol/layer** when: - -- fewer than two independently justified consumers remain; -- reuse amounts only to renaming fields into a common transfer object; -- shared changes repeatedly require domain-specific conditionals or a central - ranker; -- retrieval and maintenance cost exceed the duplicated work removed; -- the direct/no-advice routes stop receiving equivalent maintenance; or -- autonomous promotion remains in the layer but cannot satisfy the evidence or - recovery semantics actually claimed for it. - -Persistently low candidate yield is evidence that the maintenance worker should -run less often or disappear, not a reason to relax promotion quality. - -### 12.4 Subtraction order - -When removing advice machinery: - -1. deactivate the affected releases and read back the exact empty or prior - active selection; -2. restore and verify the consumer's direct/no-advice path; -3. retain useful content in the simpler prompt/profile only when evidence - supports it; -4. remove unused injector, adapter, cache, resolver, maintenance and scheduled - trigger paths, derived manifests, configuration, seed registrations, and - materialisation hooks; -5. preserve outcome/authority tests and raw evidence for audit without prompt - projection, while removing route-specific tests that fossilise the deleted - mechanism; and -6. remove unused common vocabulary only after confirming that no remaining - consumer depends on it. - -Raw episodes and evaluation evidence may remain for audit after active advice -is removed. They must not continue to shape prompts through an undeclared -fallback. - -## 13. Open questions - -A future pilot should resolve, rather than assume: - -1. Is an attached text relation sufficient, or does independent advice identity - materially improve revision and reuse? -2. Which remaining specialised advice-like projections are active and useful - for a current personal, team, organisational, or wider-purpose job, beyond - the now-subtracted dormant workflow-experience route? -3. Can the existing workflow capability projection carry advice cheaply enough, - or is a separate visibility-aware query needed? -4. How much applicability should be typed versus left to model judgement? -5. When may a trusted release process automatically promote low-consequence - personal, team, organisational, or public-purpose advice, and what evidence - or review is actually needed for the selected consequence and audience? -6. Does role advice belong on role/responsibility concepts, or is it usually - better expressed in the conversation situation and workflow metadata? -7. At what point should a recurring advice item graduate into a prompt, - workflow, or independently justified invariant? -8. Does a shared projection protocol remove measurable lifecycle and scope work - for a second consumer without acquiring domain branches or a common ranker? -9. Which scoped maintenance effects have standing release authority, and how - should the relevant people or team inspect or revise that delegation without - per-item review? -10. Does role-linked advice reduce team burden and improve work products, or - merely make Von's existing habits more persistent? - -## 14. Current handoff decision - -The architectural decision is **retain federated convergence of soft guidance -as a falsifiable hypothesis**. Phase 0 subtracted the dormant -workflow-experience route; Phase 0.5 retained its useful evidence substrate and -added source-grounded, explicitly non-active candidate capture, inspection, and -revision. Phase 1 now has an isolated, task-specific A/B implementation under -JVNAUTOSCI-2720. It can form one Von-authored candidate from the frozen source, -project the exact bytes only in experimental Arm B, persist and blindly evaluate -24 read-only trials, recompute the result, update the exact revision's -disposition when warranted, and read the terminal state back. Live execution -proved source-derived formation, exact experimental exposure, outcome -evaluation, evidence-derived rejection and subsequent verified absence. The -[completed dossier](jvnautosci_2720_learning_cycle_2026-09-05.md) records the -negative result and preserves earlier inconclusive runs. The candidate remains -non-active, with its text retained and use disposition rejected. No positive -advice efficacy or broader role competence is claimed. - -No ordinary configured advice consumer, active binding, resolver, represented -Arm C, maintenance worker, or general advice layer is selected. Workflow Studio -remains a human inspection consumer for retained structured suggestions, and -the MCP surface remains a human/AI deliberation and maintenance consumer for -non-active candidates. The experiment-only sidecar does not turn either surface -into automatic production projection. - -The exercised candidate did not earn Arm C. Later work can inspect or revise -the retained lesson in light of new experience without treating this bounded -negative result as a universal prohibition. JVNAUTOSCI-2721 owns the separate -longitudinal role pilot and reuses fitting delivered surfaces. A future positive -content comparison would still require represented Arm C and prospective later -use before claiming represented retrieval or production value. Represented C -must preserve the measured content -benefit and improve a named maintenance, scope, provenance, inspection, -revision, or retract-to-empty outcome over the same versioned sidecar. A shared -protocol still requires a second independently motivated consumer and measured -removal of duplicated lifecycle or scope work. Autonomous activation remains a -later, separately earned decision. - -Related guidance: - -- [Prompt programs and model routing](prompt_programs_and_model_routing_playbook.md) -- [Agent memory and enduring knowledge](agent_memory_and_enduring_knowledge.md) -- [Agent evaluation and research uptake](agent_evaluation_and_research_uptake.md) -- [Minimal imposition](minimal_imposition_design_principle.md) -- [Security considerations](security_considerations.md) -- [Testing workflows and ephemeral theories](testing_workflows_ephemeral_theories_design.md) -- [Ontology publication authority](ontology_publication_authority.md) -- [Reliability Ratchet source and cases](reliability_ratchet_articles_and_cases.md) diff --git a/docs/engineering/role_convergence_rehearsal.md b/docs/engineering/role_convergence_rehearsal.md index e0a6aeb35..bb2beaf38 100644 --- a/docs/engineering/role_convergence_rehearsal.md +++ b/docs/engineering/role_convergence_rehearsal.md @@ -177,7 +177,7 @@ selected path. Record unresolved programme work in the existing Jira plan. ## 6. Executable A–C development path -The [2721 continuity record](jvnautosci_2721_continuity_2026-09-05.md) +The [2721 continuity record](https://github.com/Strong-AI-Lab/Von/blob/5b2f8b976069935da73b82bca433fb3cefffcd3d/docs/engineering/jvnautosci_2721_continuity_2026-09-05.md) describes an exercised bounded implementation using the existing lab/project digest workflow. Reproduce the isolated rehearsal with: diff --git a/docs/engineering/role_convergence_test_programme.md b/docs/engineering/role_convergence_test_programme.md index ca8544bd3..af6d5731a 100644 --- a/docs/engineering/role_convergence_test_programme.md +++ b/docs/engineering/role_convergence_test_programme.md @@ -21,13 +21,13 @@ brief is a revisable projection, not a second task ledger. The immediate claim is bounded operational usefulness. It is not full lab management, scientific certification or evidence that represented architecture outperforms a simpler agent. The fictional A–C work in -[PR 552's evidence record](jvnautosci_2721_continuity_2026-09-05.md) supplies a +[PR 552's evidence record](https://github.com/Strong-AI-Lab/Von/blob/5b2f8b976069935da73b82bca433fb3cefffcd3d/docs/engineering/jvnautosci_2721_continuity_2026-09-05.md) supplies a development baseline and known failures, not prospective adoption. -The [first deployment/witness record](jvnautosci_2721_replay_programme_2026-09-05.md) +The [first deployment/witness record](https://github.com/Strong-AI-Lab/Von/blob/5b2f8b976069935da73b82bca433fb3cefffcd3d/docs/engineering/jvnautosci_2721_replay_programme_2026-09-05.md) contains the initial observations and their limitations. Current repair selection remains in Jira. -The [prospective readiness record](jvnautosci_2721_prospective_readiness_2026-09-05.md) +The [prospective readiness record](https://github.com/Strong-AI-Lab/Von/blob/5b2f8b976069935da73b82bca433fb3cefffcd3d/docs/engineering/jvnautosci_2721_prospective_readiness_2026-09-05.md) retains the real first encounters and subsequent replay observations. ### Selected operational tranche @@ -119,7 +119,7 @@ Record the previous runtime, selected commit and resulting state for recovery. Never force overwrite divergent live prompt/workflow authority merely to obtain a green preflight. Inspect its meaning and choose the bounded migration. -2722's [owner-read evidence](jira_authorised_read_replay_2026-09-05.md) establishes +2722's [owner-read evidence](https://github.com/Strong-AI-Lab/Von/blob/5b2f8b976069935da73b82bca433fb3cefffcd3d/docs/engineering/jira_authorised_read_replay_2026-09-05.md) establishes the bounded direct-read path. It does not pass the four-turn family. Run that family after the initial brief and unchanged encounter, using the existing prompt bank and evaluator. The normal authenticated account owner must acquire diff --git a/docs/engineering/role_learning_convergence.md b/docs/engineering/role_learning_convergence.md index ed5db7247..ce1019102 100644 --- a/docs/engineering/role_learning_convergence.md +++ b/docs/engineering/role_learning_convergence.md @@ -12,7 +12,7 @@ - **Review trigger:** A pilot cycle finishes, a material premise changes, transfer is attempted, or programme work stops producing role evidence - **Supersedes:** The planning recommendations of the - [April role-learning review](jvnautosci_2011_role_learning_review_2026-04-24.md). + [April role-learning review](https://github.com/Strong-AI-Lab/Von/blob/5b2f8b976069935da73b82bca433fb3cefffcd3d/docs/engineering/jvnautosci_2011_role_learning_review_2026-04-24.md). Its historical evidence remains intact. ## 1. Direction and meaning of convergence @@ -143,9 +143,10 @@ that none exists. Missing expected events can themselves warrant attention. ## 6. Learning must close through later use -The [represented-advice design](represented_advice_design.md) owns candidate -semantics, applicability and lifecycle. Use it rather than building a second -learning store. Learn from successful practice, demonstrations, discussion, +The historical [represented-advice proposal](https://github.com/Strong-AI-Lab/Von/blob/5b2f8b976069935da73b82bca433fb3cefffcd3d/docs/engineering/represented_advice_design.md) records candidate +semantics, applicability and lifecycle as advisory background. Use the existing +candidate store and the current task decision rather than creating a second +learning store. The private working proposal is not required public guidance. Learn from successful practice, demonstrations, discussion, corrections, failures and delayed outcomes. Preserve who contributed what and when; later clarification cannot become evidence of earlier autonomous skill. diff --git a/docs/engineering/structured_tool_calling_guide.md b/docs/engineering/structured_tool_calling_guide.md index de5f7c6cb..db74612d6 100644 --- a/docs/engineering/structured_tool_calling_guide.md +++ b/docs/engineering/structured_tool_calling_guide.md @@ -440,7 +440,7 @@ response = client.generate_with_tools_sync( - **JVNAUTOSCI-799**: Structured tool calling task - **JVNAUTOSCI-803**: LLM Workflows (depends on this module) - **JVNAUTOSCI-698**: JSON action output prevention (original issue) -- **Phase 0 Audit**: `docs/engineering/jvnautosci_799_phase0_audit.md` +- **Historical Phase 0 Audit**: [frozen evidence](https://github.com/Strong-AI-Lab/Von/blob/5b2f8b976069935da73b82bca433fb3cefffcd3d/docs/engineering/jvnautosci_799_phase0_audit.md) ## File Structure @@ -459,9 +459,6 @@ src/backend/languagemodels/structured_tool_calling/ tests/backend/ ├── test_structured_tool_calling_types.py └── test_structured_tool_calling_client.py - -docs/engineering/ -└── jvnautosci_799_phase0_audit.md # Current state audit ``` ## Authors & Maintainers diff --git a/docs/engineering/testing_workflows_ephemeral_theories_design.md b/docs/engineering/testing_workflows_ephemeral_theories_design.md deleted file mode 100644 index f501d4415..000000000 --- a/docs/engineering/testing_workflows_ephemeral_theories_design.md +++ /dev/null @@ -1,672 +0,0 @@ -# Testing Workflows and Ephemeral Theories - -Status: Design only -Date: 2026-03-18 -Audience: Human engineers and AI agents -Related epics/tasks: `JVNAUTOSCI-833`, `JVNAUTOSCI-964`, `JVNAUTOSCI-936`, `JVNAUTOSCI-1517` - -## 1. Purpose - -This document designs a workflow-first framework for "Testing Workflows" in Von. - -The intent is to let Von: - -1. create bounded, non-authoritative experimental theories inside or alongside Vontology, -2. run workflows, tools, learning loops, and capability checks against those theories, -3. record structured evidence about what happened, -4. safely promote only validated outcomes into canonical Vontology when appropriate. - -This is explicitly a design document. It does not propose implementation in this task. - -## 2. Executive Summary - -Von already has most of the runtime needed for a Testing Workflows framework: - -1. durable VWL workflows, -2. structured execution evidence and completion gates, -3. workflow-gap recovery and candidate testing, -4. isolated benchmark execution using cloned databases, -5. some support for transient microtheory-like payloads. - -The main missing layer is not more workflow engine. It is a first-class theory and experiment representation with a safe mutation boundary. - -The clean design is a two-tier system: - -1. `Ephemeral theory` tier: lightweight, bounded, non-canonical theory slices for rapid experiments, workflow checks, prompt/tool comparisons, and safe reasoning tests. -2. `Isolated benchmark world` tier: heavyweight DB-clone execution for destructive or system-wide experiments where theory-local isolation is not enough. - -The critical architectural rule is that Testing Workflows themselves should be VWL/Vontology artefacts. Python should provide only reusable theory-slice, diff, rollback, promotion, and telemetry primitives. - -## 3. Why This Matters - -Von already wants to be workflow-first, ontology-backed, and capable of learning from its own execution. Testing Workflows would make that explicit and safer. - -Potential outcomes: - -1. workflow regression testing without contaminating canonical knowledge, -2. tool and prompt benchmarking grounded in real execution traces, -3. bounded self-experimentation on workflow choice, plan structure, and repair strategies, -4. reusable experiment records for measurement and later learning, -5. a principled path from "candidate idea" to "validated behaviour" to "published workflow or promoted knowledge". - -This may be novel in the particular combination of: - -1. ontology-backed theory representation, -2. durable workflow execution, -3. explicit promotion gates, -4. self-experimentation bounded by namespace and theory isolation. - -The literature scan in this task found adjacent work, but not an exact match for that combination. - -## 4. Existing Bases in the Repo - -### 4.1 Workflow runtime - -Von already has the core workflow runtime and durable execution model: - -1. [von_workflow_language_manual.md](von_workflow_language_manual.md) -2. [durable_workflow_system_design.md](durable_workflow_system_design.md) -3. [models.py](../../src/backend/workflows/durable/models.py) -4. [plan_state_runtime.py](../../src/backend/workflows/plan_state_runtime.py) - -Relevant consequences: - -1. durable instances already exist, -2. checkpoints and resumes already exist, -3. subworkflows and control-flow actions already exist, -4. workflow publication and runnability gates already exist. - -### 4.2 Transient microtheory substrate - -There is already a notion of `transient_microtheory`, but only as a payload shape for renderer applicability: - -1. [renderer_applicability_service.py](../../src/backend/services/renderer_applicability_service.py) -2. [orchestrator.py](../../src/backend/integrations/internal_mcp/orchestrator.py) -3. [renderer_applicability_vontology_service.py](../../src/backend/services/renderer_applicability_vontology_service.py) - -This is important because it shows the system already accepts an object kind that is neither a canonical concept nor a free-text blob. It is a strong seed for a more general ephemeral-theory design. - -### 4.3 Structured execution evidence - -Von already records structured execution evidence and benchmarkable failure signals: - -1. [turn_execution_completion_schema_proposal.md](turn_execution_completion_schema_proposal.md) -2. [turn_execution_record_service.py](../../src/backend/services/turn_execution_record_service.py) -3. `turn_execution_*` MCP tools in [catalogue.py](../../src/backend/integrations/internal_mcp/catalogue.py) - -This means Testing Workflows do not need to invent result evidence from scratch. They should extend this discipline to theory experiments. - -### 4.4 Existing benchmark harness - -Von already has a guarded end-to-end benchmark harness: - -1. [kb_clone_benchmark_harness.md](kb_clone_benchmark_harness.md) -2. [kb_clone_benchmark_service.py](../../src/backend/services/kb_clone_benchmark_service.py) - -This is already suitable for destructive integration tests and system-wide experiments. It is too heavy for rapid internal experimentation, but it should become Tier 2 of the Testing Workflows design rather than a parallel subsystem. - -### 4.5 Workflow-gap candidate testing - -Von already has the pattern "identify a workflow gap, create a candidate, optionally test it, then decide whether to use it": - -1. [workflow_gap_recovery_workflow.py](../../src/backend/workflows/durable/workflow_gap_recovery_workflow.py) -2. [workflow_gap_workflow_contracts.py](../../src/backend/workflows/workflow_gap_workflow_contracts.py) -3. [workflow_creation_contracts.py](../../src/backend/workflows/workflow_creation_contracts.py) - -That is already very close in spirit to Testing Workflows. - -### 4.6 Learning and episode signals - -Von already has learning-oriented workflow selection and episode history machinery: - -1. [workflow_selection_experience.py](../../src/backend/services/workflow_selection_experience.py) -2. [workflow_selection_policy_service.py](../../src/backend/services/workflow_selection_policy_service.py) -3. [workflow_episode_service.py](../../src/backend/services/workflow_episode_service.py) - -Testing Workflows should consume and emit signals compatible with those services. - -## 5. What Is Missing - -### 5.1 First-class theory representation - -Current `transient_microtheory` support is payload-only. It is not yet a first-class Vontology theory model with canonical types, predicates, inclusion, or promotion semantics. - -Missing capabilities: - -1. canonical type for ephemeral or testing theories, -2. canonical representation for assumptions, hypotheses, assertions, theorems, and observations inside a theory, -3. explicit provenance, expiry, and promotion metadata, -4. a standard way to compare theory-local state with canonical state. - -### 5.2 First-class experiment representation - -Von has turn records and benchmark manifests, but not a general `experiment_run` model for theory testing. - -Missing capabilities: - -1. experiment specification artefact, -2. experiment run record, -3. variable and control representation, -4. expected versus observed outcomes, -5. formal pass/fail/partial/inconclusive result contract, -6. experiment suite aggregation. - -### 5.3 Theory-safe mutation boundary - -The safety model is not complete enough yet for theory inclusion and self-experimentation. - -Relevant evidence: - -1. [security_considerations.md](security_considerations.md) explicitly says namespace is intended to interact with future microtheory and theory-inclusion work. -2. [effective_namespace_contract.md](effective_namespace_contract.md) already defines one authoritative namespace-resolution path, but not theory inclusion semantics. - -Missing capabilities: - -1. theory inclusion and exclusion rules bound to namespace, -2. theory-local write isolation, -3. explicit promotion from theory-local to canonical ontology, -4. rollback or garbage collection of ephemeral theories. - -### 5.4 Theory operation control surface - -There is no reusable action surface for theory operations such as: - -1. create ephemeral theory, -2. clone a theory slice from canonical context, -3. assert or retract theory-local claims, -4. compute canonical-versus-theory diff, -5. execute a workflow against a specified theory, -6. promote validated results, -7. discard or expire theory-local state. - -### 5.5 Lightweight, frequent testing harness - -The DB-clone harness is too expensive for rapid, repeated self-experiments. A theory-slice harness is needed between "full clone" and "no isolation at all". - -### 5.6 Introspection/tooling reliability - -During this design task, `workflow_list_definitions` and `workflow_list_event_bindings` timed out through the Vontology MCP surface. This is not the main design blocker, but it is a practical tooling gap for agents trying to inspect the workflow/testing surface reliably. - -## 6. Design Principles - -1. Workflow-first: Testing Workflows themselves should be authored in VWL/Vontology, not encoded as bespoke Python orchestration. -2. Canonical protection: testing writes must not silently leak into canonical Vontology. -3. Explicit promotion: promotion is a separate gated workflow, never an incidental side effect. -4. Evidence first: every run produces structured evidence, not only narrative conclusions. -5. Fail closed: missing prompts, theory slices, or workflow contracts block the experiment rather than triggering silent fallback. -6. Bounded lifetime: ephemeral theories must expire or be garbage-collected unless deliberately retained. -7. Reproducibility: experiment specs and theory inputs must be sufficient to replay a run. -8. Safety over convenience: self-experimentation may tune prompts, workflow choice, or candidate graphs, but should not directly self-patch authoritative behaviour without explicit promotion gates. - -## 7. Proposed Core Model - -## 7.1 Two-tier testing architecture - -### Tier 1: Ephemeral theory tests - -Use for: - -1. prompt comparison, -2. tool-plan comparison, -3. workflow-routing comparison, -4. theory-local reasoning checks, -5. safe provisional assertions, -6. candidate workflow or model-policy evaluation. - -Properties: - -1. cheap, -2. fast, -3. theory-local, -4. bounded lifetime, -5. not authoritative. - -### Tier 2: Isolated benchmark worlds - -Use for: - -1. destructive workflow tests, -2. ingestion and indexing workflows, -3. multi-turn and multi-workflow integration tests, -4. experiments that require real persistence semantics across many collections. - -Properties: - -1. heavy, -2. slower, -3. DB-clone based, -4. stronger isolation, -5. suited for final verification and regression suites. - -## 7.2 New artefact families - -The framework should add four main artefact families. - -### A. Testing theory - -Purpose: non-canonical, bounded theory slice for experimentation. - -Suggested concept IDs and types: - -1. `#V#testing_theory` as a subtype of a future `#V#theory` or temporary first-class type, -2. `#V#ephemeral_theory` as a subtype of `#V#testing_theory`, -3. `#V#canonical_promotion_candidate` for validated theory outputs. - -Core fields or relations: - -1. source namespace, -2. included canonical concepts, -3. theory-local assertions, -4. assumptions, -5. hypotheses, -6. observations, -7. expiry timestamp, -8. provenance, -9. promotion policy, -10. experiment suite linkage. - -### B. Experiment specification - -Purpose: declarative definition of what is being tested and how success is measured. - -Suggested concept or schema family: - -1. `#V#experiment_spec` -2. `#V#testing_workflow_spec` -3. `#V#capability_test_spec` - -Key contents: - -1. target capability or workflow, -2. isolation mode (`theory_slice` or `clone_db`), -3. fixtures and inputs, -4. required tools and workflows, -5. expected outcomes, -6. evaluation rubric, -7. allowable side effects, -8. promotion eligibility rules. - -### C. Experiment run - -Purpose: durable record of what actually happened. - -Suggested concept or schema family: - -1. `experiment_run.v1` -2. `experiment_suite_run.v1` - -Key contents: - -1. run ID, -2. linked experiment spec, -3. linked theory or clone environment, -4. workflow execution evidence, -5. tool invocation evidence, -6. expected versus observed diffs, -7. safety and policy decisions, -8. final verdict, -9. candidate learning signals, -10. promotion recommendations. - -### D. Testing workflow - -Purpose: VWL workflow that consumes an experiment spec plus a theory or clone environment and produces an experiment run. - -Suggested families: - -1. `#V#testing_workflow` -2. `#V#theory_slice_test_workflow` -3. `#V#clone_benchmark_test_workflow` -4. `#V#promotion_gate_workflow` - -## 8. Suggested Ontology Additions - -The exact ontology naming should be decided carefully, but the design likely needs types and predicates in this general shape. - -### 8.1 Candidate types - -1. `#V#theory` -2. `#V#testing_theory` -3. `#V#ephemeral_theory` -4. `#V#experiment_spec` -5. `#V#experiment_run` -6. `#V#experiment_suite` -7. `#V#experiment_observation` -8. `#V#promotion_decision` - -### 8.2 Candidate predicates - -1. `#V#includes_theory` -2. `#V#derived_from_canonical_concept` -3. `#V#has_assumption` -4. `#V#has_hypothesis` -5. `#V#has_observation` -6. `#V#tests_capability` -7. `#V#tests_workflow` -8. `#V#uses_isolation_mode` -9. `#V#has_expected_outcome` -10. `#V#has_observed_outcome` -11. `#V#has_experiment_verdict` -12. `#V#promotion_requires` -13. `#V#promotes_to_canonical_relation` -14. `#V#expires_at` -15. `#V#belongs_to_experiment_suite` - -### 8.3 Assertion model choice - -One unresolved question is the canonical unit of theory content: - -1. free-text claims, -2. binary or n-ary assertions, -3. uncertain relationship assertions, -4. a mixed model. - -Current recommendation: - -1. use structured assertions wherever possible, -2. allow attached free text for rationale and notes, -3. reuse uncertain-relationship style provenance fields when the theory contains provisional relation claims. - -Relevant existing substrates: - -1. [uncertain_relationship_service.py](../../src/backend/services/uncertain_relationship_service.py) -2. [relation_elicitation_service.py](../../src/backend/services/relation_elicitation_service.py) - -## 9. Workflow Architecture - -The cleanest flow is: - -1. create or resolve experiment spec, -2. create theory slice or clone world, -3. materialise fixtures and candidate assertions, -4. execute the target workflow or tool plan, -5. capture structured evidence, -6. compare expected versus observed outcomes, -7. decide `pass`, `fail`, `partial`, or `inconclusive`, -8. optionally run a promotion gate, -9. either discard the theory, retain it for debugging, or promote validated artefacts. - -### 9.1 Recommended workflow families - -1. `theory_slice_setup_workflow` -2. `capability_test_execution_workflow` -3. `workflow_regression_test_workflow` -4. `tool_contract_test_workflow` -5. `learning_signal_harvest_workflow` -6. `promotion_gate_workflow` -7. `ephemeral_theory_gc_workflow` - -### 9.2 Recommended reusable actions - -Python should provide only generic actions such as: - -1. `theory.create_slice` -2. `theory.import_canonical_context` -3. `theory.assert_local_claim` -4. `theory.compute_diff` -5. `theory.rollback_local_writes` -6. `theory.promote_validated_claims` -7. `experiment.record_observation` -8. `experiment.compute_verdict` -9. `experiment.emit_learning_signal` - -These should be reusable control surfaces, not task-specific orchestration. - -## 10. Safety Model - -## 10.1 Canonical Vontology protection - -Testing Workflows must default to writing only into: - -1. theory-local state, or -2. isolated benchmark clones. - -Canonical ontology writes should occur only in a separate promotion workflow. - -## 10.2 Namespace and theory inclusion - -Namespace must become the main access-safety primitive for theory inclusion and promotion. This is already anticipated in [security_considerations.md](security_considerations.md). - -Required policy: - -1. theory inherits namespace and organisation scope, -2. theory inclusion cannot widen visibility beyond its namespace policy, -3. promotion into canonical space requires explicit policy and audit trail, -4. cross-namespace theory inclusion is disallowed unless a future access-control policy explicitly allows it. - -## 10.3 Expiry and garbage collection - -Ephemeral theories should carry TTL semantics. - -Recommended outcomes: - -1. discard by default after successful non-promoting tests, -2. retain on failure when debugging is requested or policy requires it, -3. retain promoted theories only as auditable history or condensed provenance, not as active hidden memory. - -## 10.4 What self-experimentation is allowed - -Allowed targets should likely include: - -1. workflow selection policy, -2. prompt variants, -3. tool sequencing, -4. candidate workflow graphs, -5. reasoning strategies inside isolated theory space. - -Not allowed without stronger gates: - -1. direct self-editing of authoritative runtime code, -2. direct publication of new canonical workflows without validation, -3. theory-to-canonical promotion without explicit promotion workflow success. - -## 11. Integration With Current Systems - -### 11.1 Turn execution records - -Testing Workflows should reuse the same evidence discipline as turn execution: - -1. candidate set, -2. selected workflow, -3. required effects, -4. tool history, -5. postcondition checks, -6. completion or test verdict. - -The natural extension is an `experiment_run.v1` schema parallel to `turn_execution_record.v1`. - -### 11.2 Benchmark harness - -The current KB-clone harness should become the Tier 2 runner, not a separate evaluation universe. - -### 11.3 Workflow-gap recovery - -Testing Workflows should absorb and generalise the existing gap-recovery candidate test pattern rather than duplicating it. - -### 11.4 Learning loops - -Experiment runs should emit structured learning signals consumable by the workflow-selection experience and policy services. - -## 12. Missing Bases and Related Jira - -The following pending Jira work appears directly relevant. - -### 12.1 Workflow-first convergence - -1. `JVNAUTOSCI-1527`: rewrite planning, rumination, parent-specificity, and gap-recovery workflows as Vontology-authored definitions. -2. `JVNAUTOSCI-1528`: rewrite support and maintenance durable workflows as Vontology-authored definitions. - -Why relevant: - -1. Testing Workflows should reuse planning, rumination, and gap-recovery patterns. -2. Introducing Testing Workflows before these migrate risks inheriting hybrid Python-defined orchestration. - -### 12.2 Evaluation and measurement - -1. `JVNAUTOSCI-964`: Von Experimental Evaluation and Measurement. -2. `JVNAUTOSCI-1274`: define benchmark-driven evaluation framework for VQL reasoning. -3. `JVNAUTOSCI-1057`: episode log analysis for tool learning feedback loops. - -Why relevant: - -1. experiment-run metrics belong under the evaluation epic, -2. theory testing should reuse the benchmark-driven evaluation mindset, -3. test outputs should feed learning loops rather than ending as isolated reports. - -### 12.3 Theory and access-control foundations - -1. `JVNAUTOSCI-296`: implement concept-level access control with `#V#accessible_to` relations. -2. `JVNAUTOSCI-254`: Vontology-first context bundles and reconstructed dossiers for concepts and workflows. -3. `JVNAUTOSCI-1266`: specify context semantics, entailment profiles, and capability negotiation. -4. `JVNAUTOSCI-343`: develop definitions and relationships for theory, claim, assertion, and hypothesis. - -Why relevant: - -1. ephemeral theories need safe visibility boundaries, -2. theory slices need richer context semantics, -3. testing theories need a stronger ontology for claims, assertions, and hypotheses than is currently obvious from concept search, -4. context bundles may become the natural input form for theory slices. - -## 13. Recommended Delivery Sequence - -This is the recommended incremental path. - -### Phase 1: Represent theory and experiments - -1. Define ontology vocabulary for theory, experiment spec, and experiment run. -2. Decide whether ephemeral theories are persisted TTL concepts, separate theory slices, or both. -3. Publish the schema and safety contract. - -### Phase 2: Build generic theory actions - -1. create theory slice, -2. assert local claim, -3. compare observed versus expected state, -4. compute diff, -5. promote validated output, -6. discard or expire theory. - -### Phase 3: Materialise first testing workflows - -1. workflow regression test, -2. tool contract test, -3. prompt comparison test, -4. candidate workflow publication test. - -### Phase 4: Connect learning loops - -1. feed experiment signals into workflow selection experience, -2. add experiment dashboards and summaries, -3. add policy-controlled promotion suggestions. - -### Phase 5: Expand to bounded self-experimentation - -1. permit workflow-choice experiments, -2. permit candidate workflow graph experiments, -3. keep publication and promotion behind explicit validation gates. - -## 14. Hard Open Questions - -These are real design questions, not placeholders. - -1. Should ephemeral theories live as transient payloads, TTL concepts in main Vontology, or a separate theory-slice store? -2. How should theory inclusion compose with namespace and future access control? -3. What is the canonical unit of assertion inside a theory: free text, predicate triples, n-ary claims, uncertain assertions, or mixed? -4. What counts as safe self-experimentation versus unsafe self-modification? -5. Should experiment promotion produce canonical knowledge only, or may it also publish new workflow graphs and policies? -6. How reversible must promotion be? -7. Should theory-local execution be able to call all normal tools, or only a capability-scoped safe subset? -8. How much of experiment evaluation should be deterministic versus LLM-judged? -9. Should experiment runs be stored as Vontology artefacts, Mongo projections, or both? -10. How should an agent decide when to stay in Tier 1 versus escalate to Tier 2? - -## 15. Literature Notes - -This task included a bounded literature pass. The most relevant material found was adjacent rather than exact. - -### 15.1 Closest adjacent work - -1. Agent Laboratory: structured multi-agent research workflow spanning literature review, experimentation, and report writing. - Source: - Paper: -2. AgentRxiv: collaborative autonomous research with cumulative access to prior generated research. - Source: - Paper page: -3. A Self-Improving Coding Agent: self-editing agent loop using benchmark-driven improvement. - Source: -4. MLR-Bench: open-ended machine-learning research benchmark with staged evaluation and automated judging. - Source: - Poster summary: -5. SUPER: benchmark for setting up and executing tasks from research repositories. - Source: - ACL PDF: -6. ML-Bench: repository-scale end-to-end execution benchmark for LLMs and agents. - Source: - OpenReview PDF: - -### 15.2 Classical microtheory background - -The direct modern literature search here was weak on "microtheories" specifically, but the classical idea is clearly relevant: compartmentalised knowledge contexts with controlled inheritance and inclusion. One accessible pointer found during this task is a Cognitive Systems poster discussing Cyc-style microtheories and their use as cases: - -1. Poster discussing microtheories as cases and contextual bins of facts: - -### 15.3 Literature conclusion - -The scan did not find an exact precedent for: - -1. ontology-backed ephemeral theories, -2. durable workflow execution over those theories, -3. explicit promotion gates into canonical knowledge, -4. safe self-experimentation by the same agentic system on its own workflow behaviour. - -That does not prove novelty, but it does suggest the precise combination is at least underexplored in currently visible agent literature. - -## 16. Literature Review Plan for a Non-Coding Agent - -If a deeper literature review is needed, use this plan. - -### Search themes - -1. `LLM agent self-improvement benchmark` -2. `autonomous research agent evaluation benchmark` -3. `microtheory knowledge representation context inclusion` -4. `safe self-modification agents` -5. `sandboxed agent evaluation workflow` -6. `scientific discovery agent benchmark` -7. `ontology-backed experimentation` or `knowledge graph experiment provenance` - -### Databases - -1. arXiv -2. ACL Anthology -3. NeurIPS proceedings -4. ICLR / OpenReview -5. Google Scholar -6. Semantic Scholar - -### Questions to answer - -1. What is the closest work to safe self-experimentation on agent workflows? -2. How do existing systems represent provisional theories, hypotheses, or contexts? -3. What safety mechanisms are used when agents modify or evaluate themselves? -4. What experiment record schemas and evaluation standards already exist? -5. Are there prior systems that separate provisional theory from canonical knowledge with explicit promotion gates? - -### Output format - -For each paper: - -1. problem addressed, -2. experiment model, -3. isolation mechanism, -4. whether self-improvement or self-experimentation is involved, -5. whether knowledge representation is ontology-backed, -6. direct relevance to Von. - -## 17. Bottom Line - -Von is already closer to Testing Workflows than it may appear. - -The workflow engine, benchmark harness, execution evidence, and candidate-testing patterns mostly exist. The missing base is a first-class theory and experiment layer with safe inclusion, isolation, and promotion semantics. - -If that layer is added cleanly, Testing Workflows can be built mostly by composing existing workflow, benchmark, and learning primitives rather than by inventing a new subsystem. diff --git a/docs/engineering/testing_workflows_framework_plan.md b/docs/engineering/testing_workflows_framework_plan.md deleted file mode 100644 index b0fb39173..000000000 --- a/docs/engineering/testing_workflows_framework_plan.md +++ /dev/null @@ -1,683 +0,0 @@ -Status: Planning document only -Date: 2026-03-18 -Authoring context: Compiled by Codex from prior design exploration, current repo state, and current Jira state -Primary Jira task: `JVNAUTOSCI-1531` - -# Testing Workflows Framework Plan - -## 1. Purpose - -This document defines a concrete planning framework for "Testing Workflows" in Von. - -The goal is to let Von: - -1. create bounded experimental theories or isolated benchmark worlds, -2. run workflows, tools, prompts, and learning policies against them, -3. collect structured evidence about what happened, -4. decide pass, fail, partial, or inconclusive outcomes, -5. promote only validated results into authoritative Vontology or workflow publication paths. - -This is a planning artefact, not an implementation. - -## 2. Executive Summary - -Von now has most of the workflow-runtime substrate required for a testing framework: - -1. Vontology-authoritative workflow execution and routing, -2. durable workflow instances, schedules, and event bindings, -3. structured turn-execution evidence, -4. workflow-gap recovery and candidate workflow testing patterns, -5. a heavyweight isolated benchmark harness. - -The missing layer is not another workflow engine. The missing layer is a first-class theory and experiment model plus a safe isolation boundary for testing. - -The clean architecture is two-tier: - -1. Tier 1: ephemeral-theory tests for cheap, fast, bounded experiments. -2. Tier 2: isolated benchmark worlds for destructive or system-wide verification. - -Testing Workflows themselves should be VWL/Vontology workflows. Python should provide only reusable theory, experiment, comparison, promotion, rollback, and telemetry primitives. - -## 3. Why This Matters - -This framework would let Von do several things it cannot yet do cleanly: - -1. validate candidate workflows without leaking writes into canonical knowledge, -2. test existing workflows against synthetic or semi-synthetic data, -3. compare prompt, tool, and routing variants with structured evidence, -4. run bounded self-experiments on its own capabilities, -5. accumulate reusable experiment records that feed evaluation and learning loops. - -The combination appears unusual: - -1. ontology-backed provisional theories, -2. durable workflow execution over those theories, -3. explicit promotion gates into canonical knowledge or published workflows, -4. bounded self-experimentation by the same agentic system. - -The literature found adjacent work, but not this exact combination. - -## 4. Concrete Example Scenarios - -### 4.1 Meeting-invitation example - -User action: - -1. paste a meeting invitation into chat, -2. ask Von to test developing, using, and validating a workflow for that meeting type. - -The intended framework behaviour: - -1. create or resolve an `experiment_spec` for "meeting invitation workflow derivation and validation", -2. create an ephemeral theory containing: - - the invitation text, - - candidate meeting-type hypotheses, - - provisional assumptions about desired workflow outcomes, - - test-local expected outputs, -3. derive or select a candidate workflow for that meeting kind, -4. execute the candidate workflow against the ephemeral theory, -5. validate: - - whether the workflow classified the meeting appropriately, - - whether it extracted the expected structure, - - whether it produced the expected downstream actions or summaries, - - whether it avoided forbidden writes or unsafe external actions, -6. record an explicit verdict and evidence bundle, -7. optionally promote the workflow or just retain the run as evidence. - -Expected example assertions: - -1. a recurring lab meeting should produce a recurring-meeting workflow candidate, not a one-off event workflow, -2. participant and organiser resolution should remain theory-local during testing, -3. no canonical calendar or task mutations should occur unless the test explicitly enters a promotion or execution gate. - -### 4.2 Synthetic workflow-regression example - -Operator action: - -1. choose an existing workflow, -2. supply or generate a synthetic dataset of representative inputs, -3. ask Von to test the workflow against the dataset. - -The intended framework behaviour: - -1. resolve the target workflow and the synthetic fixture set, -2. create one theory slice per case or one benchmark world per suite, depending on risk, -3. run the workflow over each case, -4. compare observed versus expected effects, -5. aggregate suite metrics such as pass rate, failure modes, and mutation safety, -6. emit structured learning or benchmark signals, -7. retain failing cases for replay and debugging. - -Good first synthetic targets: - -1. file-copy upload classification and routing, -2. meeting or event interpretation, -3. workflow-selection and tool-routing decisions, -4. relation-elicitation or enrichment workflows, -5. workflow-gap recovery candidate tests. - -### 4.3 Bounded self-experimentation example - -System action: - -1. compare two candidate workflow-routing policies or prompt variants, -2. run both against a bounded suite of test cases, -3. determine whether one policy improves outcomes without increasing unsafe behaviour, -4. surface a recommendation, not a silent self-patch. - -The framework must ensure that: - -1. candidate policies are not published directly from test results, -2. learning signals are explicit and inspectable, -3. promotion remains a separate gated workflow. - -## 5. Existing Foundations Now Present - -### 5.1 Workflow-authoritative runtime - -The recent workflow-first convergence sweep substantially changed the starting point for this plan. The following are now done: - -1. `JVNAUTOSCI-1517` -2. `JVNAUTOSCI-1518` -3. `JVNAUTOSCI-1519` -4. `JVNAUTOSCI-1520` -5. `JVNAUTOSCI-1521` -6. `JVNAUTOSCI-1523` -7. `JVNAUTOSCI-1524` -8. `JVNAUTOSCI-1525` -9. `JVNAUTOSCI-1526` -10. `JVNAUTOSCI-1527` -11. `JVNAUTOSCI-1528` -12. `JVNAUTOSCI-1529` - -Important consequences: - -1. production workflow authority is now Vontology-first, -2. legacy selector fallback is gone, -3. capability override fallbacks are gone, -4. persisted workflow bindings and authoritative workflow descriptions are now the norm. - -This matters because Testing Workflows can now rely on an authoritative workflow substrate rather than inheriting hybrid runtime ambiguity. - -### 5.2 Workflow runtime and durable execution - -Relevant existing bases: - -1. [von_workflow_language_manual.md](von_workflow_language_manual.md) -2. [durable_workflow_system_design.md](durable_workflow_system_design.md) -3. [models.py](../../src/backend/workflows/durable/models.py) -4. [plan_state_runtime.py](../../src/backend/workflows/plan_state_runtime.py) - -What these already give us: - -1. durable instances, -2. checkpoints and resumes, -3. subworkflows, -4. verification and completion gates, -5. publication and runnability concepts. - -### 5.3 Structured execution evidence - -Relevant existing bases: - -1. [turn_execution_completion_schema_proposal.md](turn_execution_completion_schema_proposal.md) -2. [turn_execution_record_service.py](../../src/backend/services/turn_execution_record_service.py) -3. `turn_execution_*` internal MCP tools. - -This gives Testing Workflows a model for: - -1. structured evidence capture, -2. expected versus observed effects, -3. benchmarkable failure categories. - -### 5.4 Heavyweight benchmark harness - -Relevant existing bases: - -1. [kb_clone_benchmark_harness.md](kb_clone_benchmark_harness.md) -2. [kb_clone_benchmark_service.py](../../src/backend/services/kb_clone_benchmark_service.py) - -This is already a good Tier 2 runner. It should be treated as the heavyweight backend of the testing framework, not a separate evaluation universe. - -### 5.5 Candidate testing and learning loops - -Relevant existing bases: - -1. [workflow_gap_recovery_workflow.py](../../src/backend/workflows/durable/workflow_gap_recovery_workflow.py) -2. [workflow_selection_experience.py](../../src/backend/services/workflow_selection_experience.py) -3. [workflow_selection_policy_service.py](../../src/backend/services/workflow_selection_policy_service.py) -4. [workflow_episode_service.py](../../src/backend/services/workflow_episode_service.py) - -These already suggest the right shape: - -1. test candidates explicitly, -2. record episode-like evidence, -3. emit learning-compatible signals. - -### 5.6 Existing seed for transient theories - -There is already a `transient_microtheory` payload idea in renderer applicability work. It is not yet a full theory model, but it is a strong seed for a broader ephemeral-theory design. - -## 6. What Is Still Missing - -### 6.1 First-class theory model - -Von does not yet have a complete, canonical theory model for testing contexts. - -Missing pieces: - -1. a type hierarchy for theory, ephemeral theory, hypothesis, claim, assertion, and observation, -2. theory inclusion and composition semantics, -3. theory-local provenance and expiry, -4. canonical-versus-theory diff semantics, -5. promotion semantics from theory-local state to canonical state. - -### 6.2 First-class experiment model - -Von has turn-execution records and benchmark harnesses, but not a general-purpose experiment model. - -Missing pieces: - -1. `experiment_spec`, -2. `experiment_run`, -3. variables and controls, -4. expected versus observed outcomes, -5. pass/fail/partial/inconclusive contracts, -6. suite aggregation and replay. - -### 6.3 Theory-safe mutation boundary - -The current namespace and access-control model is not yet sufficient for safe theory inclusion and promotion. - -Missing pieces: - -1. theory-aware visibility and inclusion rules, -2. explicit isolation boundaries for test-local writes, -3. promotion workflows that preserve auditability, -4. rollback and garbage collection semantics for ephemeral theories. - -### 6.4 Theory and experiment control surfaces - -Testing Workflows need reusable generic actions. These do not yet exist as a coherent family. - -Needed generic actions: - -1. `theory.create_slice` -2. `theory.import_canonical_context` -3. `theory.assert_local_claim` -4. `theory.compute_diff` -5. `theory.rollback_local_writes` -6. `theory.promote_validated_claims` -7. `experiment.record_observation` -8. `experiment.compute_verdict` -9. `experiment.emit_learning_signal` - -### 6.5 Lightweight frequent test harness - -The benchmark clone harness is too heavy for every experiment. A theory-slice runner is needed between: - -1. "mutate nothing and hope", -2. "clone a full database every time". - -### 6.6 Reliable inspection surface - -The earlier exploration hit practical MCP introspection issues for workflow listing and binding inspection. This is not the main conceptual blocker, but it is a real implementation friction point for agent-driven testing. - -## 7. Proposed Architecture - -## 7.1 Two-tier testing architecture - -### Tier 1: Ephemeral theory tests - -Use for: - -1. prompt comparisons, -2. routing comparisons, -3. tool-plan comparisons, -4. meeting-invitation workflow derivation and validation, -5. theory-local reasoning checks, -6. safe provisional assertions, -7. candidate workflow or policy evaluation. - -Properties: - -1. cheap, -2. fast, -3. bounded lifetime, -4. theory-local, -5. non-authoritative by default. - -### Tier 2: Isolated benchmark worlds - -Use for: - -1. destructive or multi-collection workflows, -2. ingestion/indexing workflows, -3. integration tests across many services, -4. final verification before promotion or publication, -5. regression suites that need real persistence semantics. - -Properties: - -1. slower, -2. heavier, -3. DB-clone based, -4. stronger isolation, -5. more faithful for end-to-end verification. - -## 7.2 Core artefact families - -### A. Testing theory - -Purpose: - -1. represent a bounded, non-canonical experimental theory or context slice. - -Should include: - -1. source namespace, -2. included canonical concepts, -3. theory-local assertions, -4. assumptions, -5. expected observations, -6. expiry and retention policy, -7. promotion policy, -8. provenance. - -### B. Experiment specification - -Purpose: - -1. define what is being tested, against what fixtures, with what expected outcomes. - -Should include: - -1. target workflow, prompt, tool contract, or policy, -2. fixture source, -3. theory setup instructions, -4. expected effects, -5. allowed side effects, -6. verdict rules, -7. retention and replay policy. - -### C. Experiment run - -Purpose: - -1. record the actual execution and verdict. - -Should include: - -1. the theory or benchmark world used, -2. resolved workflow and candidate set, -3. tool history, -4. observed outputs and mutations, -5. comparison against expectations, -6. verdict, -7. metrics, -8. replay handles. - -### D. Testing workflow - -Purpose: - -1. orchestrate test setup, execution, comparison, and retention or promotion. - -Recommended families: - -1. `theory_slice_setup_workflow` -2. `capability_test_execution_workflow` -3. `workflow_regression_test_workflow` -4. `tool_contract_test_workflow` -5. `learning_signal_harvest_workflow` -6. `promotion_gate_workflow` -7. `ephemeral_theory_gc_workflow` - -## 8. Safety Model - -### 8.1 Canonical protection - -Testing Workflows must default to writing only into: - -1. theory-local state, -2. isolated benchmark clones, -3. experiment-run evidence stores. - -Canonical Vontology writes must occur only in an explicit promotion workflow. - -### 8.2 Namespace and theory inclusion - -Required policy: - -1. a testing theory inherits namespace and organisation scope, -2. theory inclusion cannot widen visibility, -3. cross-namespace theory inclusion is disallowed unless future access control explicitly allows it, -4. promotion into canonical space requires policy, provenance, and an audit trail. - -### 8.3 Allowed self-experimentation - -Allowed targets should likely include: - -1. workflow selection policy, -2. prompt variants, -3. tool sequencing, -4. candidate workflow graphs, -5. reasoning strategies inside isolated theory space. - -Not allowed without stronger gates: - -1. direct self-editing of authoritative runtime code, -2. direct publication of canonical workflows from a test verdict alone, -3. direct canonical mutation without an explicit promotion workflow. - -### 8.4 Expiry and garbage collection - -Ephemeral theories should have TTL semantics: - -1. discard by default after successful non-promoting tests, -2. retain failures when needed for debugging, -3. retain promoted theories only as auditable history or condensed provenance. - -## 9. Evidence, Metrics, and Success Criteria - -Each experiment run should capture: - -1. candidate set, -2. selected workflow or policy, -3. expected effects, -4. observed effects, -5. tool and mutation history, -6. prohibited side-effect checks, -7. verdict, -8. replay handle. - -Useful aggregate metrics: - -1. pass rate, -2. partial and inconclusive rates, -3. false-success rate, -4. prohibited-write rate, -5. benchmark-world escalation rate, -6. promotion acceptance rate, -7. learning-signal quality or reuse rate. - -Framework success criteria for an MVP: - -1. one meeting-invitation-derived workflow test runs end to end in Tier 1, -2. one existing workflow regression suite runs over synthetic fixtures, -3. at least one failing case is replayable, -4. no canonical writes occur outside promotion workflows, -5. experiment outputs are structured enough to feed dashboards or learning loops. - -## 10. Related Jira and Dependency Map - -### 10.1 Existing foundations now largely complete - -Completed work that lowers risk: - -1. `JVNAUTOSCI-1517` and its convergence subtasks, -2. `JVNAUTOSCI-1523` strict Vontology-authoritative runtime mode, -3. `JVNAUTOSCI-1524` workflow purity scoreboard and non-regression gates. - -### 10.2 Directly relevant open work - -Evaluation and measurement: - -1. `JVNAUTOSCI-964` `To Do`: umbrella evaluation and measurement epic -2. `JVNAUTOSCI-1274` `To Do`: benchmark-driven evaluation framework for VQL reasoning -3. `JVNAUTOSCI-1057` `To Do`: episode log analysis for tool learning feedback loops - -Theory, context, and ontology semantics: - -1. `JVNAUTOSCI-936` `In Progress`: representationally rich Vontology and inference engine -2. `JVNAUTOSCI-254` `In Progress`: context bundles and reconstructed dossiers -3. `JVNAUTOSCI-1266` `To Do`: context semantics, entailment profiles, capability negotiation -4. `JVNAUTOSCI-343` `To Do`: theory, claim, assertion, and hypothesis definitions -5. `JVNAUTOSCI-296` `To Do`: concept-level access control - -Agentic behaviour context: - -1. `JVNAUTOSCI-833` `In Progress`: agentic behaviours in Von - -### 10.3 Interpretation - -This testing framework should not wait for all of those issues to finish before starting. But the implementation plan should explicitly absorb or depend on them where appropriate: - -1. use `JVNAUTOSCI-964` as the umbrella for evaluation semantics, -2. use `JVNAUTOSCI-936`, `JVNAUTOSCI-343`, and `JVNAUTOSCI-1266` to define theory semantics, -3. use `JVNAUTOSCI-296` for access and visibility safety, -4. use `JVNAUTOSCI-1057` and related episode infrastructure for learning signals. - -## 11. Recommended Delivery Sequence - -### Phase 0: Planning and contracts - -Deliverables: - -1. this planning document, -2. a canonical problem statement and success criteria, -3. agreement on Tier 1 versus Tier 2 boundaries. - -Exit criteria: - -1. the framework scope is stable enough to create implementation tasks without re-arguing the architecture. - -### Phase 1: Theory and experiment ontology - -Deliverables: - -1. ontology vocabulary for theory, experiment spec, experiment run, observation, verdict, promotion candidate, -2. initial theory and experiment schemas, -3. context and inclusion rules. - -Exit criteria: - -1. we can represent a theory and an experiment run in Vontology without inventing ad hoc fields each time. - -### Phase 2: Generic theory and experiment control surfaces - -Deliverables: - -1. generic reusable actions for theory create/import/assert/diff/rollback/promote, -2. experiment observation, verdict, and learning-signal actions, -3. theory TTL and garbage-collection mechanisms. - -Exit criteria: - -1. no task-specific test orchestration code is required for the first real Testing Workflows. - -### Phase 3: First real Testing Workflows - -Deliverables: - -1. a meeting-invitation testing workflow, -2. a synthetic-data workflow-regression test workflow, -3. a promotion-gate workflow, -4. initial replay and debug retention policy. - -Exit criteria: - -1. at least two example scenarios run end to end with structured evidence. - -### Phase 4: Benchmark-world integration - -Deliverables: - -1. clean escalation from Tier 1 to Tier 2, -2. suite-level reporting across benchmark worlds, -3. heavyweight verification path for destructive workflows. - -Exit criteria: - -1. workflows that need stronger realism can be tested without inventing a separate framework. - -### Phase 5: Learning-loop integration - -Deliverables: - -1. experiment-run signals feed workflow selection experience, -2. dashboards or experiment summaries, -3. policy-controlled recommendations from experiment outcomes. - -Exit criteria: - -1. experiment outputs are usable beyond human inspection. - -### Phase 6: Bounded self-experimentation - -Deliverables: - -1. policy for allowed self-experiment targets, -2. gated workflow and policy comparisons, -3. explicit non-promotion default for self-experiment outcomes. - -Exit criteria: - -1. Von can compare its own workflow or routing strategies safely without silently self-modifying. - -## 12. Recommended First Implementation Tasks - -These are the first implementation tasks I would create after this planning task. - -1. define the ontology vocabulary for testing theories and experiment runs, -2. implement a minimal Tier 1 theory-slice control surface, -3. implement `experiment_run.v1` evidence capture parallel to turn-execution records, -4. materialise a meeting-invitation testing workflow, -5. materialise a synthetic-data workflow-regression suite workflow, -6. add a promotion-gate workflow for validated outcomes, -7. integrate retained failing cases with replay and benchmark dashboards. - -## 13. Hard Open Questions - -These are real unresolved questions: - -1. should ephemeral theories be persisted TTL concepts, a separate slice store, or both, -2. what is the canonical unit of assertion inside a testing theory, -3. how should theory inclusion compose with namespace and future access control, -4. what counts as safe self-experimentation versus unsafe self-modification, -5. should experiment promotion be able to publish workflows as well as knowledge, -6. how reversible must promotion be, -7. should theory-local execution be able to call all normal tools or only a safe subset, -8. how much of verdicting should be deterministic versus LLM-judged, -9. should experiment runs live primarily in Vontology, Mongo projections, or both, -10. how should the system decide when Tier 1 is insufficient and escalation to Tier 2 is required. - -## 14. Literature Notes - -The earlier exploration found adjacent work rather than a direct precedent. - -Closest adjacent sources: - -1. Agent Laboratory: and -2. AgentRxiv: and -3. A Self-Improving Coding Agent: -4. MLR-Bench: and -5. SUPER: and -6. ML-Bench: and - -Classical contextual-knowledge pointer: - -1. microtheory discussion poster: - -Current conclusion: - -1. there is clear adjacent work on autonomous research, benchmarked self-improvement, and sandboxed evaluation, -2. there is classical context-theory work, -3. the exact combination of ontology-backed provisional theories, durable workflow execution, explicit promotion, and bounded self-experimentation still appears underexplored. - -## 15. Literature Review Plan For Follow-up - -If a deeper literature review is needed, the next non-coding pass should search: - -1. `LLM agent self-improvement benchmark` -2. `safe self-modification agents` -3. `microtheory knowledge representation context inclusion` -4. `sandboxed agent evaluation workflow` -5. `knowledge graph experiment provenance` -6. `autonomous research agent evaluation` - -Databases: - -1. arXiv -2. ACL Anthology -3. OpenReview -4. NeurIPS proceedings -5. Semantic Scholar -6. Google Scholar - -For each paper, extract: - -1. problem addressed, -2. experiment model, -3. isolation mechanism, -4. whether self-experimentation is involved, -5. whether knowledge representation is ontology-backed, -6. direct relevance to Von. - -## 16. Bottom Line - -Von is now in a strong position to build a real Testing Workflows framework. - -The workflow substrate, routing authority, and heavy benchmark harness are already present. The main missing work is to make theory and experiment structure first-class, add reusable theory and experiment actions, and materialise the first real Testing Workflows around concrete scenarios such as meeting invitations and synthetic workflow-regression suites. - -If implemented cleanly, this framework would not be a side utility. It would become part of how Von safely develops, validates, compares, and eventually improves its own behaviour. diff --git a/docs/engineering/turn_execution_completion_schema_proposal.md b/docs/engineering/turn_execution_completion_schema_proposal.md deleted file mode 100644 index b32fd319f..000000000 --- a/docs/engineering/turn_execution_completion_schema_proposal.md +++ /dev/null @@ -1,320 +0,0 @@ -# Turn Execution Completion Schema Proposal (Draft) - -**Status**: Draft -**Date**: 2026-02-19 -**Scope**: Conversation-turn execution reliability, postcondition verification, completion gating - -## 1. Problem Evidence - -Two recent turn traces show a consistent failure mode: the system marks a turn as completed without proving that required mutations occurred. - -### 1.1 Request `25bc29ec-3890-4e3b-8980-108d5dcaaaf4` -- Prompt preview: `Gill Dobbie was invited but declined. Proceed with the predicates etc` -- `tools_started=0`, `tools_completed=0` -- Orchestrator reached `completed` with `workflow_task=buttonify` -- No mutation execution evidence exists in tool history - -### 1.2 Request `9b6c67f8-9e08-4e07-9b71-389347d4a71a` -- Prompt preview: `Those relations were not added. At least, they don't show in the UI for the event` -- Only tool call: forced retry to `search_concepts` (read-only) -- `result_summary`: `Found 0 concepts for 'Those relations were not added least'` -- Turn still reached `completed` with `workflow_task=buttonify` -- No postcondition workflow step blocked completion - -### 1.3 Root Gap -- There is no canonical per-turn contract for: - - required effects (especially implicit KB mutations), - - postcondition checks, - - completion gate decision criteria. -- Current diagnostics are rich but unstructured for enforcement. - -## 2. Design Goals - -1. Represent each turn as a structured execution contract. -2. Encode required effects before execution. -3. Require critic/postcondition checks for mutation effects. -4. Gate completion on verified evidence, not narration output. -5. Make failures queryable across all conversations. - -## 3. Canonical Schema (`turn_execution_record.v1`) - -### 3.1 JSON shape - -```json -{ - "schema_version": "turn_execution_record.v1", - "request_id": "string", - "session_id": "string", - "namespace": "string", - "user_id": "string", - "org_id": "string|null", - "created_at_utc": "ISO-8601", - "prompt": { - "preview": "string", - "sha256": "hex-string", - "source": "user_message|system_trigger|event_trigger" - }, - "workflow_selection": { - "selected_workflow_id": "string", - "selector_verdict": "string|null", - "selector_source": "workflow_selector|default|event_binding", - "workflow_discovery": { - "candidate_ids": ["string"], - "excluded_candidate_ids": ["string"] - } - }, - "required_effects": [ - { - "effect_id": "effect_1", - "intent_origin": "explicit|implicit", - "effect_type": "kb_mutation|jira_mutation|file_mutation|external_side_effect|read_only_lookup", - "description": "string", - "required_tools": ["string"], - "targets": [ - { - "kind": "concept|issue|file|url|text", - "id": "string" - } - ], - "required_predicates": [ - { - "source_id": "string", - "predicate": "string", - "target": "string" - } - ], - "postcondition_required": true, - "postcondition_strategy": "state_requery|state_diff|ui_projection|not_required", - "status": "pending|satisfied|not_satisfied|not_executed|inconclusive", - "status_reason": "string|null" - } - ], - "execution": { - "tool_invocations": [ - { - "tool": "string", - "status": "ok|error|blocked", - "started_at_utc": "ISO-8601|null", - "completed_at_utc": "ISO-8601|null", - "error": "string|null", - "result_summary": "string|null", - "payload_fingerprint": "sha256|null" - } - ], - "diagnostic_events": [ - { - "sequence_no": 1, - "at_utc": "ISO-8601", - "stage": "string", - "phase": "string", - "status": "string", - "tool": "string|null", - "workflow_task": "string|null", - "result_summary": "string|null" - } - ], - "retry": { - "attempts": 0, - "budget": 0, - "reason": "string|null" - } - }, - "postcondition_checks": [ - { - "check_id": "check_1", - "effect_id": "effect_1", - "check_type": "predicate_exists|predicate_absent|entity_exists|ui_projection", - "check_tool": "string", - "check_payload": {}, - "observed": {}, - "status": "verified|not_verified|inconclusive|error", - "evidence": "string|null", - "error": "string|null" - } - ], - "critic": { - "enabled": true, - "workflow_id": "#V#kb_mutation_postcondition_critic_workflow", - "summary": { - "verified_count": 0, - "not_verified_count": 0, - "inconclusive_count": 0, - "error_count": 0 - } - }, - "completion_gate": { - "workflow_id": "#V#turn_completion_gate_workflow", - "decision": "completed|partial|failed|escalation_required", - "decision_reason": "string", - "blocking_effect_ids": ["effect_1"], - "safe_to_claim_completion": false, - "requires_follow_up": true - }, - "final_response": { - "response_sha256": "hex-string", - "completion_claim_detected": true, - "completion_claim_validated": false - }, - "execution_correctness": { - "schema_version": "turn_execution_correctness.v1", - "overall_outcome": "successful_completion|false_success|unresolved_follow_up_needed|tool_or_workflow_misrouting|abstain_escalate_no_safe_route|mutation_failed_or_blocked|unknown", - "failure_mode": "completed_verified|false_completion_claim|false_completion_gate_state|mutation_not_executed|mutation_failed_or_blocked|postcondition_inconclusive|unresolved_required_effects|partial_unspecified|unvalidated_completion_claim|unknown", - "likely_failure_to_act": true, - "metric_labels": { - "successful_completion": false, - "false_success": false, - "unresolved_follow_up_needed": true, - "tool_or_workflow_misrouting": true, - "abstain_escalate_no_safe_route": false - } - } -} -``` - -### 3.2 Required enforcement rules - -1. Any `required_effects[*].effect_type` ending in `_mutation` must set `postcondition_required=true`. -2. A turn cannot end with `completion_gate.decision="completed"` if any required effect is `not_satisfied`, `not_executed`, `inconclusive`, or missing checks. -3. If a completion claim is detected in assistant text while mutation effects are unresolved, force `decision="escalation_required"` or `decision="partial"`. - -### 3.3 Shared execution-correctness label model - -To support JVNAUTOSCI-965 and downstream consumers such as `turn_execution_build_benchmark`, each turn record should also carry a compact `execution_correctness` summary. This is the shared label surface that Phase 1 reporting should consume, rather than re-deriving success/failure semantics independently in every benchmark or dashboard path. - -The minimum Phase 1 labels are: - -1. `successful_completion` -2. `false_success` -3. `unresolved_follow_up_needed` -4. `tool_or_workflow_misrouting` -5. `abstain_escalate_no_safe_route` - -The record should also preserve a more detailed `failure_mode` for triage and replay selection. - -## 4. Storage Mapping - -### 4.1 Chat history (authoritative per-turn debug) -- Path: `chat_history.history.$.llm_debug_data.turn_execution_record` -- Reason: keeps diagnostics aligned with existing request-level debug payload - -### 4.2 Durable workflow instances -- `workflow_instances.inputs.turn_execution_record`: - immutable contract at workflow start -- `workflow_instances.workflow_data.turn_execution_runtime`: - mutable in-flight state (effect statuses, interim checks) -- `workflow_instances.outputs.turn_execution_outcome`: - terminal critic + completion-gate verdict - -### 4.3 Query-optimised projection collection -Create `turn_execution_records` (one doc per assistant turn) to avoid expensive array scans in `chat_history`. - -Suggested indexes: - -```javascript -db.turn_execution_records.createIndex({ request_id: 1 }, { unique: true, name: "request_id_unique" }); -db.turn_execution_records.createIndex({ namespace: 1, created_at_utc: -1 }, { name: "namespace_created_desc" }); -db.turn_execution_records.createIndex({ "completion_gate.decision": 1, created_at_utc: -1 }, { name: "decision_created_desc" }); -db.turn_execution_records.createIndex({ "required_effects.status": 1, created_at_utc: -1 }, { name: "effect_status_created_desc" }); -db.turn_execution_records.createIndex({ "workflow_selection.selected_workflow_id": 1, created_at_utc: -1 }, { name: "workflow_created_desc" }); -``` - -## 5. Workflow Design - -### 5.1 New workflow concepts created (MCP) - -Created as organisation-scoped instances of `#V#durable_workflow`: - -- `#V#conversation_turn_execution_workflow` -- `#V#kb_mutation_postcondition_critic_workflow` -- `#V#turn_completion_gate_workflow` - -Descriptions have been attached via `hasDescription` text relations (`en-NZ`). - -### 5.2 Proposed responsibilities - -### `#V#conversation_turn_execution_workflow` -- derive `required_effects` -- execute tool plan -- hand off to critic workflow -- invoke completion gate workflow - -### `#V#kb_mutation_postcondition_critic_workflow` -- evaluate postcondition checks for each mutation effect -- classify each effect: `verified|not_verified|inconclusive|error` - -### `#V#turn_completion_gate_workflow` -- compute final decision from effect statuses + critic evidence -- block unverified completion claims - -### 5.3 Event binding recommendation - -Do not bind these new workflows yet. First implement executable definitions and action handlers, then bind: - -- `message.direct_created` -> `#V#conversation_turn_execution_workflow` - -The existing `message.direct_created` binding to `#V#chat_assistant_workflow` should be migrated only after parity tests pass. - -## 6. MCP Failure-Mining Tooling - -### 6.1 Implemented tools - -1. `turn_execution_list` (implemented) -- Filters: namespace, date range (`from_utc`, `to_utc`), completion decision, workflow_id, `requires_follow_up`, prompt substring. -- Backed by `turn_execution_records`. - -2. `turn_execution_get` (implemented) -- Fetches full structured record by `request_id` (`session_id` alias accepted). - -3. `turn_execution_search_failures` (implemented) -- Returns filtered turn records with deterministic failure-mode classification and aggregate counts. -- Adds recommendations for workflow/critic/gate hardening based on observed patterns. - -4. `turn_execution_build_benchmark` (implemented) -- Produces reproducible corpus-level metrics, seeded replay cases, and capability-gap signals. -- Intended for ongoing reliability benchmarking and regression tracking. -- Benchmark metrics should count the shared `execution_correctness.metric_labels` directly in addition to detailed failure modes, so selector-layer and turn-level reporting share one label vocabulary. -- Adds Jira-linkable triage metadata (`triage_index` + per-case issue-key extraction) so replay evidence can be routed directly into issue triage workflows. -- Supports optional baseline-rate comparison (`baseline_*_rate_pct` + `regression_tolerance_pct`) to flag metric regressions in automated benchmark runs. -- Includes pre-dispatch latency summaries and day-bucket trend views so `JVNAUTOSCI-1429` attribution feeds the same reporting surface instead of a parallel timing report. - -5. `turn_execution_build_dashboard` (implemented) -- Composes the turn-execution benchmark with the selector-routing benchmark into one dashboard/reporting payload. -- Intended for engineering debugging and portfolio review of selector accuracy, intent completion, false success, unresolved follow-up, and pre-dispatch latency. -- Supports drill-down from trend and regression views into replay cases and Jira-linked triage evidence. - -### 6.1A Operating protocol - -The active engineering protocol for running, reviewing, and gating this measurement stack now lives in: - -- `docs/engineering/execution_correctness_benchmark_protocol.md` - -That document is the authoritative repo-side operating guide for the Phase 1 evaluation cadence. This schema proposal remains the design note for the record structure and tool surfaces. - -6. `turn_execution_backfill_from_chat_history` (implemented) -- Replays assistant-message `llm_debug_data.turn_execution_record` payloads into `turn_execution_records`. -- Namespace-scoped and dry-run by default for safe backfill planning. -- Supports synthesis from legacy `llm_debug_data` + message context when embedded records are absent. -- Synthesis infers workflow routing metadata (`selected_workflow_id`, verdict/source) from debug/tool traces when possible. - -7. `turn_execution_namespace_coverage_report` (implemented) -- Produces namespace-level instrumentation/projection coverage, request-id overlap, and gap signals. -- Intended to validate benchmark readiness before interpreting failure-rate metrics. - -8. Chat-history write-path synthesis guardrail (implemented) -- `add_message_to_history(...)` now synthesises `turn_execution_record` for assistant turns when `llm_debug_data.request_id` exists but embedded record is missing. -- Ensures both persisted `llm_debug_data.turn_execution_record` and projection upsert occur on forward writes. - -### 6.2 Why this was needed - -- `search_knowledge_base` is text-centric and not deterministic for structured failure triage. -- Reliability analysis requires deterministic filtering and aggregation over execution fields. - -## 7. Immediate Implementation Sequence - -1. Add writer that emits `turn_execution_record.v1` into `llm_debug_data`. -2. Add `turn_execution_records` projection and indexes. -3. Implement code-backed actions for critic and completion gate. -4. Register executable workflow definitions for the three new workflow IDs. -5. Add regression tests using the two request patterns above: - - no-tool completion - - read-only tool on mutation-required prompt diff --git a/docs/engineering/von_mvp_task_messaging_mcp_alignment_2026-04-24.md b/docs/engineering/von_mvp_task_messaging_mcp_alignment_2026-04-24.md deleted file mode 100644 index 06b422c21..000000000 --- a/docs/engineering/von_mvp_task_messaging_mcp_alignment_2026-04-24.md +++ /dev/null @@ -1,291 +0,0 @@ -# Von MVP Task and Messaging Alignment - -Date: 2026-04-24 -Source: `C:\Users\mwit860\OneDrive - The University of Auckland\Downloads\VON MVP look like.docx` - -## Purpose - -This note explains how the intent in the MVP mock-up document can be satisfied by Von, and which roles and actions implied by its MCP framing still need to be implemented. - -The document is aligned with Von's direction if it is treated as a user-facing view over represented task, workflow, messaging, evidence, and role state. It should not become a separate flat task tracker. The authoritative state should remain in Vontology concepts, VWL workflow definitions, event bindings, schedules, prompt/programme artefacts, and typed evidence records. Python and MCP tools should provide reusable execution and integration surfaces. - -## Executive Summary - -The MVP document describes a structured coordination system for research labs. A request should become a task record with an owner, report-to person, Von role, status, checkpoint, deadline, progress signal, evidence, notes, attachments, messages, reminders, confirmations, and completion. - -Von already has much of the support surface: - -- Vontology-backed task records with rich parity fields. -- REST and UI surfaces for task creation, inspection, editing, history, comments, attachments, and links. -- Internal MCP task tools for creating, updating, listing, searching, transitioning, linking, commenting on, and attaching to tasks. -- Vontology-backed direct messages with REST and UI surfaces. -- Workflow event integration for task creation, task status changes, and direct message creation. -- Workflow MCP tools for instances, execution, event bindings, and schedules. -- Knowledge and research support surfaces for paper metadata, paper ingestion, RAG, concept relations, and paper recommendation messaging. - -The main gap is not the existence of a task object. The gap is the durable coordination policy around that object: represented roles, lifecycle milestones, messaging and reminder actions, evidence gates, meeting/calendar connectors, research-method support workflows, and lab-status reporting workflows. - -## Architectural Interpretation - -The mock-up should map to Von as follows: - -- A "Task Number" is a human-facing reference code. The canonical identity remains the Vontology concept ID. -- Task fields are represented task attributes and relationships, not merely UI fields. -- The timeline is a task lifecycle workflow plus history and evidence events, not a loose list of labels. -- "Von Role" is a represented role-performance concept, not a prompt persona or hard-coded Python branch. -- "Send via chat", reminders, confirmations, and completion checks are workflow actions with receipts. -- Evidence is a typed provenance object or linked artefact, not only free text. -- The UI task page is a projection of authoritative state, not the source of authority. -- MCP tools are reusable execution surfaces under workflow control; they should not become the home of decision policy. - -## Intent-to-Implementation Mapping - -| MVP intent | Current Von support | Remaining implementation | -| --- | --- | --- | -| Turn a natural-language request into a structured task record | `TaskManagementService.create_task`, task REST routes, task UI, and `task_create` MCP already support rich fields. | Author lab-specific task-construction workflows/prompts that infer task type, owner, report-to, deadlines, checkpoints, expected evidence, and role from the turn context. | -| Preserve owner, report-to, priority, type, status, checkpoint, evidence, notes, and attachments | These are already represented in task service, ontology service, task routes, task panel, comments, attachments, and task MCP tools. | Harden canonical predicates for participant roles, watchers, reviewers, escalation targets, and typed evidence receipts. | -| Display a lifecycle such as Created -> Sent -> Reminder -> Confirmed -> Complete | Task history and status updates exist; workflow event integration can launch workflows on task events. | Define canonical task milestone concepts and workflow-produced timeline events instead of deriving the timeline from generic history alone. | -| Send task context through chat | Direct messages exist as Vontology concepts with REST/UI support, and `message.direct_created` events can launch workflows. | Expose generic direct-message send/reply/search/read operations as internal MCP tools so workflows can use the same authoritative messaging path. | -| Remind people before deadlines | Workflow schedules and task due dates exist as support surfaces. | Implement reminder workflows that bind task deadlines/checkpoints to schedules, record delivery receipts, and escalate if acknowledgement is missing. | -| Confirm progress or completion | Task statuses, comments, attachments, evidence text, and worklogs exist. | Implement confirmation-request and completion-gate workflows that check expected progress signals and evidence before marking work complete. | -| Coordinate paper deadlines between professor and student | Task model can represent the deadline and ownership; research/paper tools can retrieve paper context. | Implement paper-submission tracking workflows with checkpoint schedules, evidence expectations, and role-aware messaging. | -| Arrange a Google Meet or Rolemation channel | Von has messaging concepts and workflow infrastructure. | Implement calendar/meeting/channel MCP actions such as `calendar_create_event`, `meeting_link_create`, and `rolemation_send_message`, with auth, receipts, and namespace checks. | -| Bring in recent relevant papers, methods, and citations | Paper metadata, paper download/finalisation, RAG, KB search, concept relations, and recommendation messaging surfaces exist. | Implement represented workflows for method-issue support, citation suggestion, paper tracking, and evidence-backed recommendation recording. | -| Produce weekly lab meeting updates from active tasks | Task list/search/history surfaces exist. | Implement lab-status reporting workflows that reconstruct task state by person/project, identify blockers, and distinguish evidence-backed status from missing evidence. | -| Track current students, past students, papers, affiliations, and trajectories | Vontology can represent people, roles, organisations, papers, and relationships. | Define canonical memory schemas and workflows for lab membership, student trajectories, paper stewardship, alumni tracking, privacy, and consent-aware outreach. | - -## Current Implementation Anchors - -The relevant existing support surfaces are: - -- `src/backend/services/task_management_service.py` - - Vontology-backed task creation and update logic. - - Rich task fields: assignee, creator, source, report-to, task role, next checkpoint, progress signal, evidence, notes, reference code, priority, start date, due date, epic, components, fix versions, sprint, and backlog rank. - - Task history events and best-effort workflow launch on task creation/status changes. -- `src/backend/services/task_ontology_service.py` - - Canonical task predicates and task type definitions. - - Existing task source, report-to, task role, checkpoint, progress signal, evidence, and reference-code predicates. -- `src/backend/server/routes/task_routes.py` - - Task creation/update/list/search/history/comment/attachment/link surfaces. -- `src/frontend/web/von_interface/static/js/components/taskPanel.js` - - User-facing task inspector with fields matching the MVP mock-up closely. - - Timeline rendering based on task history. -- `src/backend/services/message_service.py` - - Vontology-backed direct messages with sender, recipient, thread, reply, body, metadata, organisation scope, visibility, and workflow launch. -- `src/backend/server/routes/message_routes.py` - - Authenticated direct-message REST surface and episode logging. -- `src/frontend/web/von_interface/static/js/components/messagePanel.js` - - User-facing message thread UI. -- Internal MCP task tools - - `task_create`, `task_get`, `task_list`, `task_search`, `task_update_fields`, `task_get_transitions`, `task_transition`, `task_assign`, `task_update_status`, `task_add_comment`, `task_add_attachment`, `task_add_worklog`, `task_get_history`, `task_create_subtask`, `task_link`, and related operations. -- Internal MCP workflow tools - - Workflow definition listing, validation, instance creation, execution, traces, event bindings, schedules, schedule triggering, cancellation, and retry. -- Research and KB tools - - Paper metadata/download/finalisation, paper recommendation delivery, KB search, concept creation, relationship creation, RAG operations, and related concept lookup. - -## Roles Implied by the MVP - -The mock-up names "Communicator" as a Von role. That should become one role in a broader represented role system. - -Recommended first-class roles: - -| Role | Responsibility | Current status | -| --- | --- | --- | -| Task steward | Convert requests into structured task records and maintain task state. | Partly supported by task service, UI, and MCP task tools; role policy still needs represented workflow/prompt authority. | -| Communicator | Deliver task context to the right people through the right channel. | Direct messages exist; generic workflow-addressable MCP message actions are still missing. | -| Reminder and escalation steward | Schedule reminders, request acknowledgements, and escalate overdue work. | Workflow schedules exist; task-specific reminder/escalation workflows remain to be authored. | -| Meeting scheduler | Create meeting invitations, meeting links, and participant-specific context. | Not yet implemented as a confirmed MCP action surface. | -| Research-context aide | Retrieve relevant papers, methods, prior project context, and related concepts. | KB/RAG/paper surfaces exist; task-bound research support workflows remain to be authored. | -| Paper and citation steward | Track papers, citations, deadlines, submission artefacts, and evidence. | Paper tools exist; canonical paper-submission workflow remains to be implemented. | -| Lab-status reporter | Produce weekly updates by reconstructing task state across people and projects. | Task/search/history surfaces exist; report workflow remains to be implemented. | -| Evidence auditor | Check that completion claims match expected evidence and provenance. | Evidence fields exist; typed evidence receipts and completion gates remain to be implemented. | -| Team memory curator | Maintain represented knowledge about students, alumni, collaborators, teams, roles, and trajectories. | Vontology can support this; canonical schemas and workflows remain to be authored. | -| Outreach scout | Identify external teams, prior students, or paper authors worth contacting and prepare outreach. | Research surfaces provide input; outreach workflows and send-with-approval actions remain to be implemented. | - -These roles should be represented objects with capabilities, constraints, performance traces, and evaluable outcomes. They should not be implemented as a switch statement over role names. - -## MCP Actions Still Needed - -The task MCP surface is relatively mature. The missing MCP work is mostly around messaging, notifications, scheduling semantics, external collaboration channels, evidence, and research operations. - -Recommended action surfaces: - -| Action surface | Purpose | -| --- | --- | -| `message_send_direct` | Send a Vontology-backed direct message through the same path as the UI/REST messaging service. | -| `message_reply` | Reply in an existing represented thread. | -| `message_search` | Search message threads by participant, task, topic, or workflow instance. | -| `message_mark_read` | Record user-visible read state where appropriate. | -| `notification_send_channel` | Deliver a message through a selected channel while preserving task/workflow provenance. | -| `notification_delivery_status` | Retrieve delivery state for workflow diagnostics and escalation. | -| `notification_acknowledgement_record` | Record that a recipient acknowledged a reminder or request. | -| `task_schedule_checkpoint` | Bind a task checkpoint/deadline to a workflow schedule. | -| `task_request_confirmation` | Ask a participant to confirm progress, completion, or blocker status. | -| `task_confirm_progress` | Record a structured confirmation against the expected progress signal. | -| `task_record_evidence` | Attach typed evidence to a task, not only notes or comments. | -| `task_escalate_overdue` | Escalate a task according to represented workflow rules. | -| `calendar_create_event` | Create a calendar event with participants, agenda, task link, and provenance. | -| `calendar_update_event` | Update a represented calendar event from workflow context. | -| `meeting_link_create` | Generate or attach a meeting link for a scheduled meeting. | -| `meeting_invitation_send` | Send participant-specific invitations with task and context links. | -| `rolemation_send_message` | Deliver through the Rolemation channel, or through a generic channel adapter if Rolemation is one backend. | -| `paper_reference_context_build` | Build a task-bound context package of relevant references and methods. | -| `citation_suggest` | Suggest citations with evidence and provenance. | -| `paper_corpus_track` | Track a set of papers for updates, relevance, or downstream action. | -| `paper_relevance_review` | Record why a paper is relevant or not relevant to a task/project. | -| `research_method_issue_analyse` | Analyse a method problem against represented project context and literature. | -| `similar_methods_retrieve` | Retrieve similar method examples from the KB/paper corpus. | -| `method_recommendation_record` | Store an evidence-backed recommendation and link it to task/workflow state. | -| `lab_status_report_generate` | Generate a structured lab update from active task records and evidence. | -| `student_work_summary` | Summarise a person's current task state, blockers, recent evidence, and upcoming deadlines. | -| `blocker_detect` | Identify likely blockers from task history, missing evidence, messages, and overdue checkpoints. | -| `person_role_update` | Update represented person-role relationships with provenance. | -| `student_trajectory_record` | Record student/alumni trajectory facts with namespace and privacy controls. | -| `team_affiliation_link` | Link people, teams, organisations, projects, and papers. | -| `outreach_candidate_find` | Find candidate teams/authors/alumni for potential outreach. | -| `outreach_draft_message` | Draft outreach without sending it automatically. | -| `outreach_send_with_approval` | Send outreach only after the required confirmation or workflow approval. | -| `evidence_receipt_create` | Create a typed receipt for artefacts, confirmations, delivery events, or completion proof. | -| `artifact_link_to_task` | Link a file, URL, document, paper, or meeting artefact to a task with provenance. | -| `completion_gate_evaluate` | Decide whether a task can move to complete based on represented workflow criteria and evidence. | - -These names are illustrative. The important point is that the action contract must be reusable, Vontology-aware, permission-aware, and workflow-callable. - -## Workflow and Vontology Artefacts Needed - -The following artefacts should be authored before treating the MVP as complete: - -- Task intake workflow for research-lab coordination requests. -- Task lifecycle workflow with canonical milestones: created, delivered, acknowledged, reminder due, reminder sent, progress confirmed, evidence received, completion checked, completed, blocked, escalated, cancelled. -- Reminder and escalation workflow bound to task deadlines and checkpoints. -- Direct-message delivery workflow that records message receipts and links messages to tasks. -- Paper-submission tracking workflow. -- Research-method support workflow for retrieving comparable methods and evidence. -- Weekly lab-status reporting workflow. -- Meeting-scheduling workflow with participant context and task links. -- Evidence receipt type hierarchy and predicates. -- Participant-role schema: owner, reporter, reviewer, watcher, escalation target, sender, recipient, meeting participant, responsible role. -- Represented role-performance schema for Von roles such as Communicator, Task steward, Paper steward, and Lab-status reporter. -- Lab memory schema for students, alumni, supervisors, projects, papers, affiliations, placements, and privacy/consent state. - -## Proposed Implementation Phases - -### Phase 0: Canonical Mapping - -Confirm the canonical Vontology concepts and predicates for tasks, participant roles, milestones, evidence receipts, messages, schedule bindings, and task-linked artefacts. - -Output: - -- Ontology mapping note. -- Any missing predicates or types created through Vontology pathways. -- Decision on which MVP fields are free text, typed relations, or derived workflow state. - -### Phase 1: Workflow-Addressable Messaging - -Expose the direct-message service as internal MCP actions and bind task-created/message-created events to workflows. - -Output: - -- Generic direct-message MCP actions. -- Message receipt records. -- Tests through the real MCP gateway path. -- Workflow trace showing task context delivered to a recipient. - -### Phase 2: Task Lifecycle, Reminders, and Evidence - -Author the canonical task lifecycle workflow and reminder schedule bindings. - -Output: - -- Milestone-producing workflow. -- Checkpoint and reminder schedules. -- Evidence receipt creation. -- Completion gate workflow. -- Task UI timeline backed by semantic milestones rather than generic history alone. - -### Phase 3: Research-Lab MVP Workflows - -Author the first lab workflows: - -- Paper submission tracker. -- Meeting and action-follow-up workflow. -- Research-method issue workflow. -- Weekly lab-status report workflow. - -Output: - -- Replayable end-to-end examples. -- Telemetry showing workflow decisions, context, actions, and receipts. -- User-facing task records and messages. - -### Phase 4: Role Learning and Team Memory - -Make Von roles and lab memory first-class represented objects. - -Output: - -- Role objects with capabilities and constraints. -- Role-performance traces. -- Student/project/paper/team memory schemas. -- Evaluation cases for task-state reconstruction and role usefulness. - -### Phase 5: External Connectors - -Add the external action surfaces required by the MVP. - -Output: - -- Calendar and meeting actions. -- Rolemation or generic channel adapter actions. -- Email/send actions if required. -- Outreach actions with approval gates. -- Connector-specific receipts and error telemetry. - -## Acceptance Evidence for MVP Readiness - -Von should be considered aligned with this MVP only when the following real-path checks pass: - -- A user request to track a paper deadline creates a task with owner, report-to, due date, checkpoint, Von role, progress signal, expected evidence, and a workflow instance. -- The task-created workflow sends the relevant message through a represented messaging path and records a delivery receipt. -- A reminder fires from a workflow schedule before the deadline and records acknowledgement or non-acknowledgement. -- If acknowledgement is missing, the task escalates according to represented workflow policy. -- Completion is blocked or qualified when expected evidence is missing. -- A weekly lab update reconstructs active tasks by person/project and explicitly marks uncertain or evidence-missing status. -- A research-method question retrieves relevant papers/methods, records the evidence used, and links the recommendation back to the task. -- The same state is visible through the UI task page, MCP task tools, workflow traces, and Vontology queries. -- No durable task, role, routing, reminder, or completion policy lives only in Python. - -## Anti-Patterns to Avoid - -- Do not make the mock-up UI fields the authority instead of Vontology/workflow state. -- Do not make the human-facing task number the canonical task identity. -- Do not hard-code research-lab coordination policy in Python. -- Do not implement "Von Role" as a prompt persona or string switch. -- Do not treat "message sent" as equivalent to "recipient acknowledged" or "task progressed". -- Do not store enduring paper, student, or alumni knowledge only as task notes. -- Do not create workflow-specific one-off MCP tools where a reusable channel, evidence, message, calendar, or task action would do. -- Do not let execution summaries replace the answer users need. - -## Target Flow - -```mermaid -flowchart LR - A["User request"] --> B["Chat-turn workflow"] - B --> C["task_create MCP action"] - C --> D["Vontology task concept"] - D --> E["task.created event binding"] - E --> F["Task lifecycle workflow"] - F --> G["Message or notification action"] - F --> H["Checkpoint schedule"] - H --> I["Reminder or escalation action"] - G --> J["Delivery and acknowledgement receipts"] - I --> J - J --> K["Evidence and completion gate"] - K --> L["Task state, UI, reports, and traces"] -``` - -## Bottom Line - -The MVP document is not asking Von to become a conventional task tracker. It is asking for a structured coordination layer where tasks, messages, reminders, evidence, and research context are represented, inspectable, workflow-driven, and useful to a lab. - -Von is already close at the task-record and workflow-infrastructure level. The next implementation work should focus on represented lifecycle policy, workflow-addressable messaging, typed evidence, reminder/escalation workflows, role-performance objects, research-lab workflows, and the external connector MCP actions needed to communicate through real channels. diff --git a/docs/engineering/vontology_tooling_from_ka_kcap_kr_literature.md b/docs/engineering/vontology_tooling_from_ka_kcap_kr_literature.md deleted file mode 100644 index 2b344c509..000000000 --- a/docs/engineering/vontology_tooling_from_ka_kcap_kr_literature.md +++ /dev/null @@ -1,596 +0,0 @@ -# Vontology Tooling Directions from KA, KCAP, Common-Sense KR, and Scientific KR - -**Status:** design note -**Date:** 24 April 2026 -**Scope:** proposed Vontology read/write/support tools informed by knowledge acquisition, knowledge capture, commonsense knowledge representation, and scientific knowledge representation literature. - -## 1. Purpose - -This note identifies Vontology tools that would make Von better at grounded retrieval, mixed-initiative knowledge acquisition, scientific reasoning, and long-horizon agent memory. - -The design assumption is the core Von one: Vontology, workflows, prompt programmes, represented metadata, and durable KB assertions should hold the authored policy. Python should provide reusable support surfaces only: query execution, validation, telemetry, provenance capture, gateway exposure, and safe mutation pathways. - -The tool names below are proposals. They should be treated as capability sketches to be refined against existing Vontology predicates and workflow concepts before implementation. - -## 1.1 Tool Versus Workflow Interpretation - -Several "tool families" below should not be implemented as standalone MCP tools -that make their own policy decisions. The intended split is: - -- **Workflow-primary capabilities:** mixed-initiative KA agendas, concept - placement, scientific claim/mechanism materialisation, conflict resolution, - analogy/pattern use, authority-surface audit, and ephemeral theory promotion. - These should be Vontology/VWL-authored workflows whose prompts, policies, - context choices, approval gates, and acceptance criteria live in represented - authority surfaces. -- **Support-tool primitives:** bounded graph reads, typed incidence summaries, - context-bounded relation lookup, provenance/evidence bundle retrieval, shape - validation, role-frame retrieval, external identifier lookup, graph matching, - and safe promotion/write primitives. These tools should be reusable and - policy-neutral enough for multiple workflows to call. -- **Hybrid capabilities:** context/microtheory querying, provenance tracing, - belief-state comparison, and scientific mechanism support often need both a - reusable read primitive and a supervising workflow that decides which context, - claim, or representation is appropriate for the user turn. - -When turning this note into Jira work, phrase implementation tasks in terms of -"author the workflow and add the missing support surfaces" unless the capability -is clearly just a low-level query/validation primitive. Avoid moving acquisition, -resolution, conflict-handling, or promotion policy into Python merely because it -is exposed through an MCP tool. - -## 2. Literature Signals - -### 2.1 Assisted Knowledge Acquisition - -The Cyc KA papers point to a mixed-initiative acquisition loop rather than passive form filling. "Knowledge Begets Knowledge" describes using an existing KB and inference system to elicit, verify, and accelerate new knowledge entry. "An Interactive Dialogue System for Knowledge Acquisition in Cyc" emphasises topic/user modelling, prioritised interactions, a transparent agenda, and system-initiated questions aimed at knowledge gaps and inferential utility. TextLearner adds another important point: acquisition from text benefits from storing intermediate document models and competing hypotheses, not only final extracted facts. - -For Von this implies tools that can: - -- discover what knowledge is missing for a concept, type, task, workflow, or paper; -- explain why a question is worth asking; -- rank acquisition opportunities by expected downstream utility; -- keep hypotheses, ambiguous readings, and rejected interpretations visible; -- validate new assertions against type, context, provenance, and consistency constraints before durable promotion. - -### 2.2 CommonKADS and Ontology Engineering - -CommonKADS separates task, inference, domain, and communication models. Protege's long-lived success comes partly from making ontology editing, frames/classes/properties, constraints, and plugins accessible to working knowledge engineers. SHACL shows the value of explicit validation shapes over relying on schema comments or downstream code failures. - -For Von this implies tools that can: - -- inspect the expected relation profile for a type or task; -- validate a concept or assertion bundle against represented constraints; -- separate user-facing task competence from domain ontology facts; -- expose ontology modelling gaps without making the LLM infer graph integrity from raw rows. - -### 2.3 Contexts and Microtheories - -Cyc's use of microtheories is a central lesson for Von. Context boundaries allow different assumptions, competing hypotheses, fictional claims, domain-specific defaults, and localised inference control. Context work by Guha/McCarthy/Buvac reinforces that the context of an assertion is not metadata decoration; it changes what follows from the assertion. - -For Von this implies tools that can: - -- query a concept or relation under an explicit context; -- compare what is asserted, inherited, or contradicted across contexts; -- return the context lineage that made an answer available; -- let workflows choose the correct context without hard-coded Python policy. - -### 2.4 Commonsense Knowledge Graphs - -ConceptNet shows the value of broad, labelled, natural-language-facing relations gathered from heterogeneous sources. ATOMIC and COMET show a complementary event-centred style: commonsense is often about likely preconditions, intents, effects, reactions, and next events. These are closer to Davidsonian or frame-like representations than to simple binary facts. - -For Von this implies tools that can: - -- retrieve by event frame, participant role, likely cause/effect, and mental-state role; -- distinguish direct entity predicates from event-mediated role predicates; -- surface low-confidence/generated commonsense separately from curated assertions; -- turn commonsense completion into reviewable candidate knowledge, not silent authority. - -### 2.5 Scientific KR - -Scientific KR standards point in a stricter direction than open commonsense graphs. OBO Foundry stresses coordinated ontology evolution and orthogonality. PROV-O and nanopublications make provenance and attribution first-class. BioPAX and SBML show that scientific mechanisms and models need formal exchange representations, not only text search hits. - -For Von this implies tools that can: - -- materialise scientific claims with assertion, provenance, publication, context, and confidence separated; -- represent mechanisms as entities, activities, conditions, causal/temporal structure, evidence, and competing hypotheses; -- align Vontology concepts to external identifiers and ontology terms; -- compare claims from different papers, datasets, and contexts; -- validate units, quantities, experimental methods, and model dependencies where a domain schema exists. - -## 3. Current Vontology Surface - -Von already has a useful base surface: - -- concept and KB search: `search_concepts`, `vontology_concept_search`, `search_knowledge_base`; -- concept/text access: `get_text_relations`, `get_text_relations_summary`; -- relation access: `find_relations_with_argument`, `get_predicate_incidence`; -- scholarly acquisition: `search_arxiv`, `get_paper_metadata`, `materialise_scholarly_representation_for_file_copy`; -- workflow tools and Jira/task tools. - -The important gap is not "more broad search". It is structured, context-aware, provenance-aware, role-aware Vontology interrogation and safe knowledge promotion. - -## 4. Proposed Tool Families - -### 4.1 Typed Predicate Incidence and Relation Profiles - -**Already identified near-term gap:** extend `get_predicate_incidence` or add a companion tool that reports argument-type distributions for predicates connected to a concept or type. - -Useful tool shapes: - -- `get_typed_predicate_incidence` -- `summarise_type_relation_profile` -- `compare_relation_profile_to_type_expectations` - -Core output: - -- predicate id and name; -- direction or matched argument position; -- direct relation count; -- distinct target count; -- target type counts per argument role; -- sample assertions with relation ids; -- Davidsonian expansion summary when the concept participates through event roles; -- source/context/provenance counts. - -Why it matters: - -- For non-Davidsonian binary facts, it answers "what kinds of things is this entity related to, and how?" -- For Davidsonian/event representations, it answers "what event frames connect this entity to other typed participants?" -- For workflow selection, it helps the planner choose a narrow relation lookup instead of a broad semantic search. -- For ontology engineering, it exposes modelling drift such as predicates used with unexpected argument types. - -### 4.2 Context and Microtheory Query Tools - -Proposed tools: - -- `query_relations_in_context` -- `compare_context_assertions` -- `explain_context_lineage` -- `list_applicable_contexts_for_turn` - -Core output: - -- assertions available in the requested context; -- assertion source: direct, inherited, inferred, or hypothetical; -- context inclusion path; -- conflicts or shadowed assertions from neighbouring contexts; -- whether the assertion is durable, provisional, fictional, workflow-local, or user-private. - -Useful situations: - -- A user asks "what do we believe about this paper's claim?" where lab belief, paper claim, and general background differ. -- A workflow needs one answer under the user's private namespace and another under project-wide knowledge. -- Scientific hypotheses need to be stored without becoming settled fact. -- A LLM-facing answer needs to justify that it used the right context rather than a globally convenient one. - -Implementation notes: - -- The authored context hierarchy and policies must live in Vontology/VWL. -- Python should only execute context-bounded retrieval, compute lineage, and shape telemetry. -- Tool output must be namespace-scoped and must not leak private context existence through counts. - -### 4.3 Provenance, Evidence, and Explanation Tools - -Proposed tools: - -- `trace_assertion_provenance` -- `explain_relation_path` -- `get_supporting_evidence_bundle` -- `compare_evidence_for_claim` - -Core output: - -- assertion ids and relation ids; -- source artefacts, authors, timestamps, and acquisition route; -- provenance graph in a PROV-like shape: entity, activity, agent, source; -- confidence and review status; -- short LLM-facing evidence summary plus machine-readable evidence rows; -- explicit gaps where no source or verifier is recorded. - -Useful situations: - -- A user asks for a factual answer and Von must show whether it is grounded in Vontology, a paper, a user note, or model recall. -- A paper-derived claim is contradicted by a later source. -- An agent needs to decide whether a claim can be promoted from ephemeral theory to durable knowledge. -- Evaluation needs to distinguish "answer correct by luck" from "answer grounded by inspected evidence". - -Implementation notes: - -- Use a nanopublication-like split between assertion, provenance, and publication/context metadata. -- Do not hide provenance behind human prose. Return structured evidence rows for validators and telemetry. - -### 4.4 Shape, Constraint, and Modelling Validation Tools - -Proposed tools: - -- `validate_concept_against_shape` -- `validate_assertion_bundle` -- `list_shape_requirements_for_type` -- `find_shape_violations` - -Core output: - -- applicable type or workflow shapes; -- required, recommended, and disallowed predicates; -- domain/range/type mismatches; -- cardinality or uniqueness problems; -- missing provenance or missing context; -- severity, repair suggestions, and affected assertion ids. - -Useful situations: - -- Before creating a workflow, prompt, paper, person, project, or mechanism concept. -- Before promoting extracted paper claims into durable represented knowledge. -- During ontology cleanup when predicate usage has drifted. -- As an acceptance gate for knowledge-writing workflows. - -Implementation notes: - -- Shapes should be represented in Vontology, not hard-coded Python dictionaries. -- Python may provide a SHACL-inspired validator over Vontology's own data model. -- Validation should be non-destructive by default and return a repair plan rather than mutating automatically. - -### 4.5 Mixed-Initiative KA Agenda Tools - -Proposed tools: - -- `suggest_knowledge_acquisition_questions` -- `rank_knowledge_gaps` -- `explain_acquisition_question` -- `record_ka_answer_with_validation` - -Core output: - -- candidate questions; -- target concept, missing predicate, expected argument type, and context; -- why the question matters: validation failure, workflow blocker, high-value relation, user-requested goal, or inference utility; -- answer capture schema; -- validation result and proposed assertion bundle. - -Useful situations: - -- A user adds a new concept and Von should ask the one or two highest-value follow-up questions, not a long form. -- A workflow cannot proceed because a required relation is missing. -- A project role, paper, dataset, or model deployment lacks enough represented state for future automation. -- Von wants to reduce repeated user interruptions by filling high-utility slots opportunistically. - -Implementation notes: - -- The agenda policy belongs in Vontology/workflows. -- The tool should expose candidate gaps and utility evidence; the workflow decides whether to ask. -- User-authored text must be preserved exactly. - -### 4.6 Concept Placement and Ontology Alignment Tools - -Proposed tools: - -- `suggest_concept_placement` -- `compare_candidate_parent_types` -- `align_external_identifier` -- `find_external_ontology_mappings` - -Core output: - -- candidate parent types and sibling concepts; -- supporting names, descriptions, predicates, and sources; -- conflicts with existing type hierarchy; -- external ids and ontology terms: DOI, ORCID, arXiv, PubMed, GO/OBO, DBLP, Wikidata, OpenAlex, where applicable; -- confidence and review state. - -Useful situations: - -- A new paper, method, dataset, person, institution, or biological entity is introduced. -- A scientific claim uses a term that should align to an external controlled vocabulary. -- Retrieval fails because the local concept and external identifier have not been reconciled. -- Duplicate concepts are suspected. - -Implementation notes: - -- External alignment should remain evidence-bearing and reversible. -- The canonical Vontology concept id should remain the local authority even when external ids are attached. - -### 4.7 Scientific Claim and Mechanism Materialisation Tools - -Proposed tools: - -- `materialise_scientific_claim` -- `build_mechanism_skeleton` -- `compare_mechanism_models` -- `validate_scientific_model_bundle` - -Core output: - -- claim concept and assertion bundle; -- paper/source provenance; -- involved entities, processes, measurements, methods, datasets, and conditions; -- causal, temporal, and part-whole structure; -- competing or alternative mechanism links; -- model exchange references where applicable: BioPAX-like pathway, SBML-like quantitative model, or domain-specific representation. - -Useful situations: - -- Reading a molecular biology paper and extracting mechanism hypotheses. -- Representing a scientific result separately from the general belief that the result is true. -- Comparing two papers' causal models. -- Asking "what evidence supports this pathway?" or "which assumptions does this model require?" - -Implementation notes: - -- The tool should create a reviewable bundle, not silently assert all extracted claims as settled truth. -- Mechanism schemas should be Vontology concepts and predicates. -- Units and quantities should be validated where a domain shape exists. - -### 4.8 Davidsonian Role-Frame Retrieval Tools - -Proposed tools: - -- `find_event_frames_by_participant` -- `query_role_frame` -- `summarise_event_role_incidence` -- `expand_binary_relation_to_event_frames` - -Core output: - -- event/frame concept ids; -- roles played by the anchor concept; -- other participants grouped by role and type; -- time, place, modality, negation, source, and context qualifiers; -- related direct binary assertions when they exist. - -Useful situations: - -- Questions such as "which papers did I write with X?" where authorship may be direct or mediated through paper/event roles. -- Commonsense questions about actions, intentions, preconditions, and effects. -- Scientific mechanism queries where entities participate in processes under conditions. -- Translation between direct binary relations and more expressive event representations. - -Implementation notes: - -- The tool should not assume one event schema. It should use Vontology role metadata and type constraints. -- Outputs need to make the direct-vs-event-mediated distinction explicit to the LLM. - -### 4.9 Contradiction, Compatibility, and Belief-State Tools - -Proposed tools: - -- `find_claim_conflicts` -- `compare_belief_states` -- `summarise_assertion_truth_state` -- `propose_conflict_resolution_workflow` - -Core output: - -- mutually incompatible assertions; -- context in which each assertion holds; -- source, confidence, review status, and recency; -- whether the conflict is logical, temporal, terminological, or only apparent; -- candidate next actions: ask user, inspect source, align concepts, create hypothesis context, or mark obsolete. - -Useful situations: - -- Two papers make incompatible claims. -- A user corrects a previously stored fact. -- A workflow sees both "task is closed" and "task remains open" under different sources. -- A generated commonsense candidate conflicts with curated knowledge. - -Implementation notes: - -- The tool should report conflicts; workflows decide resolution. -- Do not let answer-generation collapse context-specific contradictions into one unqualified statement. - -### 4.10 Analogy, Pattern, and Case Retrieval Tools - -Proposed tools: - -- `find_structurally_similar_cases` -- `find_analogous_mechanisms` -- `summarise_recurrent_relation_pattern` -- `recommend_reusable_workflow_pattern` - -Core output: - -- matched structures and bindings; -- predicate/type/frame overlap; -- differences that matter; -- provenance and confidence; -- whether the analogy is asserted, inferred, or generated. - -Useful situations: - -- A new workflow resembles a prior research-operations workflow. -- A biological mechanism resembles a known pathway pattern. -- A knowledge acquisition task should reuse the question agenda from a similar concept type. -- A role-learning loop needs prior cases before asking the user. - -Implementation notes: - -- Similarity policy should be represented and inspectable where possible. -- Python can provide graph matching and ranking primitives, but not domain-specific hidden scoring doctrine. - -### 4.11 Authority-Surface and Drift Audit Tools - -Proposed tools: - -- `audit_authority_surface_for_concept` -- `trace_prompt_workflow_authority` -- `find_repo_policy_drift_candidates` -- `summarise_workflow_tool_contracts` - -Core output: - -- authoritative concept ids for prompts, workflows, predicates, and policy artefacts; -- repo-side seeds/snapshots and whether they are only bootstrap material; -- code paths that appear to contain semantic policy; -- missing Vontology artefacts required by a workflow; -- recommended Jira/refactor tasks when authority drift is discovered. - -Useful situations: - -- Before changing prompt, routing, workflow, or grounded retrieval behaviour. -- During staleness reviews of old Jira tasks. -- When tests appear to pin removed heuristic behaviour. -- When a workflow failure is actually a missing represented artefact. - -Implementation notes: - -- This is a meta-tool family, but it fits Von's architecture: represented authority is a runtime dependency, not a documentation preference. -- Outputs should be suitable for Jira comments and implementation planning. - -### 4.12 Ephemeral Theory Promotion Tools - -Proposed tools: - -- `list_ephemeral_theories_for_turn` -- `promote_ephemeral_theory_candidate` -- `discard_ephemeral_theory` -- `trace_theory_promotion_history` - -Core output: - -- temporary hypotheses generated during a turn or workflow; -- evidence and reasoning steps; -- conflicts and missing validation; -- proposed durable assertion bundle; -- promotion/discard decision record. - -Useful situations: - -- TextLearner-like reading creates competing interpretations of a passage. -- A workflow uses a temporary model of a user's intent and later needs to keep or discard it. -- A scientific paper reading path proposes claims that require human or tool validation. -- Long-horizon learning needs to avoid losing useful intermediate theories while also avoiding unreviewed KB pollution. - -Implementation notes: - -- Ephemeral hypotheses need explicit lifetime, context, owner, and promotion gates. -- Promotion must be a write workflow with validation and provenance. - -## 5. Prioritised Roadmap - -### Jira Decomposition - -The first implementation backlog created from this note is under -`JVNAUTOSCI-936`: - -| Capability family | Jira task | Authority split | -| --- | --- | --- | -| Typed predicate incidence and relation profiles | `JVNAUTOSCI-2036` | Support-tool primitive | -| Context and microtheory query | `JVNAUTOSCI-2038` | Hybrid workflow plus query support | -| Provenance, evidence, and explanation | `JVNAUTOSCI-2039` | Hybrid workflow plus evidence support | -| Shape, constraint, and modelling validation | `JVNAUTOSCI-2040` | Workflow plus validation primitive | -| Mixed-initiative KA agenda | `JVNAUTOSCI-2041` | Workflow-primary | -| Concept placement and ontology alignment | `JVNAUTOSCI-2042` | Workflow-primary plus lookup support | -| Scientific claim and mechanism materialisation | `JVNAUTOSCI-2043` | Workflow-primary | -| Davidsonian role-frame retrieval | `JVNAUTOSCI-2044` | Hybrid retrieval support plus workflow integration | -| Contradiction, compatibility, and belief-state comparison | `JVNAUTOSCI-2045` | Workflow-primary plus conflict-candidate support | -| Analogy, pattern, and case retrieval | `JVNAUTOSCI-2046` | Hybrid graph support plus workflow use | -| Authority-surface and drift audit | `JVNAUTOSCI-2047` | Workflow-primary | -| Ephemeral theory promotion | `JVNAUTOSCI-2048` | Workflow-primary | - -### Near Term - -1. `get_typed_predicate_incidence` or equivalent extension to `get_predicate_incidence`. -2. `trace_assertion_provenance` and `get_supporting_evidence_bundle`. -3. `query_relations_in_context` with explicit context lineage. -4. `validate_concept_against_shape` with Vontology-authored shapes. -5. `find_event_frames_by_participant` for Davidsonian role-frame retrieval. - -These directly improve the current failure class around entity-relative information requests and make 1894-style tests less dependent on broad semantic search. - -### Medium Term - -1. `suggest_knowledge_acquisition_questions`. -2. `suggest_concept_placement` and `align_external_identifier`. -3. `materialise_scientific_claim`. -4. `find_claim_conflicts`. -5. `summarise_type_relation_profile`. - -These shift Von from retrieval to active knowledge growth and scientific memory. - -### Later - -1. `find_structurally_similar_cases`. -2. `build_mechanism_skeleton`. -3. `compare_mechanism_models`. -4. `promote_ephemeral_theory_candidate`. -5. `audit_authority_surface_for_concept`. - -These support higher-order self-improvement, mechanistic scientific reasoning, and role-learning loops. - -## 6. Evaluation and Acceptance Strategy - -Each new tool family should be validated through the real gateway path, not only service-level unit tests. - -Minimum acceptance pattern: - -- Invoke through `InternalMCPGateway.invoke()` and the stdio MCP surface when externally exposed. -- Add focused service tests for edge cases and schema validation. -- Add orchestration-path tests where selector/planner behaviour depends on the tool. -- Use replay-style turns modelled on the 1894 class of failures: - - "What papers are mine?" - - "Which projects is this person affiliated with?" - - "What evidence supports this claim?" - - "What does this paper say under its own claims context versus what our lab believes?" - - "Which event frames connect this entity to this process?" -- Inspect turn telemetry for: - - tool chosen; - - context lineage; - - predicate/type filters; - - evidence rows; - - excluded candidates and exclusion reasons; - - answer-first final response rather than execution bookkeeping. - -Success criteria: - -- The LLM can choose a narrow Vontology tool when the entity is known. -- The answer can cite structured evidence and provenance. -- Context-sensitive claims are not flattened into global facts. -- Davidsonian/event-mediated relations are retrievable without hand-authored per-domain Python. -- Missing knowledge leads to useful KA questions or explicit uncertainty, not hallucinated completion. - -## 7. Security and Access Constraints - -All tools must preserve Von's namespace and authority model: - -- Count summaries must not leak the existence of private concepts or contexts. -- Provenance output must respect source-level access controls. -- Write or promotion tools must fail closed and use explicit workflow escalation for destructive or authority-changing mutations. -- Tool payloads must avoid printing secrets or private prompt bodies unless the caller is authorised and the workflow requires it. -- Generated commonsense and extracted scientific hypotheses must be marked as candidates until validated and promoted. - -## 8. Design Guidance for Implementation Tickets - -When turning a proposed tool into Jira work, include: - -- the canonical Vontology concepts, predicates, shapes, contexts, and workflows that should become authoritative; -- the exact existing service and MCP surfaces to extend; -- the read/write authority boundary; -- the telemetry contract; -- the LLM-facing shaped payload; -- real-path validation scenarios; -- examples covering both direct binary relations and event/frame-mediated Davidsonian relations; -- source/provenance expectations; -- the user-visible improvement in plain language. - -The implementation should not smuggle domain-specific scoring, relation lists, or prompt bodies into Python. If a reusable primitive is missing, add the primitive and keep authored policy in Vontology/VWL. - -## 9. Sources Consulted - -- Witbrock et al., "Knowledge Begets Knowledge: Steps towards Assisted Knowledge Acquisition in Cyc", AAAI Spring Symposium KCVC 2005. -- Witbrock et al., "An Interactive Dialogue System for Knowledge Acquisition in Cyc", IJCAI 2003 Workshop on Mixed-Initiative Intelligent Systems. -- Curtis et al., "Methods of Rule Acquisition in the TextLearner System", AAAI Spring Symposium 2009. -- Witbrock et al., "Cyc and the Big C: Reading that Produces and Uses Hypotheses about Complex Molecular Biology Mechanisms", AAAI Workshop 2015. -- Wielinga et al., "The CommonKADS Framework for Knowledge Modelling", 1992. -- Musen and the Protege Team, "The Protege Project: A Look Back and a Look Forward", AI Matters 2015. -- W3C, "Shapes Constraint Language (SHACL)", W3C Recommendation 2017. -- W3C, "PROV-O: The PROV Ontology", W3C Recommendation 2013. -- Smith et al., "The OBO Foundry: coordinated evolution of ontologies to support biomedical data integration", Nature Biotechnology 2007. -- Groth, Gibson, and Velterop, "The anatomy of a nanopublication", Information Services and Use 2010. -- Demir et al., "The BioPAX community standard for pathway data sharing", Nature Biotechnology 2010. -- Finney and Hucka, "Systems biology markup language: Level 2 and beyond", Biochemical Society Transactions 2003. -- Speer, Chin, and Havasi, "ConceptNet 5.5: An Open Multilingual Graph of General Knowledge", AAAI 2017. -- Sap et al., "ATOMIC: An Atlas of Machine Commonsense for If-Then Reasoning", AAAI 2019. -- Bosselut et al., "COMET: Commonsense Transformers for Automatic Knowledge Graph Construction", ACL 2019. -- Lenat, "CYC: A large-scale investment in knowledge infrastructure", Communications of the ACM 1995. -- Buvac, "Formalizing Context" resource page, including links to McCarthy and Guha context work. diff --git a/docs/engineering/vwl_implementation_audit_2026-03-24.md b/docs/engineering/vwl_implementation_audit_2026-03-24.md deleted file mode 100644 index 5f8ba595f..000000000 --- a/docs/engineering/vwl_implementation_audit_2026-03-24.md +++ /dev/null @@ -1,181 +0,0 @@ -# VWL Implementation Audit 2026-03-24 - -> **Document status: Frozen audit and correction record from March 2026.** Its -> authority lessons may remain relevant, but completion state, code anchors, -> fallback paths, and Jira sequencing require current verification. Use the VWL -> manual and live Vontology/runtime evidence as the implementation reference. - -Task: `JVNAUTOSCI-1575` -Date: 2026-03-24 - -Correction after `JVNAUTOSCI-1582`: - -- the earlier wording in this audit overstated the completion state of `JVNAUTOSCI-1576` and `JVNAUTOSCI-1577` by treating repo-side JSON workflow/template bundles as acceptable authoritative VWL/Vontology assets; -- the stronger contract is now explicit: direct authoritative workflow/template/routing/prompt logic belongs in Vontology-native artefacts, while repo-side files count only as migration seeds, generated snapshots, test fixtures, or exports; -- `JVNAUTOSCI-1581` now tracks the substantive migration back to Vontology-native workflow authority, while `JVNAUTOSCI-1582` through `JVNAUTOSCI-1585` provide the policy, purity, tooling, and repo-affordance guardrails around that migration. - -Update after `JVNAUTOSCI-1581`: - -- canonical workflow publication now derives default publication specs and workflow text from authoritative Vontology state first, with repo-side workflow bundles retained only as missing/incomplete seed fallback; -- workflow-template authority now lives in first-class Vontology template concepts plus template/profile text relations, and `workflow_template_profile_service.py` hydrates from the old JSON bundle only when those template concepts are absent; -- family bootstrap helpers now preserve any valid current Vontology workflow family rather than enforcing repo-seed parity on every run; -- repo-side workflow/template bundles remain in the tree only as seed fixtures pending `JVNAUTOSCI-1585` affordance cleanup. - -Update after `JVNAUTOSCI-1585`: - -- the misleading `src/backend/workflows/authored_sources` path has been retired in favour of `src/backend/workflows/repo_seed_bundles`; -- the generic seed-materialisation support now lives in `src/backend/services/workflow_repo_seed_bootstrap.py`, and the remaining repo-side helper names explicitly say `repo_seed` rather than `authored`; -- repo bundle filenames and schema versions now declare themselves as repo seed bundles, and the directory carries a README that states the non-authoritative contract directly. - -Update after `JVNAUTOSCI-1583`: - -- workflow-purity telemetry now treats repo-seed authority drift as a first-class regression via `repo_seed_authority_drift_path_count`; -- workflow-purity telemetry now also guards the two current Vontology-first seed-fallback paths via `vontology_first_seed_fallback_violation_count`; -- deterministic tests now fail if production code introduces repo-seed workflow/template authority outside the designated seed/bootstrap support modules, or if the guarded authority loaders stop resolving Vontology before seed fallback. - -Update after `JVNAUTOSCI-1584`: - -- `scripts/workflow_authority_review_snapshot.py` now provides deterministic `export` and `diff` commands for workflow-authority review; -- generated review snapshots now live under `docs/generated/workflow_authority_review_snapshots/`, with filenames explicitly marked as `.generated.json` and README guidance stating that they are non-authoritative derived artefacts; -- the snapshot export reads the same Vontology-first workflow and template authority surfaces the runtime consumes, so review ergonomics improve without reintroducing repo-side workflow authority. - -Update after `JVNAUTOSCI-1580`: - -- remaining workflow-governed prompt bodies were migrated out of Python constants/templates and into authoritative Vontology prompt concepts; -- `workflow_prompt_authority_service.py` now centralises only generic prompt-concept creation, validation, linking, and rendering support, while workflow/testing/bootstrap services no longer persist prompt bodies from Python; -- planning, enrichment, workflow-gap analysis/test/candidate execution, testing-workflow prompt support, parent-specificity prompt support, and workflow-description prompt support now all fail closed when their authoritative Vontology prompt concepts are missing or empty; -- workflow-purity telemetry now includes `python_authored_workflow_prompt_source_count`; current expected value is `0`. - -Update after `JVNAUTOSCI-1576`: - -- canonical workflow publication sources were migrated out of Python workflow-family spec builders and into version-controlled repo seed bundles under `src/backend/workflows/repo_seed_bundles/*.json` -- generic publication/materialisation support now lives in `src/backend/workflows/workflow_concept_authority_service.py` and `src/backend/services/workflow_repo_seed_bootstrap.py` -- workflow-purity telemetry now includes `python_authored_canonical_workflow_source_count`; current expected value is `0` - -Update after `JVNAUTOSCI-1577`: - -- workflow creation and workflow-gap candidate specs now resolve through repo seed template bundles in `src/backend/workflows/repo_seed_bundles/workflow_template_seed_bundle.json` -- workflow-family-specific workflow-creation discovery seeding was removed from `src/backend/services/workflow_discovery_service.py` -- canonical governance workflows now publish routing profiles and discovery exemplars from authored metadata, and capability-index text incorporates those exemplars for retrieval -- routing-role heuristics remain only as a minimal generic fallback for legacy workflows that still lack explicit routing metadata - -## Executive Summary - -As of 2026-03-24, Von's workflow runtime is substantially cleaner than earlier conversion-sweep documents suggest. Running `pdm run python scripts/workflow_purity_report.py` produced: - -- `built_in_registration_count=0` -- `remaining_python_workflow_family_count=0` -- `direct_instance_create_callsite_count=0` -- `env_event_binding_count=0` -- `legacy_selector_mode_count=0` -- `builtin_capability_override_count=0` -- `non_vontology_discoverable_workflow_count=0` -- `repo_seed_authority_drift_path_count=0` -- `vontology_first_seed_fallback_violation_count=0` - -That means the concrete workflow-authority and source-authority failures identified by this audit are now resolved. The original migration set is complete: - -- canonical workflow publication, template selection, routing metadata, and discovery exemplars resolve from Vontology-native authority first; -- typed-subworkflow route-map policy resolves from authoritative Vontology metadata; -- testing-workflow fixture preparation and regression-suite escalation semantics resolve from reusable testing metadata rather than Python-held workflow branches; -- workflow-governed prompt bodies now resolve from authoritative Vontology prompt concepts, with Python limited to generic resolution/rendering/validation infrastructure. - -Any remaining work in this area is therefore no longer “finish the identified migration”; it is either regression prevention, future capability extension, or broader ontology-mutation generalisation outside the concrete findings below. - -## Areas Reviewed - -The audit covered these current workflow-first surfaces: - -- `docs/engineering/von_workflow_language_manual.md` -- `docs/engineering/workflow_first_conversion_sweep_jvnautosci_1210.md` -- `src/backend/workflows/workflow_purity_report.py` -- `scripts/workflow_purity_report.py` -- `src/backend/workflows/workflow_concept_authority_service.py` -- `src/backend/workflows/durable/workflow_creation_workflow.py` -- `src/backend/workflows/durable/workflow_gap_recovery_workflow.py` -- `src/backend/workflows/durable/file_copy_upload_classification_workflow.py` -- `src/backend/workflows/durable/testing_workflow_actions.py` -- `src/backend/workflows/workflow_selector.py` -- `src/backend/services/workflow_discovery_service.py` -- `src/backend/services/workflow_override_policy_service.py` -- `src/backend/services/paper_representation_workflow_vontology_service.py` -- `src/backend/services/talk_representation_workflow_vontology_service.py` -- `src/backend/services/testing_workflow_vontology_service.py` -- `src/backend/services/experiment_run_service.py` - -The audit also used `workflow_list_definitions(limit=200)` to cross-check the current authoritative registry state. - -## Baseline Interpretation - -The workflow-purity counters are accurate for runtime purity and now also flag Python-authored canonical workflow publication surfaces via `python_authored_canonical_workflow_source_count`, Python-authored workflow prompt sources via `python_authored_workflow_prompt_source_count`, repo-seed authority drift via `repo_seed_authority_drift_path_count`, and guarded Vontology-first seed-fallback regressions via `vontology_first_seed_fallback_violation_count`. - -They still do not classify every possible hard-coded workflow-adjacent builder outside the canonical publication/template/prompt authority paths, so the report remains a strong regression gate rather than a proof that no further design opportunities exist. - -`JVNAUTOSCI-1584` complements those counters by making the authoritative KB state easier to inspect directly. That matters because ergonomics was part of the original failure mode: if Vontology-first review is awkward, developers and agents will keep drifting back toward file-first local authority. - -So the zeroed runtime-authority counters plus zero canonical-source, prompt-source, and repo-seed-drift counters should now be read as "runtime authority is Vontology-backed, canonical publication sources are no longer Python-authored, workflow-governed prompt bodies are no longer Python-authored, repo-seed authority drift is mechanically guarded, the repo names the remaining filesystem artefacts honestly as seed bundles, and typed-subworkflow plus testing-workflow policy can be authored as authoritative Vontology metadata". They should still not be read as "all future workflow design work is done", because generic capability evolution can still reveal further reusable VWL/runtime extensions. - -## Exhaustive Findings - -| Family | Locations | Current behaviour | Example workflows / task classes | Current VWL sufficient? | Equivalent VWL representation or smallest coherent extension | Other examples covered | Delete / migrate / retain | Migration risk, dependencies, priority | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | -| Canonical workflow publication authority | Historical locations before `JVNAUTOSCI-1576`: `src/backend/workflows/workflow_concept_authority_service.py`; `src/backend/services/paper_representation_workflow_vontology_service.py`; `src/backend/services/talk_representation_workflow_vontology_service.py`; `src/backend/services/testing_workflow_vontology_service.py`. Transitional repo seed bundles now live under `src/backend/workflows/repo_seed_bundles/*.json`. | Resolved by `JVNAUTOSCI-1581` and `JVNAUTOSCI-1585`. Canonical workflow publication now derives specs and workflow text from authoritative Vontology workflow state first, and the remaining filesystem artefacts are explicitly named as repo seed bundles rather than authored sources. | Chat narration, buttonify, tool-calling, turn execution, file-copy typing/upload/interpretation, planning, rumination, workflow authoring, scholarly/arXiv paper workflows, talk workflows, testing workflows. | Yes. | Keep the authoritative workflow definition as Vontology workflow graph data and retain only generic import, validation, publication, or export helpers in Python. Repo-side files remain seed-only. | Any future canonical workflow family. | Mark `JVNAUTOSCI-1581` and `JVNAUTOSCI-1585` done for authority migration plus repo-affordance cleanup. Retain generic publication/import/validation utilities. | Low residual risk. The remaining risk is future regression, which is why `JVNAUTOSCI-1583` and `JVNAUTOSCI-1584` remain important. | -| Code-authored workflow prompts | Historical locations before `JVNAUTOSCI-1580`: `src/backend/services/testing_workflow_vontology_service.py`; `src/backend/services/workflow_gap_vontology_service.py`; `src/backend/services/parent_specificity_vontology_service.py`; `src/backend/services/workflow_description_vontology_service.py`; `src/backend/workflows/durable/workflow_gap_recovery_workflow.py`; `src/backend/workflows/durable/planning_workflow.py`; `src/backend/workflows/durable/enrichment_workflow.py`. | Resolved by `JVNAUTOSCI-1580`. Workflow-governed prompt bodies now live in authoritative Vontology prompt concepts, while Python retains only generic prompt validation/linking/rendering infrastructure and fail-closed runtime behaviour when prompt concepts are missing or empty. | Meeting-invitation structure/evaluation prompts, workflow-gap analysis/test/candidate prompts, parent-specificity analysis prompt, concept/workflow description prompts, planning prompt, enrichment prompt resolution. | Yes. | Store prompt bodies directly in Vontology prompt concepts and treat Python only as prompt resolution/rendering infrastructure. | Future workflow-repair prompts, authoring prompts, test-fixture prompts, planning prompts, enrichment prompts. | Mark `JVNAUTOSCI-1580` done. Retain generic prompt rendering, validation, workflow-linking, and fail-closed diagnostics support. | Low residual risk. The main remaining risk is regression back to Python-authored prompt bodies, now guarded by `python_authored_workflow_prompt_source_count`. | -| Workflow templates in workflow creation and gap recovery | Historical locations before `JVNAUTOSCI-1577`: `src/backend/workflows/durable/workflow_creation_workflow.py`; `src/backend/workflows/durable/workflow_gap_recovery_workflow.py`. Transitional repo seed fixture now lives at `src/backend/workflows/repo_seed_bundles/workflow_template_seed_bundle.json`. | Resolved by `JVNAUTOSCI-1581` and `JVNAUTOSCI-1585`. Template selection/rendering now resolves authoritative metadata from first-class Vontology template concepts and text relations, with repo seed hydration only when the authoritative template concepts are missing. | Text-driven workflow creation for scholarly-paper representation, PhD-student representation, single-step recovery candidate workflows. | Yes, for the covered cases. | Store reusable authored workflow-spec templates directly in Vontology-native structures and resolve them through generic template-profile selection/rendering support. Repo-side files remain seed-only. | User-onboarding workflows, new representation workflows, future recovery-generated candidate workflows. | Mark `JVNAUTOSCI-1581` and `JVNAUTOSCI-1585` done for template-authority migration and repo-affordance cleanup. Retain the generic template loader/selector/renderer and fail-closed synthesis-policy checks. | Low residual risk. The main remaining risk is future regression back to file-authority paths rather than a missing capability. | -| Workflow discovery seeds and override-role heuristics | Historical workflow-family seed location before `JVNAUTOSCI-1577`: `src/backend/services/workflow_discovery_service.py`. Remaining generic fallback: `src/backend/services/workflow_override_policy_service.py`. Transitional repo seed metadata now lives in `src/backend/workflows/repo_seed_bundles/canonical_workflow_publication_seed_bundle.json` and `src/backend/workflows/repo_seed_bundles/testing_workflow_seed_bundle.json`. | Resolved for authority by `JVNAUTOSCI-1581`, with repo-affordance cleanup completed by `JVNAUTOSCI-1585`. Capability discovery now resolves routing profiles and discovery exemplars from authoritative Vontology text relations; repo-side metadata remains only as seed fallback when the authoritative workflow concepts are absent or incomplete. Override policy still retains a minimal generic fallback for legacy workflows that lack explicit routing metadata. | Workflow creation requests, workflow repair/meta workflow promotion, workflow introspection maintenance requests. | Yes, for the migrated cases. | Publish `workflow_routing_profile.v1` and `workflow_discovery_exemplars.v1` metadata directly from Vontology-native source data; incorporate exemplar text into authoritative capability-index text. Keep only minimal generic fallback while legacy workflows are backfilled. | Workflow repair, workflow introspection, testing maintenance workflows, future maintenance/meta workflows. | Mark `JVNAUTOSCI-1581` and `JVNAUTOSCI-1585` done for metadata authority migration plus repo-affordance cleanup. Retain the generic legacy fallback only until metadata coverage is complete. | Low-medium residual risk. The remaining risk is legacy workflows without backfilled routing metadata plus future reintroduction of file-authority shortcuts. | -| Hard-coded typed-subworkflow route maps in file-copy upload classification | Historical locations before `JVNAUTOSCI-1578`: `src/backend/workflows/durable/file_copy_upload_classification_workflow.py`; `src/backend/workflows/workflow_concept_authority_service.py`. Authoritative route-map metadata is now published on `#V#file_copy_upload_classification_workflow` via `#V#hasWorkflowTypedSubworkflowRouteMapJson`. | Resolved by `JVNAUTOSCI-1578`. Python now retains generic file typing, score selection, and route execution support, while candidate workflow lists, threshold defaults, and unavailable/low-confidence fallback semantics resolve from authoritative Vontology metadata. | Upload -> scholarly paper workflow, upload -> CV workflow, upload -> business-card workflow, upload -> meeting workflow. | Yes. | Keep the declarative route-map surface generic so future artefact-routing workflows can publish their own route maps without adding workflow-family-specific Python. | URL-ingestion routing, document subtype dispatch, future structured artefact-routing pipelines. | Mark `JVNAUTOSCI-1578` done, retain the generic loader/classifier support, and keep classification fail-closed when route-map metadata is missing or invalid. | Low-medium residual risk. The remaining risk is future drift back to code-owned route maps or incomplete metadata publication in new workflow families. | -| Testing-workflow-specific fixture preparation and benchmark escalation | Historical locations before `JVNAUTOSCI-1579`: `src/backend/workflows/durable/testing_workflow_actions.py`; `src/backend/services/experiment_run_service.py`; `src/backend/services/testing_workflow_vontology_service.py`. Canonical testing workflows now publish `testing_experiment_scenario_template.v1` and `testing_regression_suite_policy.v1` metadata via `hasInputMap` JSON payloads. | Resolved by `JVNAUTOSCI-1579`. Meeting fixture preparation now goes through the generic `testing.prepare_experiment_spec` runtime surface, and regression-suite tier/escalation behaviour resolves from workflow-authored suite policy metadata rather than Python-owned benchmark branching. The legacy meeting helper remains only as a compatibility alias. | Meeting-invitation testing workflow, synthetic workflow regression suite, future acceptance tests for arXiv/talk/file-copy flows. | Yes. | Keep the scenario-template and suite-policy semantics generic so future testing workflows can author their own fixtures and escalation rules in VWL/Vontology. | arXiv paper-ingestion testing, talk-representation testing, workflow-gap candidate evaluation, future benchmark-backed acceptance workflows. | Mark `JVNAUTOSCI-1579` done and retain only the generic runtime/template-resolution support. Do not reintroduce workflow-specific Python branches for new testing scenarios. | Low-medium residual risk. The remaining risk is drift back to workflow-specific helpers instead of publishing scenario/policy metadata, not a missing capability. | - -## What Should Remain as Reusable Runtime Support - -These areas are not themselves contract failures and should not be deleted merely because they are written in Python: - -- `publish_canonical_chat_workflow_graphs(...)` and related validation/import helpers in `workflow_concept_authority_service.py` should remain as generic publication infrastructure. The problem is that workflow-family specs still live beside them in Python. -- Generic testing-theory and experiment-run actions in `src/backend/workflows/durable/testing_workflow_actions.py` are reusable execution surfaces, not workflow-specific drift. -- Deterministic paper/talk materialisation actions remain legitimate runtime primitives until VWL gains a general, declarative concept-mutation-and-verification surface. -- File typing and route execution are reusable runtime support. The route-map policy is the part that should move into VWL. - -## Prioritised Extension and Migration Plan - -1. `P0`: Completed 2026-03-24 via `JVNAUTOSCI-1582`: correct the workflow-authority contract so repo-side declarative workflow files are no longer counted as compliant authoritative VWL/Vontology assets. -2. `P1`: Completed 2026-03-24 via `JVNAUTOSCI-1581`: move canonical published workflow definitions, workflow-template authority, routing metadata, and discovery exemplars out of repo-authored bundles and into Vontology-native authoritative sources, leaving repo bundles seed-only. -3. `P1`: Completed 2026-03-25 via `JVNAUTOSCI-1578`: add declarative typed-subworkflow route maps and migrate file-copy upload classification policy out of Python. -4. `P1`: Completed 2026-03-25 via `JVNAUTOSCI-1579`: generalise testing-workflow fixture preparation and test-tier escalation semantics so meeting-specific and benchmark-specific logic no longer needs to reappear in Python. -5. `P2`: Completed 2026-03-25 via `JVNAUTOSCI-1580`: move remaining workflow prompt text out of Python and into Vontology prompt concepts as first-class authored artefacts. -6. `P2`: Completed 2026-03-24 via `JVNAUTOSCI-1585`: rename and quarantine repo seed bundle paths, filenames, schema identifiers, and helper names so the repo layout itself no longer advertises file-first workflow authority. -7. `P2`: Completed 2026-03-25 via `JVNAUTOSCI-1583`: add source-authority purity gates so repo-seed authority drift and Vontology-first fallback regressions now fail deterministically. -8. `P2`: Completed 2026-03-25 via `JVNAUTOSCI-1584`: add Vontology-authored snapshot/diff tooling so the stronger authority model remains easy to inspect without drifting back to file-first authoring. - -## Residual Uncertainty - -This audit was exhaustive across the current workflow runtime, authoring, discovery, routing, and testing modules under `src/backend/workflows` and `src/backend/services/*workflow*`, plus adjacent services used directly by those modules. - -Residual uncertainty remains in one narrow area: - -- Generic concept-mutation services were not audited line-by-line unless a workflow module depended on them directly. There may still be future opportunities to turn some domain-specific durable actions into more general VWL mutation primitives. - -No additional workflow-family-specific authoring surfaces were found by broad searches over: - -- `_CanonicalWorkflowPublicationSpec` -- `_build_*workflow_spec` -- `publish_canonical_chat_workflow_graphs` -- `routing_profile` -- `specialised_workflow_ids` -- workflow-gap candidate-spec builders - -## Guidance Updates - -`JVNAUTOSCI-1582` now justifies further `AGENTS.md` and manual changes. The durable additions are: - -- a named warning against repo-declarative masquerading as KB-first; -- an explicit no-self-serving-policy-relaxation rule for workflow-authority tasks; -- a workflow-authority readiness/closure checklist requiring the authoritative KB artefacts to be named up front and requiring closure checks against permissive docs drift and repo-file dependency. - -The next durable documentation changes should happen when the remaining extension families above land, at which point `docs/engineering/von_workflow_language_manual.md` should be updated further with: - -- typed-subworkflow route-map semantics -- testing-fixture preparation and tier-escalation semantics -- generated snapshot/export guidance once `JVNAUTOSCI-1584` lands -- prompt-source discipline and fail-closed workflow prompt semantics, now landed via `JVNAUTOSCI-1580` - -The parent epic `JVNAUTOSCI-833` should continue to track this report plus the follow-up migration/guardrail tasks created from it. diff --git a/docs/engineering/workflow_audit_2026-02-07.md b/docs/engineering/workflow_audit_2026-02-07.md deleted file mode 100644 index ce964638a..000000000 --- a/docs/engineering/workflow_audit_2026-02-07.md +++ /dev/null @@ -1,125 +0,0 @@ -# Workflow System Audit — 2026-02-07 - -> **Document status: Frozen audit snapshot from 7 February 2026.** The fixes, -> gaps, Jira states, and implementation anchors below describe that session, -> not current workflow behaviour. Use the VWL manual and live -> Vontology/code/telemetry evidence for new work. - -## Summary - -This audit reviews the recent workflow system changes in the context of Jira tasks -and Confluence design documents, with focus on Vontology-first compliance and -correct functioning of background/rumination and conversation-turn workflows. - -## Code Changes Made During This Session - -### 1. Recursive Subtype Discovery (`vontology_loader.py`) -- Added `_get_recursive_subtypes()` — BFS traversal to find all transitive subtypes - of `#V#durable_workflow` and `#V#workflow` -- `discover_workflow_ids()` now finds workflows typed as subtypes-of-subtypes -- This fixed the discovery of `#V#sail_phd_student_onboarding_workflow` which was - typed as a subtype of `#V#workflow` (indirect path through the type hierarchy) - -### 2. Skip-Duplicate Logic (`registry_factory.py`) -- `build_durable_workflow_registry()` now checks `registered_ids` before loading - Vontology workflows, preventing `ValueError` on duplicate registration -- Cleaned up verbose exploratory comments (8 lines of LLM reasoning traces) and - removed a stray `pass` statement - -### 3. Async Handler Bug Fix (`considerations_workflow.py`) -- `_handle_process_batch` was declared `async def` but `ActionRegistry.execute()` - calls handlers synchronously via `spec.handler(request)` -- This caused the handler to return an unawaited coroutine object instead of - executing its body — effectively a silent no-op -- Fixed by converting to synchronous `def` with improved docstring - -### 4. Vontology Concept Bootstrap -- Created 7 workflow concepts in the database as instances of `#V#durable_workflow`: - - `#V#chat_assistant_workflow` - - `#V#chat_narration_workflow` - - `#V#generate_considerations_workflow` - - `#V#missing_tool_call_workflow` - - `#V#rag_text_relation_sync_workflow` - - `#V#todo_refresh_workflow` - - `#V#write_tool_policy_workflow` -- These were missing from Vontology, causing "non-existent" cartouches in the UI - -## Architecture: Two Execution Domains - -### Conversation-Turn Workflows (synchronous, in-process) - -Run inline during chat turns via `WorkflowExecutor` in the MCP orchestrator. -Each orchestrator instance builds its own `ActionRegistry`. - -| Workflow | Purpose | Status | -|---|---|---| -| `#V#write_tool_policy_workflow` | Gate write-tools per prompt | Active | -| `#V#missing_tool_call_workflow` | Recover missing tool calls | Active | -| `#V#chat_narration_workflow` | TTS narration | Active | -| `#V#chat_assistant_workflow` | No-op placeholder | Placeholder | -| `#V#todo_refresh_workflow` | Refresh to-dos | All no-ops | - -### Durable/Background Workflows (persistent, checkpointed) - -Run via `DurableWorkflowExecutor` in a background worker thread. -Separate `ActionRegistry` with durable-specific handlers. - -| Workflow | Purpose | Status | -|---|---|---| -| `#V#rag_text_relation_sync_workflow` | Batch RAG sync | Implemented | -| `#V#generate_considerations_workflow` | Auto-generate considerations | Implemented (async bug now fixed) | - -## Vontology-First Assessment - -### What's Good -- All workflows have Vontology concept identities (instance of `#V#durable_workflow`) -- `vontology_loader.py` can load full process-graph definitions from the ontology -- `discover_workflows_for_turn()` searches Vontology for contextually relevant workflows -- The `#V#sail_phd_student_onboarding_workflow` demonstrates full Vontology-native - workflow definition (steps, transitions, actions all as concepts) - -### Gaps -1. **Dual-source architecture**: Workflow *behaviour* (states, transitions, actions) - lives in Python code, not in the ontology. The ontology only stores - identity/typing for code-defined workflows. -2. **Code takes precedence**: `build_durable_workflow_registry()` registers code - workflows first, then skips matching Vontology workflows. -3. **Action handler discovery**: No mechanism to associate Vontology `invokesAction` - references with code implementations. -4. **Discovery is informational**: `discover_workflows_for_turn()` results appear - in debug metadata only — no auto-execution. - -### Recommended Evolution -1. Store workflow step/transition structure in Vontology -2. Register action handlers by canonical IDs matching Vontology `invokesAction` targets -3. Eventually invert precedence: Vontology definitions override code definitions - -## Known Issues - -1. **Two separate ActionRegistry instances** — no handler sharing between domains -2. **`todo_refresh_workflow` fully stubbed** — all 5 actions are no-ops -3. **`chat_assistant_workflow` is a no-op** — single terminal state placeholder -4. **Mixed MCP listing** — `workflow_list_definitions` shows all workflows including - those that can't be executed by the durable worker -5. **Durable system disabled by default** — requires `VON_DURABLE_WORKFLOWS_ENABLE=1` -6. **Pre-existing pyright errors** in `considerations_workflow.py` (None-safety, - `max_tokens` parameter) - -## Jira Task Status - -| Issue | Summary | Status | Notes | -|---|---|---|---| -| JVNAUTOSCI-803 | Design and implement LLM Workflows | In Progress | Parent epic | -| JVNAUTOSCI-1054 | MCP Agentic Workflow Enhancement | To Do | | -| JVNAUTOSCI-1075 | Durable Workflow System | Done (Phase 1-3) | Phase 4-5 pending | -| JVNAUTOSCI-1076 | Workflow discovery | Done | | -| JVNAUTOSCI-1077 | Student Onboarding Workflow | Done | | -| JVNAUTOSCI-1082 | Vontology-driven scheduling | To Do | | -| JVNAUTOSCI-1083 | MCP Tools for Workflow Management | In Progress | 12 tools implemented | - -## Confluence Pages Updated - -- **Von Agentic Workflows Design** (page 87949313) — pending update with - architecture status, bootstrapped concepts table, known gaps, recent fixes -- **How to design and use Von WorkFlows** (page 96862209) — no changes needed - (already covers design principles and lifecycle) diff --git a/docs/engineering/workflow_authoritative_publication_process.md b/docs/engineering/workflow_authoritative_publication_process.md deleted file mode 100644 index b73647117..000000000 --- a/docs/engineering/workflow_authoritative_publication_process.md +++ /dev/null @@ -1,165 +0,0 @@ -# Workflow Authoritative Publication Process - -> **Document status: Dated implementation note from 20 February 2026.** The -> Vontology-authority invariant remains applicable. The named bootstrap and -> publication functions below are historical implementation anchors, not the -> exclusive current authoring procedure. Follow the current VWL manual, live -> Vontology tooling, and authority read-back for new work. - -- **Kind:** Implementation note with a retained authority invariant -- **Lifecycle:** Frozen -- **Authority:** Advisory implementation detail; the authority invariant is - governed by `AGENTS.md` - -Date: 2026-02-20 -Related: `JVNAUTOSCI-803`, `JVNAUTOSCI-1217` - -## Authority Rule - -Workflow definition authority is the Vontology graph. - -Runtime registries, traces, and monitor payloads are execution reflections of that authority, not a parallel source of truth. - -## Canonical Publication Path - -Use the existing canonical pathway in `src/backend/workflows/workflow_concept_authority_service.py`: - -1. `bootstrap_workflow_concepts(...)` -2. `publish_canonical_chat_workflow_graphs(...)` - -`publish_canonical_chat_workflow_graphs(...)` now applies strict post-publication validation before a workflow is counted as published. - -For explicit Vontology-authored workflow families such as paper/talk -representation, use `publish_canonical_chat_workflow_graphs(...)` with explicit -publication specs/definitions/purposes rather than creating temporary -`source="built_in"` runtime registrations. The runtime registry must consume the -published Vontology authority; it must not materialise those families on the -production registry path. - -## Strict Validation Gates - -`validate_workflow_definition_contract(...)` in `src/backend/workflows/workflow_definition_identity_service.py` enforces: - -1. Non-vacuous step contracts. -2. Control-flow completeness for action-bearing states. -3. Transition target resolvability. -4. Input/output/context mapping validity (including unresolved/invalid mapping specs). -5. Action resolvability when supported-action sets are supplied. - -If any gate fails, publication is rejected for that workflow with actionable reason codes under: - -- `errors_by_workflow_id` -- `validation_failures_by_workflow_id` - -## Runtime Gate - -Durable instance submission preflight/postflight (`verify_workflow_runnable(...)`) now includes strict contract validation and definition identity payloads. - -Unrunnable workflows are rejected with structured errors and no optimistic success status. - -## Definition Identity in Introspection - -Definition identity payloads (`workflow_definition_identity.v1`) are exposed on: - -1. REST definitions list/detail: - - `/api/workflows/definitions` - - `/api/workflows/definitions/` -2. Internal MCP: - - `workflow_list_definitions` -3. Execution telemetry: - - workflow episode metadata - - turn execution runtime snapshots - - workflow traces (when trace storage is enabled) - -Identity payloads include version, source, and deterministic hash fields for traceability and drift detection. - -## CI Drift Check - -Canonical workflow runtime-vs-authority drift is enforced by tests in: - -- `tests/backend/test_workflow_concept_authority_service.py` - -The CI check asserts: - -1. Canonical workflow registrations are `source=vontology`. -2. Runtime definition hash matches authoritative Vontology-loaded definition hash. - -Any mismatch fails CI and must be resolved by updating the Vontology graph/publication path, not by adding fallback code paths. - -## Workflow Purity Scoreboard - -`JVNAUTOSCI-1524` extends the parity inventory with a structured -`workflow_purity` report that tracks the remaining hybrid authority surface. - -The report is attached to the existing `parity_inventory` payloads and logged at -startup under the `workflow_purity` logger key. The counters currently include: - -1. `built_in_registration_count` -2. `remaining_python_workflow_family_count` -3. `direct_instance_create_callsite_count` -4. `env_event_binding_count` -5. `legacy_selector_mode_count` -6. `builtin_capability_override_count` -7. `non_vontology_discoverable_workflow_count` - -Parity enforcement defaults to `fail`, not `warn`, for executable-registry and -authority defects: registry-only or Vontology-only workflow IDs, incomplete -represented process graphs, missing authority concepts or types, and purity -counters that identify a workflow source or policy violation. The deferred -parity policy marks those defects failed and raises within its worker unless a -developer explicitly relaxes `VON_WORKFLOW_PARITY_ENFORCEMENT` for bounded -local diagnostics. The current deferred worker catches and records that -failure; it does not retroactively make the already-running server unready. - -Two kinds of engineering debt are reported but are not runtime-readiness -failures: - -- `synthesized_launch_contract_count`, including the exact sorted workflow IDs; -- `monolith_line_count_*` source-size ratchets. - -An otherwise healthy registry with only those findings completes parity work as -`completed_with_findings` and emits a `workflow_parity_advisory` warning. This -keeps the debt visible without making a runnable workflow registry unavailable -because of code-maintenance metrics. The standalone purity gate remains strict -and returns non-zero for the authority and policy counters above; callers may -still choose a warn-only integration policy. - -The runtime registry must also keep bootstrap-only workflow families separate -from production registration authority. Python definitions that remain for -bootstrap/test support, such as the file-copy family during convergence, may be -used to publish or repair authoritative Vontology graphs, but they must not be -counted as discoverable production registrations. - -Capability indexing now also fails closed for routing: the workflow capability -index should only index `source=vontology` workflows with non-empty -authoritative narrative text. Non-authoritative registrations and textless -workflows must be skipped rather than receiving guessed fallback routing prose. - -The checked-in CI baseline lives at: - -- `tests/backend/fixtures/workflow_purity_baseline.json` - -The checked-in baseline is repository-observable and may be generated without -a populated runtime registry. It therefore does not record an artificial zero -for registry-dependent measurements such as synthesised launch contracts, nor -does it use them as a historical ratchet. A complete live non-zero observation -is reported separately as present advisory debt, with exact workflow IDs; an -incomplete or empty observation is merely incomparable. Neither case is -misreported as a static-baseline regression. - -To emit the current report locally: - -```powershell -pdm run python scripts/workflow_purity_report.py -``` - -To refresh the baseline deliberately after an approved convergence change: - -```powershell -pdm run python scripts/workflow_purity_report.py --refresh-baseline -``` - -Refreshing the baseline is a conscious workflow-first migration step. Do it only -when the new counter values reflect intended authority reduction or an approved -re-baselining decision, and commit the updated JSON in the same change as the -code that justified the shift. diff --git a/docs/engineering/workflow_execution_architecture_analysis.md b/docs/engineering/workflow_execution_architecture_analysis.md deleted file mode 100644 index be4c45c30..000000000 --- a/docs/engineering/workflow_execution_architecture_analysis.md +++ /dev/null @@ -1,411 +0,0 @@ -# Von Workflow Execution Architecture: Intended Design vs Current Reality - -> **Document status: Dated diagnostic snapshot; not a current implementation -> guide.** Present-tense claims, source line numbers, workflow inventories, and -> Jira conclusions describe evidence available on 7 April 2026. Architectural -> questions may remain useful, but revalidate every factual claim against -> current code, live Vontology, telemetry, and Jira before implementation. - -- **Kind:** Diagnostic analysis -- **Lifecycle:** Frozen -- **Authority:** Evidence only; not a specification -- **State as of:** 2026-04-07 -- **Author:** Analysis prepared for Michael Witbrock -- **Trigger:** Observed failure-to-communicate in a live arXiv paper ingestion - turn (session `d57ab88b`, request `51a65483`), where the system silently - created a file copy and a paper concept without telling the user - ---- - -## 1. Executive Summary - -Von's intended execution architecture is a layered, LLM-supervised, KB-authoritative workflow system. The user's chat message enters a conversation turn workflow; an LLM analyses the intent and selects an appropriate VWL workflow from Vontology; that workflow is instantiated and executed step by step, with each step guided by the Vontology-authored workflow definition; and within each step the system may use LLM reasoning, MCP tool calls, or (in the future) skill invocations. At every layer, the LLM remains a reasoning participant, able to narrate, explain, and communicate what is happening. - -The current implementation subverts this vision in several important ways. At the orchestrator level, a dispatch decision (line 22306 of `orchestrator.py`) silently promotes discovered workflows to a "custom workflow" execution mode, handing them off to a deterministic Python state machine that runs the entire workflow without LLM supervision between steps. The result is that whole categories of user-facing work — paper ingestion, entity representation, talk/meeting materialisation — execute as opaque Python pipelines. The user sees a narration of the *output* but has no visibility into what the system did, no confirmation that concepts or file copies were created, and no opportunity to guide the process. - -This paper documents the intended design hierarchy, the current implementation reality, every identified point where Python monolithic code subverts the vision, and what the architectural consequences are. - ---- - -## 2. The Intended Execution Hierarchy - -The intended architecture is a five-layer hierarchy. Each layer has a clear role, and the LLM is present at the top level and available within any layer that requires reasoning. - -### Layer 1: Chat Turn Workflow - -The top-level construct. A user message arrives at `/von/generate`. The system enters a KB-defined conversation turn workflow (`#V#conversation_turn_execution_workflow`). This workflow governs the entire turn lifecycle: context building, workflow discovery, dispatch, execution, response composition, narration. - -**Authority**: This workflow is defined in Vontology and represented as a VWL workflow concept. Its stages and transitions are authored KB artefacts, not hardcoded Python logic. - -### Layer 2: LLM Analysis and Dispatch - -Within the conversation turn workflow, an LLM analyses the user's message. It may: - -- **Answer directly** — when no workflow is needed (a question, a greeting, simple factual recall). -- **Select an initial workflow** — discover a matching VWL workflow from the capability index, evaluate its fitness, and dispatch to it. -- **Use tools directly** — invoke MCP tools without a formal workflow, for ad hoc one-off actions. -- **Invoke a skill** (future) — delegate to a reusable structured skill definition. - -The key property of this layer is that the LLM is the decision-maker. It reasons about the user's intent, evaluates candidates, and chooses a path. This reasoning is visible in telemetry and can produce explanatory output to the user. - -### Layer 3: Workflow Instantiation - -When the LLM selects a workflow, that workflow is loaded from Vontology via the VWL loader (`vontology_loader.py`). The workflow definition — its steps, transitions, preconditions, effects, prompt contracts, action contracts — all come from the KB. The definition is compiled into an executable state machine (`WorkflowDefinition` with `WorkflowStateSpec` objects). - -This instantiation should be visible. The user should know that a workflow was selected, what it is called, and what it intends to do. - -### Layer 4: Step-by-Step Workflow Execution - -Each step in the instantiated workflow executes according to its authored execution mode: - -- **`llm` mode**: The step uses an LLM to reason, generate, or decide. The LLM receives the step's prompt contract (from Vontology) and the current workflow context. It may call tools, produce outputs, or branch the workflow. -- **`deterministic` mode**: The step executes a reliable, pre-defined action (e.g., a database write, an API call, a validation check). No LLM is needed because the operation is mechanically reliable. -- **`control` mode**: Flow control primitives — branching, looping (`for_each`), forking, joining. -- **`subworkflow` mode**: Delegates to a child workflow. - -The critical design property: **the workflow graph (which steps, in what order, with what transitions) is authored in Vontology. The LLM does not choose the next step — the VWL definition does.** But within each step, the LLM may be the actor (for `llm`-mode steps), and between steps the system should be able to communicate progress. - -### Layer 5: Tool and Skill Use Within Steps - -Within any step (whether LLM-driven or via a registered action handler), the system may: - -- Call MCP tools via the internal gateway. -- Use LLM generation for reasoning or text production. -- (Future) Invoke skill definitions for structured multi-step sub-procedures. - -### Why This Hierarchy Is a Good Idea - -This hierarchy is architecturally sound for several reasons: - -1. **KB-authoritative workflow definitions** mean that the organisation's actual operational processes are represented explicitly, versionable, inspectable, and modifiable without changing Python code. New workflows can be authored, published, validated, and retired entirely through knowledge engineering. - -2. **LLM supervision at the top** means the system can understand user intent in natural language, select appropriate workflows dynamically, handle edge cases, and communicate. The LLM is not replaced by the workflow — it *supervises* the workflow. - -3. **Deterministic execution for reliable steps** means that once a workflow step is known to be mechanically reliable (e.g., "normalise this arXiv URL", "write this metadata to the database"), it does not need to waste an LLM call. The workflow definition distinguishes reasoning from routine. - -4. **LLM-mode steps within workflows** mean that when a step genuinely requires reasoning (e.g., "assess whether this paper's metadata is complete", "decide whether to create a new author concept or link to an existing one"), the LLM can be brought in at exactly the right points, scoped to exactly the right prompt contract. - -5. **Visibility and narration** are natural consequences. Because the LLM is present at the top level, it can narrate what the workflow did. Because the workflow steps are declared with names, descriptions, and output contracts, the system has the vocabulary to explain what happened. - -6. **The separation of authored policy from execution support** is the fundamental AGENTS.md principle (`§4`). Workflows, prompts, routing profiles, and discovery exemplars are policy artefacts authored in Vontology. Python provides the execution engine, validators, telemetry, and tool implementations. When this separation holds, the system is genuinely a neuro-symbolic agentic architecture rather than a Python application with some metadata attached. - ---- - -## 3. How It Currently Works: The Reality - -### 3.1 The Custom Workflow Bypass - -The single most consequential deviation from the intended architecture occurs at line 22306 of `orchestrator.py`: - -```python -selector_requests_custom_workflow = bool( - has_explicit_workflow_routing - and selected_workflow_id_text - and not selected_prefers_direct_response - and not selected_uses_tool_pipeline_contract -) -``` - -When this flag is `True`, the orchestrator bypasses the LLM-driven tool-calling path entirely and dispatches directly to the workflow engine: - -```python -# Line ~27708 -if ( - selector_requests_custom_workflow - and selected_workflow_id_text - and not selected_uses_tool_pipeline_contract -): - wf_result = self.execute_workflow( - selected_workflow_id_text, - data=_build_custom_workflow_dispatch_data(), - ... - ) -``` - -This means: **any workflow that has a VWL definition with registered Python action handlers will be executed as a closed, deterministic pipeline without LLM participation between steps.** - -The conditions for this bypass are broadly met. Any workflow that: -- was discovered by the capability index (which sets `has_explicit_workflow_routing`), -- has a non-empty concept ID (which sets `selected_workflow_id_text`), -- does not explicitly opt into tool-pipeline mode, -- does not explicitly prefer a direct response, - -...will be dispatched as a custom workflow. This is the default path for all materialisation workflows (papers, entities, talks), evaluation workflows, and many utility workflows. - -### 3.2 The Deterministic State Machine - -Once dispatched as a custom workflow, execution enters `WorkflowExecutor.run()` in `engine.py` (line 1251). This executor is a **purely deterministic state machine**: - -1. It starts at the workflow's initial state. -2. For each state, it executes all declared actions sequentially. -3. It evaluates transition conditions to determine the next state. -4. It loops until a terminal state or the maximum transition count. - -**The executor never asks the LLM "what should I do next?"** The sequence is entirely pre-determined by the Vontology workflow definition. This is correct — the VWL manual specifies that workflow structure is authored, not improvised. But the consequence is that the *entire workflow* runs without any LLM-mediated communication to the user. - -### 3.3 Registered Python Action Handlers - -Each workflow step's action is resolved by looking up a registered Python handler in the `ActionRegistry`. The `_register_durable_action_modules()` function in `registry_factory.py` (line 1122) registers handlers from **20 workflow-family modules**: - -| Module | Action IDs | LLM Usage | -|--------|-----------|-----------| -| `paper_representation_workflow.py` | 9 actions (`scholarly_paper.*`, `arxiv.*`) | None | -| `entity_representation_workflow.py` | 1+ actions | None | -| `talk_representation_workflow.py` | 3 actions (`talk.*`) | None | -| `entity_identity_resolution_workflow.py` | Multiple | None | -| `file_copy_typing_workflow.py` | 2 actions | None | -| `file_copy_upload_classification_workflow.py` | Multiple | None | -| `file_copy_upload_handler_workflow.py` | Multiple | None | -| `file_copy_interpretation_workflow.py` | Multiple | None | -| `jira_task_incremental_import_workflow.py` | 1 action | None | -| `jira_task_full_reconciliation_workflow.py` | Multiple | None | -| `rag_sync_workflow.py` | 4 actions | None | -| `workflow_creation_workflow.py` | Multiple | None (metadata tracks `llm_policy`) | -| `workflow_gap_recovery_workflow.py` | Multiple | None | -| `workflow_introspection_maintenance_workflow.py` | Multiple | None | -| `episode_evaluation_workflow.py` | Multiple | None | -| `parent_specificity_concept_dossier_workflow.py` | Multiple | None | -| `parent_specificity_rumination_workflow.py` | Multiple | None | -| `planning_workflow.py` | 4 actions | **Yes** — `planning.infer_plan` calls LLM | -| `enrichment_workflow.py` | 4 actions | **Yes** — `process_batch` calls LLM | -| `rumination_workflow.py` | Multiple | **Yes** — Tracks LLM call budget | - -Of the 20 workflow families, **only 3 use LLM calls within their action handlers**. The other 17 are entirely deterministic Python code. This means the vast majority of custom-dispatched workflows run as pure Python pipelines with no LLM reasoning, no user communication, and no opportunity for the user to observe or guide the process. - -### 3.4 The MCP Fallback Handler - -There is a design escape hatch: `_durable_mcp_fallback_action()` in `registry_factory.py` (line 280). When a workflow step references an action ID that has no registered Python handler, it falls back to invoking the equivalent MCP tool through the gateway. This allows Vontology-authored workflows to reference MCP tools without explicit Python wiring. - -This fallback is architecturally correct — it means that new workflow steps can be authored in Vontology and immediately executable by referencing existing MCP tools. But it is rarely exercised for the established workflow families, which all have dedicated Python handlers. - -### 3.5 The Response Assembly Gap - -After a custom workflow completes, the orchestrator attempts to extract user-visible text from the workflow result via `_render_custom_workflow_response_text()` (line 3773). This function looks for fields like `response_text`, `summary`, or `final_response` in the workflow's output data. It then optionally feeds this text through a narration workflow (`#V#chat_narration_workflow`) to produce spoken audio. - -The problem: **the response extraction is a best-effort scrape of the workflow's output dictionary**. Most materialisation workflows do not produce a `response_text` field designed for the user. They produce structured data (concept IDs, metadata dictionaries, status flags). The narration stage receives whatever text was scraped and renders it for TTS. The *process* — what the workflow did, what concepts were created, what relationships were established — is lost. - -In the observed failure: the arXiv paper representation workflow successfully created a file copy, a scholarly paper concept, author concepts, and metadata relationships. But the user was told only the paper's abstract. Nothing about the file copy. Nothing about the concept. Nothing about the authors. The workflow succeeded technically but failed communicatively. - -### 3.6 The Bootstrap Pattern - -Each monolithic workflow family has a corresponding `*_workflow_vontology_service.py` file that bootstraps its VWL definition into Vontology at startup. For example, `paper_representation_workflow_vontology_service.py` calls `bootstrap_repo_seed_workflow_bundle()` with a JSON seed bundle. These services are explicitly documented as "startup-only seed publication/repair entry points" (§2.2 of the VWL manual), but their practical effect is to ensure that the monolithic Python-handled workflows are always present in the capability index and always discoverable. - -The seed bundles contain the complete VWL workflow definitions (steps, transitions, action contracts, discovery exemplars, routing profiles). The definitions are structurally correct — they are genuine VWL workflows authored in Vontology-native graph structures. The problem is not the definitions. The problem is that every step in these definitions is bound to a registered Python handler through `invokesAction` directives, and every handler is a deterministic Python function, so the resulting execution is a closed pipeline. - ---- - -## 4. Catalogue of Subversions - -The following is an exhaustive list of code-level patterns that subvert the intended KB-authoritative, LLM-supervised workflow architecture. - -### 4.1 The `selector_requests_custom_workflow` Flag (Orchestrator) - -**Location**: `orchestrator.py`, line 22306-22312 - -**What it does**: Any discovered workflow with a routing match is promoted to custom (deterministic) dispatch by default, unless it explicitly opts into `tool_pipeline` mode or `direct_response` mode. - -**Why this subverts the vision**: The intended hierarchy has the LLM as a supervisory participant throughout execution. This flag creates a binary fork where the LLM is either fully in control (tool-calling mode) or entirely absent (custom workflow mode). There is no middle ground — no "LLM-supervised workflow execution" path. - -**What should happen**: The dispatch decision should not be a binary choice between "LLM does everything" and "Python does everything". A discovered workflow should be instantiated and executed step-by-step with the LLM as the narrator and communicator, able to report what each step did, even when the steps themselves are deterministic. - -### 4.2 Monolithic Action Handlers Without Communication - -**Location**: All 17 deterministic workflow families in `src/backend/workflows/durable/` - -**What they do**: Each handler is a pure Python function that performs a data operation (normalise inputs, materialise concept, enrich metadata, verify postconditions) and returns a result dictionary. None of these handlers emit any user-facing text, progress indication, or explanatory message. - -**Why this subverts the vision**: In the intended hierarchy, even deterministic steps exist within a workflow that the LLM supervises. The LLM should be able to say "I'm normalising the arXiv identifier", "I've created a scholarly paper concept", "I'm resolving author identities". The current handlers are invisible operations. - -**What should happen**: Either (a) action handlers should produce structured result payloads that include human-readable summaries of what was done, which the orchestrator can surface to the user; or (b) the workflow execution engine should emit step-completion events that the top-level LLM can narrate; or (c) LLM-mode wrapper steps should bracket each deterministic step to provide before/after communication. - -### 4.3 The `execute_workflow()` Method as a Black Box - -**Location**: `orchestrator.py`, line 18741 - -**What it does**: Calls `WorkflowExecutor.run()` and waits for the entire workflow to complete, then extracts a response from the result. - -**Why this subverts the vision**: The workflow runs as a single synchronous call from the orchestrator's perspective. There is no streaming of intermediate results, no per-step progress events visible to the user, and no opportunity for the LLM to interject explanations between steps. - -**What should happen**: Workflow execution should be observable from the top level. Each step completion should produce a progress event that the orchestrator can forward to the user (either as streaming text, as structured progress indicators, or as queued narration points). - -### 4.4 `_render_custom_workflow_response_text()` as Emergency Scraping - -**Location**: `orchestrator.py`, line 3773-3850 - -**What it does**: After a workflow completes, it attempts to find human-readable text in the result dictionary by checking a prioritised list of field names: `response_text`, `summary`, `final_response`, then various fallback structures. - -**Why this subverts the vision**: This function exists because the custom workflow path does not produce user-facing communication as a first-class output. It scrapes whatever the workflow left behind. For paper representation, it finds the paper's abstract (because the workflow stores metadata including a summary field). It does not find anything about what the workflow *did* — the concepts created, the relationships established, the file copies stored. - -**What should happen**: Workflow results should carry explicit, structured communication payloads: a list of "things I did" with human-readable descriptions, alongside any content outputs. The response assembly logic should surface both "what was done" and "what the result contains". - -### 4.5 Narration as Paraphrase Rather Than Report - -**Location**: `orchestrator.py`, narration action handlers (lines 1861-1884) - -**What it does**: The narration workflow takes whatever text was extracted by `_render_custom_workflow_response_text()` and asks an LLM to reformulate it for text-to-speech delivery. The narration prompt instructs: "Use short, clear sentences. Prefer plain language. Avoid code, Markdown, URLs." - -**Why this subverts the vision**: The narration stage receives only the scrapped output text — typically the paper abstract. It has no access to the workflow's execution trace, the steps that ran, or the concepts that were created. It cannot narrate "I've stored this paper in your knowledge base and created author concepts" because it never received that information. - -**What should happen**: The narration stage should receive a structured execution summary alongside the content output, enabling it to compose a response like: "I've ingested DINO: Grounding Multimodal Large Language Models from arXiv. I created a paper concept, linked three authors, and stored the PDF. Here's what the paper is about: [abstract]." - -### 4.6 The Startup Bootstrap Pipeline - -**Location**: Six `*_workflow_vontology_service.py` files in `src/backend/services/` - -**What they do**: At application startup, each service loads a JSON seed bundle and publishes the corresponding VWL workflow family into Vontology. This ensures the capability index can discover these workflows. - -**Why this subverts the vision**: The bootstrapping itself is not inherently subversive — importing seed data is a valid startup operation. The subversion is that these bootstrap services ensure the perpetual availability of workflows whose entire step inventory is bound to monolithic Python handlers. The VWL definitions are structurally correct but functionally opaque: every step says `invokesAction: "scholarly_paper.normalise_inputs"`, and that action is implemented as a Python function that runs silently. - -**What should happen**: Bootstrapped workflows should either (a) have their step execution modes reconsidered — some steps that are currently `deterministic` might benefit from `llm` mode or from a hybrid approach where the deterministic handler runs but an LLM wrapper narrates the result; or (b) the execution engine should provide a progress-reporting surface that makes deterministic execution visible regardless of whether the step uses an LLM. - -### 4.7 Missing `execution_mode: "llm"` on Judgement Steps - -**Location**: Various seed bundles and Vontology workflow definitions - -**What happens**: Steps that involve genuine judgement (e.g., `arxiv.decide_acquisition_mode` — deciding whether to download a PDF) are implemented as deterministic Python handlers rather than as LLM-mode steps. The handlers contain conditional logic that makes the decision based on heuristics (does the file already exist? is the URL reachable?). - -**Why this subverts the vision**: The VWL manual states (§4): "Prompt-bearing reasoning steps MUST compile as `llm` execution-mode steps." While these specific steps do not currently carry prompt contracts, they perform decisions that could benefit from LLM reasoning — and more importantly, from LLM narration ("I already have this paper, so I'll skip downloading it" vs "I need to download the PDF from arXiv"). - -### 4.8 No User-Facing Step Completion Events - -**Location**: `engine.py`, `WorkflowExecutor.run()` (line 1251) - -**What happens**: The executor loops through states and actions, recording results in the execution trace. But the trace is stored internally for telemetry and debugging. No events are emitted to the user-facing response stream. - -**Why this subverts the vision**: Even in a purely deterministic workflow, the user should see something: a progress indicator, a step-by-step log, or a real-time narration. The current executor provides none of these. The user's only view of the workflow is whatever text is scraped from the final result dictionary. - ---- - -## 5. The Observed Failure: A Case Study - -In session `d57ab88b`, the user pasted `https://arxiv.org/abs/2411.04983` into the chat. Here is what happened: - -1. **Workflow Discovery** (24.8 seconds): The capability index matched `#V#arxiv_paper_representation_workflow` with confidence 1.0. - -2. **Dispatch Preparation** (88.9 seconds): Eight pre-dispatch steps completed, including a 75-second ontology preflight. The selector chose the arXiv workflow. - -3. **Custom Workflow Dispatch**: `selector_requests_custom_workflow = True`. The orchestrator handed the URL to `execute_workflow()`. - -4. **Deterministic Execution**: The paper representation workflow ran through its steps: - - `arxiv.normalise_source` — extracted the arXiv ID. - - `arxiv.decide_acquisition_mode` — determined whether to download. - - Downloaded the PDF (via subworkflow or inline). - - `scholarly_paper.materialise_from_file_copy` — created the paper concept. - - `scholarly_paper.enrich_from_metadata` — fetched and attached arXiv metadata. - - `scholarly_paper.resolve_authors` — created/linked author concepts. - - `scholarly_paper.verify_representation` — validated postconditions. - -5. **Response Assembly**: `_render_custom_workflow_response_text()` found a `summary` field in the result — the paper's abstract. - -6. **Narration**: The narration workflow paraphrased the abstract for TTS. - -7. **User Experience**: The user heard a summary of the paper's abstract. **No mention of**: - - A file copy being created. - - A paper concept being created in Vontology. - - Author concepts being created or linked. - - The paper being added to the knowledge base. - - Any persistent action having occurred. - -The user reasonably concluded that nothing structural had happened — that Von had merely read the arXiv page and summarised it. This is the direct consequence of the custom workflow bypass: the LLM was not present during execution, so it could not report what changed. - ---- - -## 6. The Warnings Explained - -The telemetry also surfaced three warnings: - -### Warning 1: `Workflow '#V#conversation_turn_execution_workflow' is not runnable; instance was not created.` - -`#V#conversation_turn_execution_workflow` is a real VWL workflow concept in Vontology. It is defined in the seed bundle (line 1487 of `canonical_workflow_publication_seed_bundle.json`) and used to model the conversation turn lifecycle. However, the orchestrator's preflight engine currently treats it as a concept to potentially instantiate as a runnable workflow, discovers that it lacks concrete backend bindings for all its stages, and emits this warning. - -**Root cause**: The conversation turn workflow is structurally defined in Vontology but its execution is actually hardcoded in the orchestrator's Python method sequence. The VWL definition exists as a planning/telemetry artefact, not as an actually-executable workflow. This is itself a manifestation of the problem: the top-level chat turn execution is Python-native rather than VWL-driven. - -### Warning 2: `Workflow '#V#arxiv_paper_representation_workflow' could not start because required launch inputs were unresolved: prompt.` - -The arXiv workflow's launch contract (defined in the seed bundle) declares `prompt` as a required input. During one of the orchestrator's parallel preflight probes, the system attempted to check launchability but failed to map the raw user message to the `prompt` variable. A later mechanism (the dispatch data builder at `_build_custom_workflow_dispatch_data()`) resolves this mapping correctly and the workflow launches fine. - -**Root cause**: The launch-input resolution logic has two code paths that run at different times. The early probe fails; the later dispatch succeeds. The warning is harmless but noisy — it indicates a preflight/dispatch asymmetry in the input resolution pipeline. - -### Warning 3: `Workflow '#V#chat_buttonify_workflow' is not runnable; instance was not created.` - -`#V#chat_buttonify_workflow` is a post-processing workflow concept that converts text elements into interactive UI buttons. Like `#V#conversation_turn_execution_workflow`, it is defined as a Vontology concept but its execution is embedded in the orchestrator's Python code rather than expressed as a runnable VWL program. - -**Root cause**: Same pattern as Warning 1 — VWL concepts exist in Vontology as planning/telemetry artefacts, but their actual execution is hardcoded Python. - ---- - -## 7. The Gap Between Vision and Code - -### 7.1 What the vision promises - -The AGENTS.md doctrine (§11) states: - -> Von is not mainly a Python application with some prompts attached. It is a neuro-symbolic agentic system in which enduring knowledge matters, workflow and prompt authority matter, model portfolios and learned policy matter, explicit representation matters, evaluation and observability matter, and Python exists to support those things rather than replace them. - -The VWL manual (§2) states: - -> Backend code SHOULD primarily supply reusable actions, validators, loaders, and telemetry for workflows, rather than embedding task-specific orchestration that VWL could already express. - -### 7.2 What the code actually does - -The code provides two execution paths: - -1. **Tool-calling path** (LLM-driven): The LLM is fully in control. It receives the user message, reasons about what to do, calls MCP tools one at a time, narrates its progress, and composes a response. This path satisfies the vision but does not leverage authored VWL workflows — it is pure LLM improvisation. - -2. **Custom workflow path** (Python-driven): A VWL workflow is loaded from Vontology and executed by the deterministic state machine. Python action handlers do the work. The LLM is entirely absent from execution. After the workflow finishes, a response is scraped from the output. This path leverages VWL definitions for *structure* but not for *communication* or *LLM supervision*. - -Neither path implements the intended hierarchy. The vision calls for a third path: - -3. **Supervised workflow path** (intended but unimplemented): A VWL workflow is loaded from Vontology and executed step by step. The LLM supervises the execution — it can narrate each step, communicate with the user, and reason when steps require judgement. Deterministic steps run their handlers but emit structured results that the LLM can report. LLM-mode steps use prompt contracts from Vontology. The user sees the workflow in action. - -### 7.3 Why the gap exists - -The custom workflow path was likely built as a reliability optimisation. Paper ingestion, entity materialisation, and similar operations require a specific sequence of steps that must happen reliably, in order, with proper error handling. Delegating this to an LLM — which might hallucinate steps, skip steps, or botch the sequence — was risky. Building deterministic handlers and a state machine was the safe engineering choice. - -This reasoning was locally correct. LLM tool-calling is unreliable for complex multi-step sequences. The problem is that the solution was *too* complete: it removed the LLM entirely rather than using the LLM as a supervisor over the deterministic execution. - -### 7.4 What the gap costs - -1. **User trust**: The user does not know what Von did. Silently creating concepts and file copies without reporting them damages the user's ability to verify, correct, or build on the system's actions. - -2. **Minimal imposition violation**: The minimal imposition principle (§6.2) requires exposing uncertainty and making actions visible. A silent pipeline is maximally opaque. - -3. **Observability debt**: The telemetry records `dispatch_zero_execution: true` and `tools_completed: 0`, creating the misleading impression that the workflow did nothing. In reality, it performed substantial mutations — they just did not occur through the MCP tool layer. - -4. **Workflow authority erosion**: Because the custom workflow path "works" silently, there is no pressure to improve it. The Python handlers accumulate, new workflow families copy the pattern, and the proportion of user-facing work that runs as opaque pipelines grows. - ---- - -## 8. Structural Fixes Required - -This paper does not prescribe code changes, but identifies the architectural capabilities that are missing. - -### 8.1 Step-level progress emission - -The workflow executor (`WorkflowExecutor.run()`) should emit structured step-completion events to a progress channel that the orchestrator can subscribe to. Each event should carry: step name, action ID, execution mode, human-readable summary of what was done, and key outputs. - -### 8.2 Workflow execution summary as a first-class result field - -Workflow results should include a `execution_summary` field: an ordered list of completed steps with their names, descriptions, and salient outputs. This replaces the current scraping approach. - -### 8.3 LLM-supervised dispatch mode - -The orchestrator should support a third dispatch mode: "supervised workflow". In this mode, the workflow executes step by step (deterministic steps run their handlers), but after each step the orchestrator feeds the step result to the LLM for narration and optional user communication. This preserves the reliability of deterministic execution while restoring LLM participation. - -### 8.4 Conversation turn workflow as an executable VWL program - -`#V#conversation_turn_execution_workflow` should be a genuinely executable VWL workflow, not just a telemetry label. The top-level chat turn logic in the orchestrator should be a VWL execution, with the current Python method sequence refactored into VWL steps with action handlers. - -### 8.5 Audit of execution modes on judgement steps - -Steps like `arxiv.decide_acquisition_mode` that make substantive decisions should be audited for whether they are correctly marked as `deterministic` or should be `llm`-mode steps with prompt contracts. - ---- - -## 9. Conclusion - -Von's workflow architecture has the right conceptual design. The five-layer hierarchy — chat turn, LLM dispatch, workflow instantiation, step-by-step execution, and within-step tool/LLM use — is well motivated and well represented in the VWL manual and AGENTS.md doctrine. The VWL language, the Vontology-native workflow definitions, the condition language, the publication lifecycle, the routing profiles, and the discovery exemplar system are genuinely sophisticated and well-engineered authority surfaces. - -The problem is that the execution layer short-circuits the hierarchy. The custom workflow dispatch path converts authored, KB-authoritative workflow definitions into opaque Python pipelines by removing the LLM from the execution loop. The result is a system where the knowledge engineering is correct but the user experience is that of a silent, unexplained backend process. - -The fix is not to remove the deterministic execution capability — that capability is valuable for reliability. The fix is to restore LLM supervision *over* deterministic execution, so that the user sees what the system did, why, and what changed. The workflow definitions already carry the names, descriptions, and contracts needed for this narration. The infrastructure to deliver it — step-level progress events, execution summaries, supervised dispatch — needs to be built. diff --git a/docs/engineering/workflow_first_conversion_sweep_jvnautosci_1210.md b/docs/engineering/workflow_first_conversion_sweep_jvnautosci_1210.md deleted file mode 100644 index 3e55f87f2..000000000 --- a/docs/engineering/workflow_first_conversion_sweep_jvnautosci_1210.md +++ /dev/null @@ -1,89 +0,0 @@ -# JVNAUTOSCI-1210 Workflow-First Conversion Sweep - -Date: 2026-02-20 - -## Scope - -This sweep targeted behaviour/action pathways under the `JVNAUTOSCI-833` epic that were represented as Vontology workflows but still classified as non-executable due vacuous step contracts. - -The goal was to: - -1. Classify conversion candidates. -2. Convert class-1 candidates directly in Vontology (without adding specialised Python orchestration). -3. Re-check executability and behavioural equivalence expectations. -4. Record follow-on class-2 items. - -## Baseline - -Before this sweep, Vontology workflow status was: - -| Workflow ID | Status | Reason | -| --- | --- | --- | -| `#V#entity_representation_workflow` | executable | `executable_now` | -| `#V#add_affiliation_workflow` | executable | `executable_now` | -| `#V#salient_predicate_governance_workflow` | non-executable | `workflow_step_partially_vacuous` | -| `#V#sail_phd_student_onboarding_workflow` | non-executable | `workflow_step_completely_vacuous` | -| `#V#von_user_onboarding_workflow` | non-executable | `workflow_step_completely_vacuous` | -| `#V#von_workflow_creation_workflow` | non-executable | `workflow_step_completely_vacuous` | - -## Conversion Matrix - -| Pathway | Classification | Action | Outcome | -| --- | --- | --- | --- | -| `#V#entity_representation_workflow` | Class 1 (convertible now) | Previously converted and completed in Vontology | Remains executable | -| `#V#add_affiliation_workflow` | Class 1 (convertible now) | Previously converted and completed in Vontology | Remains executable | -| `#V#salient_predicate_governance_workflow` | Class 1 (convertible now) | Added step contracts via `#V#workflow_step_writes_context_key` to vacuous steps | Now executable | -| `#V#sail_phd_student_onboarding_workflow` | Class 1 (convertible now) | Added step contracts via `#V#workflow_step_writes_context_key` to all steps | Now executable | -| `#V#von_user_onboarding_workflow` | Class 1 (convertible now) | Added step contracts via `#V#workflow_step_writes_context_key` to all steps | Now executable | -| `#V#von_workflow_creation_workflow` | Class 1 (convertible now) | Added step contracts via `#V#workflow_step_writes_context_key` to all steps | Now executable | -| Built-in workflow registrations (e.g. `#V#chat_assistant_workflow`, `#V#conversation_turn_execution_workflow`) | Class 2 (convertible with small extension) | Not converted in this task; require Vontology process-graph parity for code-registered workflows | Tracked as follow-on under `JVNAUTOSCI-803` / epic backlog | - -## Behavioural Equivalence Notes - -For converted class-1 workflows, this sweep preserved control-flow topology and only added explicit step contracts where missing. - -Equivalence expectation maintained: - -1. Step ordering and transition routing remain unchanged. -2. No broadening of authz/safety surface (no new privileged tools introduced). -3. Runtime intent is preserved: steps that were sequencing placeholders remain sequencing placeholders, now with explicit contract metadata so workflow submission/executability gates no longer reject them as vacuous. - -## Post-Conversion Verification - -Post-sweep Vontology workflow status: - -| Workflow ID | Status | Reason | -| --- | --- | --- | -| `#V#add_affiliation_workflow` | executable | `executable_now` | -| `#V#entity_representation_workflow` | executable | `executable_now` | -| `#V#sail_phd_student_onboarding_workflow` | executable | `executable_now` | -| `#V#salient_predicate_governance_workflow` | executable | `executable_now` | -| `#V#von_user_onboarding_workflow` | executable | `executable_now` | -| `#V#von_workflow_creation_workflow` | executable | `executable_now` | - -Summary: - -1. Vontology workflows discovered: `6` -2. Vontology workflows executable: `6` -3. Vontology workflows non-executable: `0` - -## Follow-On Gaps (Class 2) - -The registry still contains built-in (`source=built_in`) workflow definitions that classify as `non_executable_design_artifact` under Vontology graph-based executability checks. - -These are workflow-first migration candidates for follow-on work: - -1. Persist or mirror built-in workflow definitions as Vontology process graphs. -2. Keep registry/graph parity so monitor executability and routing eligibility are consistent. -3. Continue this under existing umbrella work in `JVNAUTOSCI-803` (linked from `JVNAUTOSCI-1210`). - -## Authoritative Publication Follow-On - -`JVNAUTOSCI-1217` adds strict publication/runnability gates and deterministic definition identity/hash telemetry for workflow monitor/introspection surfaces. - -The dated -[`workflow_authoritative_publication_process.md`](workflow_authoritative_publication_process.md) -records the February 2026 implementation provenance. For current authoring and -validation procedure, use the -[`von_workflow_language_manual.md`](von_workflow_language_manual.md), live -Vontology tooling, and authority read-back. diff --git a/docs/engineering/workflow_model_policy_schema.md b/docs/engineering/workflow_model_policy_schema.md index 1b3ccc69f..2f9dc94ab 100644 --- a/docs/engineering/workflow_model_policy_schema.md +++ b/docs/engineering/workflow_model_policy_schema.md @@ -127,7 +127,7 @@ IDs. The explicit `provider:model` form remains the current concrete mechanism, not the desired end-state for learned or capability-based model selection. For the actor-critic learning loop that should produce and retire those capability certifications from real conversation rollouts, see -`docs/engineering/automated_policy_learning_design.md`, especially the +[historical automated-policy-learning proposal](https://github.com/Strong-AI-Lab/Von/blob/5b2f8b976069935da73b82bca433fb3cefffcd3d/docs/engineering/automated_policy_learning_design.md), especially the `JVNAUTOSCI-2090` model-use learning section. ```json @@ -283,7 +283,7 @@ Completed: Remaining: - Add tests for graph resolver -- Update Conversation Turn Workflow doc (see [docs/Conversation Turn Workdlow.md](docs/Conversation%20Turn%20Workdlow.md)) +- Update Conversation Turn Workflow doc (see [docs/Conversation Turn Workdlow.md](https://github.com/Strong-AI-Lab/Von/blob/5b2f8b976069935da73b82bca433fb3cefffcd3d/docs/Conversation%20Turn%20Workdlow.md)) - Once graph parity is confirmed via diagnostics, consider deprecating JSON ## Diagnostic endpoint diff --git a/docs/engineering/workflow_self_authoring_gap_analysis.md b/docs/engineering/workflow_self_authoring_gap_analysis.md deleted file mode 100644 index e73f9d69a..000000000 --- a/docs/engineering/workflow_self_authoring_gap_analysis.md +++ /dev/null @@ -1,154 +0,0 @@ -# Workflow Self-Authoring and Intent Generation: Gap Analysis & Plan - -> **Document status: Dated gap analysis and proposed plan; not current programme -> authority.** “Current”, “exists”, “missing”, and task-ordering claims reflect -> the 5–6 April 2026 audit. Re-read live Jira, code, Vontology materialisation, -> and current VWL authority rules before planning. - -- **Kind:** Gap analysis and design proposal -- **Lifecycle:** Frozen proposal -- **Authority:** Advisory only -- **State as of:** 2026-04-06 -- **Related Epics:** `JVNAUTOSCI-833`, `JVNAUTOSCI-964`, `JVNAUTOSCI-1586` -- **Related Tasks:** `JVNAUTOSCI-1531`, `JVNAUTOSCI-1654`, `JVNAUTOSCI-1574` - -## 1. Purpose -This document examines the extent to which Von meets its goal of driving user interaction behaviours primarily through workflows that are self-authored following collaborative problem-solving sessions. It outlines the met goals, identifies the architectural gaps, and provides a concrete implementation plan. - -## 2. Current State Assessment - -### 2.1 Met Goals (The Substrate) -Following the recent workflow-first convergence (`JVNAUTOSCI-1517` through `1529`), Von has a robust foundation: -- **Vontology-Authoritative Execution**: The execution and routing of behaviours are now driven by VWL definitions and workflow instances, avoiding Python-based heuristics. -- **Durable Runtime**: Checkpoints, subworkflows, and verification gates are operational (`durable_workflow_system_design.md`). -- **Gap Discovery Seed**: `workflow_gap_recovery_workflow.py` provides a precedent for identifying when a required workflow is missing. -- **Testing-Workflow Substrate Exists**: The system is no longer starting from design documents alone. `testing_workflow_contracts.py`, `testing_theory_service.py`, `experiment_run_service.py`, MCP tool surfaces, and the current testing seed bundles already provide first-class IDs, schema versions, and partial execution pathways for `#V#ephemeral_theory`, `#V#experiment_spec`, `#V#experiment_run`, `#V#promotion_gate_workflow`, and related testing artefacts. -- **Workflow Studio Baseline Exists**: The Workflow Studio is no longer absent. There is now an independent Studio surface with catalogue/detail views, topology/decision/dataflow/operations inspection, and preview-first authoritative editing support. The remaining gap is candidate self-authoring integration and approval/provenance handling, not basic visualisation. - -### 2.2 Unmet Goals (The Generation & Validation Gap) -While Von can run workflows, it cannot yet reliably and safely *author* them autonomously based on user interactions. -- **Missing Intent-to-Workflow Pipeline**: As noted in `JVNAUTOSCI-1654`, there is an explicit gap in generating workflows from user intent. The system lacks the prompt-programs and structured pathways to translate a collaborative chat session into a VWL definition. -- **Incomplete Safe Mutation Boundary**: The testing-theory substrate now exists, but the exact authoritative mutation-boundary story for theory-bounded workflow execution is still incomplete. The remaining issue is not "invent ephemeral theories" but proving and hardening theory-local write isolation, promotion semantics, and garbage collection across real MCP and workflow paths. -- **Missing Candidate Proposal Lifecycle**: Self-authored workflows need a first-class inspectable proposal/review state. Draft workflow lifecycle metadata exists, but the self-authoring plan still needs an explicit authoritative story for `pending_review`, `approved`, `rejected`, provenance, and rollback/supersession rather than leaving proposal state implicit in UI or workflow-local context. -- **Opaque Side-Effect Policy for Generated Workflows**: A generated workflow must not be allowed to synthesise arbitrary executable side effects. The current plan needs an explicit validator/allowlist story for dangerous tools, external mutations, schedules, bindings, and other high-impact actions. -- **Workflow Studio Integration Gap**: The missing UI work is not the baseline Studio itself. The gap is that session-derived candidate workflows are not yet presented as mixed-initiative proposals with validation evidence, provenance, and publication gating layered onto the existing Studio path. - -## 3. Concrete Implementation Plan - -To achieve full collaborative self-authoring, we must execute the following structured sequence, prioritising safe execution correctness and inspectability. - -### Phase 1: Safe Sandboxing (Testing Workflows) -*Epic: `JVNAUTOSCI-964` / `JVNAUTOSCI-1531`* -Before Von can author workflows, it must be able to test its own drafts. -1. Complete and harden the existing `#V#ephemeral_theory`, `#V#experiment_spec`, and `#V#experiment_run` substrate rather than creating a parallel second testing system. -2. Develop the `capability_test_execution_workflow` to execute candidate self-authored workflows against theory-local data using the current testing-theory and experiment-run surfaces. -3. Ensure that outputs from these runs produce structured `experiment_run.v1` evidence rather than mutating canonical Vontology. -4. Verify the mutation boundary on the exact production pathways that matter: workflow execution, MCP-mediated writes, promotion helpers, and theory garbage collection. - -### Phase 2: Intent Generation and Repair Surface -*Epic: `JVNAUTOSCI-833`* -1. **Workflow Generation Prompt Programs**: Author Vontology-native prompt concepts designed to take session dossiers and user intents, outputting compliant VWL graphs. -2. **Automated Verification**: Route the generated VWL through Phase 1's capability tests. -3. **Repair Loop**: If the candidate workflow fails (e.g., prohibited mutations, missing dataflow mappings), trigger a secondary repair workflow to ingest the structured failure evidence and patch the candidate VWL. -4. **Reuse Existing Generic Authoring Surfaces**: This phase should build on `#V#workflow_repair_or_create_workflow`, `#V#workflow_authoring_repair_workflow`, `#V#von_workflow_creation_workflow`, and Vontology-authored template/profile metadata rather than creating a bespoke prompt-to-graph bypass path. -5. **Prompt-Evaluation Discipline**: Generation prompts are operational policy. They need replay cases, baseline prompt lineage, promotion thresholds, and rollback conditions, not only "works on one example" acceptance. - -### Phase 3: Inspectable Collaboration (Workflow Studio) -*Epic: `JVNAUTOSCI-1586` / related design tasks `JVNAUTOSCI-1587`, `JVNAUTOSCI-1588`* -The user must remain the final arbiter of newly authored behaviour policies. -1. Reuse the existing Workflow Studio baseline (topology, dataflow, conditional logic, preview-first editing) rather than rebuilding it. -2. Introduce the **Mixed-Initiative Proposal Layer**: Instead of silent application, Von presents the session-derived candidate workflow as a diff/preview in the Workflow Studio with test evidence and provenance. -3. Allow the user to collaboratively tweak edge conditions, context mappings, and schedule bindings in the Studio UI. -4. Keep proposal state authoritative and inspectable. The Studio must not become the hidden source of approval state or draft lifecycle policy. - -### Phase 4: Promotion and Publication -*Epic: `JVNAUTOSCI-833` / `JVNAUTOSCI-1531`* -1. Extend the existing `promotion_gate_workflow` and workflow publication lifecycle machinery so a validated, user-approved candidate workflow can move from ephemeral testing state to canonical Vontology. -2. Bind the new workflow to the relevant intent triggers, routing profile metadata, and discovery/publication surfaces so the collaboratively solved behaviour becomes operational only when explicitly published. -3. Ensure promotion has a first-class rollback/supersession story. Self-authored workflows should be demotable or replaceable without leaving unclear routing residue. - -## 4. Jira Task Index and Interpretive Notes - -### 4.1 Task Index - -| Key | Phase | Summary | Key Interpretation | -|-----|-------|---------|-------------------| -| JVNAUTOSCI-1720 | 1 — Sandboxing | Harden ephemeral theory isolation and prove mutation boundary on production paths | **Harden existing substrate**, not build from zero. `testing_theory_service.py` etc. already exist. | -| JVNAUTOSCI-1721 | 1 — Sandboxing | Harden experiment_spec and experiment_run evidence schema on production paths | **Verify and extend** existing `experiment_run_service.py` schema for repair-loop consumption. | -| JVNAUTOSCI-1722 | 1 — Sandboxing | Build capability_test_execution_workflow for sandboxed workflow testing | New VWL workflow composing the existing theory/experiment substrate. | -| JVNAUTOSCI-1723 | 1 — Sandboxing | Add mutation and side-effect scope enforcement for theory-bounded workflow execution | **Gate all side effects** (schedules, bindings, Jira/GitHub, MCP mutations), not just Vontology graph writes. Safety-critical. | -| JVNAUTOSCI-1724 | 2 — Generation | Author intent-to-VWL generation prompt programs as Vontology text relations | Prompts are policy. Must include replay corpus, lineage, promotion threshold, rollback condition. | -| JVNAUTOSCI-1725 | 2 — Generation | Build intent-to-workflow generation and verification pipeline | **Must reuse** `#V#workflow_repair_or_create_workflow`, `#V#von_workflow_creation_workflow`, etc. No bespoke prompt→graph bypass. | -| JVNAUTOSCI-1726 | 2 — Generation | Build VWL repair loop workflow for failed candidate generation | Extends existing `#V#workflow_authoring_repair_workflow`. Bounded retry with user escalation. | -| JVNAUTOSCI-1727 | 3 — Collaboration | Workflow Studio: integrate candidate workflow diff/preview with evidence and provenance | **Integrate into shipped Studio**, not rebuild. The baseline already exists. | -| JVNAUTOSCI-1728 | 3 — Collaboration | Mixed-initiative proposal layer for self-authored workflows | Extend `workflow_publication_lifecycle.v1` with `pending_review`/`rejected`/`superseded`. Approval state must be authoritative, not UI-local. | -| JVNAUTOSCI-1729 | 3 — Collaboration | Collaborative workflow parameter editing in proposal UI | User tweaks feed back to candidate VWL; re-validation required. | -| JVNAUTOSCI-1730 | 4 — Promotion | Extend promotion_gate_workflow and publication lifecycle with rollback/supersession support | **Extend existing** `#V#promotion_gate_workflow` and lifecycle model. Rollback/supersession are first-class. | -| JVNAUTOSCI-1731 | 4 — Promotion | Bind promoted workflows to intent triggers and update routing metadata | Include routing-eligibility checks; demotion removes eligibility cleanly. | -| JVNAUTOSCI-1732 | Cross-cutting | Add real-path performance tracking, degradation safeguards, and demotion for self-authored workflows | Nearest-real-path acceptance, replay corpus, selector regression detection. | - -### 4.2 General Interpretive Rules - -- Tasks that reference `#V#ephemeral_theory`, `#V#experiment_spec`, `#V#experiment_run`, or `promotion_gate_workflow` should be read as **complete, harden, and prove the current substrate**, not "start from zero". -- Tasks that describe Workflow Studio work should be read as **candidate self-authoring integration into the shipped Studio**, not as a second implementation of listing/topology/preview/editing. -- Intent-generation tasks must explicitly reuse the existing generic workflow-authoring primitives and template/profile pathways. A parallel raw prompt-to-graph path would be workflow-authority drift. -- Proposal/review state must be first-class and authoritative. Extend `workflow_publication_lifecycle.v1` rather than using UI-local or ad-hoc code-side state. -- Self-authored workflows need an explicit authoring-policy validator for dangerous or high-impact actions (schedules, bindings, external MCP mutations). Structural VWL validity alone is not sufficient. -- Promotion and degradation tasks must include rollback, supersession, and post-promotion routing eligibility checks rather than treating publication as a one-way event. - -## 5. From-Scratch Codebase Audit (2026-04-06) - -A fresh analysis of the actual codebase and Vontology state was performed to cross-check the above plan. The audit confirmed all 13 tasks (1720–1732) target genuine gaps, and uncovered six additional gaps not previously tracked. - -### 5.1 Audit Method - -- Deep exploration of execution engine, testing substrate, authoring workflows, side-effect boundaries, promotion lifecycle, and frontend Studio. -- MCP `concept_exists` checks against all 10 key concept IDs referenced by the self-authoring substrate. -- Vontology tree dump (154 concepts) to verify what actually exists. -- Code-level tracing of bootstrap, materialisation, and validation pathways. - -### 5.2 Key Codebase Finding: Current Vontology State vs Implemented Bootstrap - -The MCP audit correctly showed that the currently queried Vontology surface did **not** have the expected self-authoring concepts materialised at audit time. However, that should not be read as "these workflows or services do not exist in code". - -The codebase already contains substantial bootstrap and publication paths: - -- `_start_durable_workflow_system()` in `utils_flask.py` runs workflow-family bootstrap functions including `bootstrap_canonical_testing_workflows()`. -- Startup also runs `bootstrap_workflow_concept_identities()` in `workflow_concept_authority_service.py` to repair workflow identity/type drift before strict parity checks. -- Repo seed bundles already define canonical workflow concepts such as `#V#promotion_gate_workflow`, `#V#workflow_repair_or_create_workflow`, `#V#workflow_authoring_repair_workflow`, `#V#von_workflow_creation_workflow`, and `#V#workflow_discovery_gap_recovery_workflow`. - -The accurate conclusion is therefore: - -- the current queried Vontology/MCP surface is missing materialised self-authoring concepts right now; -- workflow-family bootstrap exists in code and can republish several of them at startup; but -- the parity/materialisation story is incomplete and inconsistent across workflow concepts vs testing type concepts. - -This also means repeated `exists: false` results should not automatically be interpreted as "the codebase lacks the concept". They may instead indicate one of several operational states that currently are not easy to distinguish from the outside: - -- the MCP surface is pointed at a fresh or test database; -- startup bootstrap has not run in the queried process; -- background parity/inventory publication is still pending; -- workflow-family bootstrap succeeded but testing type-concept parity did not; -- the queried environment is genuinely missing required authoritative artefacts. - -### 5.3 Newly Identified Gaps - -| ID | Gap | Severity | Evidence | Relationship to Existing Tasks | -|----|-----|----------|----------|-------------------------------| -| **A** | **Testing substrate type-concept bootstrap missing** | HIGH | `#V#ephemeral_theory`, `#V#experiment_spec`, and `#V#experiment_run` are used as `parent_concept_ids` in `testing_theory_service.py` and `experiment_run_service.py`, but unlike workflow concepts there is no dedicated bootstrap/seed parity path ensuring those type concepts exist up front. `create_concept()` does not repair missing parent types automatically. | Prerequisite for JVNAUTOSCI-1720/1721. Needs scope expansion or standalone task. | -| **B** | **No standalone MCP lint/validation surface for arbitrary candidate VWL definitions** | MEDIUM | Validation infrastructure already exists in `validate_workflow_definition_contract()`, Workflow Studio preview, and `workflow_authoring.validate_workflow_definition`. The gap is that there is no simple first-class MCP tool that accepts a candidate authoring spec/definition and returns structural validation results without routing through a larger authoring workflow. | Should be scoped into JVNAUTOSCI-1722 or standalone. Needed before generation pipeline (1724/1725) can lint candidates cleanly. | -| **C** | **No published machine-readable VWL authoring schema/contract surface** | MEDIUM | VWL structure is enforced by loader/validation code and policy schemas, but there is no single published JSON Schema or equivalent authoring contract surface that prompt programs and external tooling can target directly. `von_workflow_language_manual.md` remains narrative. | Prerequisite for JVNAUTOSCI-1724 (generation prompts need a stable reference contract to emit against). | -| **D** | **No bulk concept-parity audit MCP tool** | LOW | Cross-referencing code-referenced concept IDs against Vontology requires N sequential `concept_exists` calls. Created as **JVNAUTOSCI-1733**. | Independent observability improvement. Supports 1720/1721. | -| **E** | **No first-class rollout/lineage model for multiple published workflow revisions** | MEDIUM | `workflow_publication_lifecycle.v1` and `definition_identity` exist, but there is no explicit model for v1/v2 coexistence, canary routing, supersession chains, or gradual promotion/demotion across published revisions. | Should be scoped into JVNAUTOSCI-1730 (rollback/supersession) or standalone. | -| **F** | **Prompt authority health checks and failure signalling should be standardised across self-authoring prompt families** | LOW | The workflow-gap prompt path already aims to fail closed, and missing prompts surface explicit diagnostics in `workflow_gap_recovery_workflow.py`. The improvement is to make that behaviour consistent across self-authoring prompt families, expose health-check diagnostics, and prevent any future silent degradation. | Should be scoped into JVNAUTOSCI-1724 or a small cross-cutting prompt-authority task. | - -### 5.4 Updated Task Summary - -| Key | Phase | Summary | New Dependency | -|-----|-------|---------|----------------| -| JVNAUTOSCI-1720 | 1 | Harden ephemeral theory isolation | **Needs Gap A** resolved first (type bootstrap) | -| JVNAUTOSCI-1721 | 1 | Harden experiment schema | **Needs Gap A** resolved first | -| JVNAUTOSCI-1722 | 1 | Build capability_test_execution_workflow | Should absorb **Gap B** (VWL validation MCP tool) | -| JVNAUTOSCI-1724 | 2 | Intent-to-VWL generation prompts | **Needs Gap C** (published VWL schema for prompt reference) | -| JVNAUTOSCI-1730 | 4 | Promotion lifecycle with rollback | Should absorb **Gap E** (workflow versioning) | -| JVNAUTOSCI-1733 | Cross-cutting | Bulk concept-parity audit MCP tool | **New task** (Gap D) | diff --git a/docs/vontology/concept_relation_extent_display_design.md b/docs/vontology/concept_relation_extent_display_design.md deleted file mode 100644 index 142ef48e4..000000000 --- a/docs/vontology/concept_relation_extent_display_design.md +++ /dev/null @@ -1,341 +0,0 @@ -# Concept Relation Extent Display Redesign (Design Only) - -## Goal -Redesign the concept-tab `Relationships` section so it behaves more like predicate extent displays: - -1. Show all relation assertions where the current concept appears in either argument position (arg1 or arg2). -2. Make relation assertion symmetric, so the user can assert with the current concept fixed as arg1 or as arg2. - -This document is design-only and does not include implementation. - -## Current State and Gap -- Current UI (`dynamicTabs.js`, `concept_tab.html`) renders grouped chips by predicate and mostly follows outgoing relations. -- Current read route (`GET /vontology/api/vontology/relationships`) returns a convenience map keyed by predicate, but does not provide a canonical row-wise extent view with explicit argument position for the focal concept. -- Current add form always treats current concept as source (`source_id`) and only accepts a target field. - -## Proposed UX - -### Section Rename and Layout -- Rename section heading from `Relationships` to `Relation Extent`. -- Replace chip-only grouped display with a table-first extent display aligned to predicate extent visual language. -- Keep lightweight predicate grouping as an optional secondary view (collapsed by default). - -### Extent Table -Use a table with one row per assertion: - -| Column | Purpose | -| --- | --- | -| Role | Whether current concept is `arg1` or `arg2` in this assertion | -| Predicate | Predicate cartouche, clickable | -| Arg1 | Concept/text shown for arg1 | -| Arg2 | Concept/text shown for arg2 (or first object value for n-ary fallback) | -| Source | `structured` or `text_relations` | -| Updated | Timestamp, when available | -| Actions | Remove assertion (existing semantics) | - -Notes: -- Reuse cartouche affordances from predicate extent display (type/individual/predicate colouring, click to open, right-click to copy concept_id). -- Keep sorting and filtering parity with predicate extent (`source`, predicate filter, role filter, sort by updated/predicate/role). - -### Assert Relation Composer -Below the table, add a single inline composer: - -- `Role` control (required): `Current concept is arg1` | `Current concept is arg2`. -- `Predicate` selector/search: - - include structural + salient + searchable “Other…” predicates. - - preserve text-predicate detection behaviour from existing form. -- `Counterparty` input: - - concept search input for concept predicates. - - text/lang/type inline form for binary text predicates. -- Primary action: `Add assertion`. - -Composer behaviour: -- If role is arg1, add relation as `(current_concept, predicate, counterparty)`. -- If role is arg2, add relation as `(counterparty, predicate, current_concept)` for concept predicates. -- For text predicates, role is forced to arg1 (or arg2 option disabled with explanation), because current text APIs are subject/text-value oriented. - -## API/Data Contract Design - -### Read Path -Add a row-wise extent endpoint for concepts: - -- `GET /vontology/api/vontology/relationships/extent?concept_id=&limit=&offset=&source=&predicate=&role=` - -Response shape (conceptual): -- `success: bool` -- `concept_id: str` -- `total: int` -- `rows: [ ... ]` - -Each row: -- `relation_id` -- `relation_kind` (`structured` | `text`) -- `predicate_id` -- `arg1` (concept id or text payload) -- `arg2` (concept id or text payload) -- `matched_argument_indexes` (include `1` and/or `2`) -- `source` -- `timestamp` - -Implementation note: -- Build via existing shared service `build_concept_relations_payload(...)` with: - - `include_relations_arg1=true` - - `include_relations_any_arg=true` - - `include_text_relations_arg1=true` -- Then transform to UI rows (normalised arg1/arg2 fields + role derivation). - -### Write Path -Reuse existing routes for this phase: -- Concept predicates: `POST /vontology/api/vontology/relationships/add` -- Text predicates: `POST /api/concepts//texts` - -Directional logic in UI: -- Role `arg1`: existing behaviour. -- Role `arg2`: submit with swapped subject/target for concept predicates. - -## Styling and Reuse Plan -- Reuse predicate extent classes where feasible: - - container/table/filter/status patterns from `.predicate-extent-*` -- Extract shared table helper in follow-up implementation (to avoid duplicating predicate extent/table behaviour). -- Preserve current cartouche classes (`.concept-cartouche.*`) and search dropdown style (`.vontology-search-results`). - -## Accessibility and Interaction -- Keyboard support: - - tab through filters/composer. - - Enter submits composer. - - Arrow key navigation in search dropdowns. -- Clear status messages for add/remove success and validation errors. -- Explicit labels for role selection and predicate/counterparty inputs. - -## Non-Goals (This Task) -- No ontology semantics changes for structural predicate meaning. -- No new inference rules. -- No migration of all existing relationship chips to cards/charts. - -## Acceptance Criteria -1. The concept tab shows a relation extent table with rows where current concept appears as arg1 or arg2. -2. Each row shows predicate and both argument columns, with role indication. -3. User can add a concept-predicate relation with current concept fixed as arg1 or arg2. -4. User can remove rows from the table with existing permission/error behaviour preserved. -5. Filter/sort behaviour matches predicate extent interaction quality. -6. Existing predicate extent display remains unchanged. - -## Suggested Wireframe -```text -Relation Extent -[Role: Any v] [Predicate: any] [Source: any] [Apply] [Reset] - -| Role | Predicate | Arg1 | Arg2 | Source | Updated | Actions | -| arg1 | related_to| #V#current_concept | #V#target_concept | structured | 17 Feb 2026 13:04 | x | -| arg2 | depends_on| #V#other_concept | #V#current_concept | structured | 17 Feb 2026 12:02 | x | -| arg1 | hasName | #V#current_concept | "Example label" (en) | text_rel... | 16 Feb 2026 09:40 | x | - -Add assertion: -[Current concept is: arg1 v] [Predicate v / search] [Counterparty concept/text input] [Add assertion] -``` - -## Chat Display Element Extension (Under `JVNAUTOSCI-865`) - -### Goal -Add a dedicated display element for chat rendering of relation truth-state groups, with: -- clickable concept cartouches in minimal form, and -- explicit visual variants for asserted vs not-asserted relations. - -### Proposed Display Element Type -- `element_type`: `relation_truth_state` -- `intent`: `truth_state_relation_view` -- `channel`: `screen` - -This keeps relation-state rendering semantically distinct from generic tables and avoids broad behavioural changes to all existing table renderers. - -### Payload Sketch -```json -{ - "title": "Current Truth State", - "groups": [ - { - "label": "Conference-level", - "status": "asserted", - "assertions": [ - { - "assertion_id": "a1", - "arg1": "#V#michael_witbrock", - "predicate": "#V#attended_event", - "arg2": "#V#international_ai_cooperation_and_governance_forum_2025_melbourne", - "is_asserted": true - } - ] - }, - { - "label": "Missing (should exist)", - "status": "missing_expected", - "assertions": [ - { - "assertion_id": "a2", - "arg1": "#V#michael_witbrock", - "predicate": "#V#panelist_in_event", - "arg2": "#V#panel_4_ai_for_industry_ai_for_society_iaicgf_2025_melbourne", - "is_asserted": false - } - ] - } - ] -} -``` - -### Rendering Behaviour -- Group header: - - `asserted` -> green/check styling - - `missing_expected` -> red/cross styling - - optionally `uncertain` -> amber/warning styling -- Assertion row format (minimal): - - `[arg1 cartouche] -- [predicate cartouche] --> [arg2 cartouche]` -- Cartouches: - - clickable to open concept tabs - - right-click copies concept id - - compact presentation (name-first with id tooltip) - -### Display-Elements Contract Changes -- Add `relation_truth_state` to `ALLOWED_DISPLAY_ELEMENT_TYPES` in `display_elements_service.py`. -- Add payload validator for: - - `title: string` - - `groups: list` - - `group.status` in allow-list (`asserted`, `missing_expected`, `uncertain`) - - each assertion requiring `arg1`, `predicate`, `arg2`, `is_asserted` -- Add chat renderer in `chatTab.js` alongside current `table/timeline/task_view/workflow_view` handlers. - -### Variant Rules (Asserted vs Not Asserted) -- Asserted relation (`is_asserted=true`): - - normal stroke arrow and neutral/positive text tone. -- Not asserted relation (`is_asserted=false`): - - dashed arrow or muted line, plus red/amber accent depending on group status. - - optional suffix label: `not currently asserted`. - -### Acceptance Criteria (Chat Element) -1. Chat can render `relation_truth_state` elements without falling back to raw JSON. -2. Each assertion renders three clickable cartouches (arg1, predicate, arg2). -3. Asserted and not-asserted variants are visually distinct and accessible. -4. Existing display element types continue to validate and render unchanged. - -## Execution Plan (Relates to `JVNAUTOSCI-865`) - -### Plan Intent -Turn the above design into an implementation sequence that: -- keeps `relation_truth_state` as the primary structured rendering pathway under `JVNAUTOSCI-865`, -- closes current clickability/visibility gaps for concept and relation elements, and -- introduces a compact cartouche mode that is visually obvious and space-efficient. - -### Workstream Map - -#### Workstream A: Shared Cartouche Unification (Foundation for `JVNAUTOSCI-865`) -Objective: -- Remove divergence between legacy `.concept-cartouche` usage and shared `.vontology-cartouche` usage. - -Scope: -- Introduce a single renderer helper used by: - - concept relation extent table (`dynamicTabs.js`), - - predicate extent table (`predicateExtentDisplay.js`), - - chat relation truth-state rows (`chatTab.js`). -- Keep existing behaviour (open concept, copy id, keyboard focus), but route through one cartouche constructor. - -Acceptance gate: -- All concept-like cells in the two extent tables and relation truth-state rows use the same cartouche element family. -- Right-click copy and click-to-open are consistent across all three surfaces. - -#### Workstream B: Compact Cartouche Variant (Core UX part of `JVNAUTOSCI-865`) -Objective: -- Add a dense cartouche presentation where kind colour is expressed as background and the right kind pill is omitted. - -Scope: -- Add explicit compact variant support to cartouche rendering (`name-first`, optional id tooltip, no right-side kind pill). -- Ensure kind colour mapping remains: - - `individual` -> green - - `type` -> blue - - `predicate` -> violet -- Reuse existing settings toggle semantics (`kind-as-background`) so behaviour is predictable. - -Acceptance gate: -- Relation-heavy rows can render in compact mode without loss of discoverability. -- Visual distinction between type/individual/predicate remains clear in both standard and compact variants. - -#### Workstream C: Concept/Predicate Detection Hardening -Objective: -- Make clickability robust when values are concept-like but not already clean `#V#...` strings. - -Scope: -- Add a shared value normaliser for renderer inputs (trimming, punctuation stripping, `#v#` normalisation, canonical prefix handling). -- Use backend row metadata (`arg*_is_concept`, relation shape) plus normalisation before deciding plain text vs concept cartouche. -- Keep conservative fallback to plain text for non-concept values. - -Acceptance gate: -- Previously missed concept identifiers in extent rows become clickable cartouches. -- No false-positive conversion of clear literal text values. - -#### Workstream D: Relation Rendering Priority and Fallback (Explicitly tied to `JVNAUTOSCI-865`) -Objective: -- Ensure relation special handling is visible even when the model emits markdown/prose instead of structured elements. - -Scope: -- Primary path stays `display_elements.relation_truth_state` (authoritative for `JVNAUTOSCI-865`). -- Add deterministic fallback parser for simple triple lines (`arg1 -- predicate --> arg2`) when no structured relation display element is present. -- Mark fallback-rendered blocks with provenance metadata for diagnostics. - -Acceptance gate: -- If structured `relation_truth_state` exists, renderer uses it. -- If absent but parseable triples exist, relation block still renders with special relation styling. -- If neither exists, existing plain markdown rendering remains unchanged. - -### Implementation Order -1. Workstream A (shared cartouche helper in one place). -2. Workstream B (compact variant on top of shared helper). -3. Workstream C (hardening detection in extent renderers). -4. Workstream D (chat relation fallback, preserving `JVNAUTOSCI-865` primary path). -5. Cross-cutting clean-up and documentation update. - -Rationale: -- This order minimises churn by stabilising rendering primitives before adding parser/fallback logic. - -### Test Plan - -#### Frontend Unit/Component Tests -- Update/add tests in: - - `src/frontend/web/von_interface/static/js/test/chatTab.test.js` - - predicate extent and dynamic tab tests (new tests if missing) -- Cover: - - compact cartouche class/structure, - - click and context menu behaviours, - - relation_truth_state render precedence, - - fallback triple parsing only when structured element is absent. - -#### Backend Contract Tests -- Keep `relation_truth_state` schema validation tests in `display_elements_service` green. -- Ensure `/relationships/extent` tests still pass and add cases for concept-id normalisation edge patterns where appropriate. - -#### Manual Verification Checklist -- Chat response with structured `relation_truth_state` payload. -- Chat response with plain triple markdown only. -- Concept tab relation extent rows containing: - - canonical `#V#...`, - - lower-case `#v#...`, - - trailing punctuation variants. - -### Telemetry and Diagnostics -- Add lightweight counters/flags (debug-safe) for: - - `relation_truth_state_source` (`structured` vs `derived_from_text`), - - `cartouche_render_count`, - - `concept_token_parse_miss_count`. -- Include counters in existing debug pathways, not as separate infrastructure. - -### Risk Notes -- Main risk is accidental over-linkification of plain text that resembles ids. -- Mitigation: - - conservative normalisation rules, - - strict fallback conditions, - - tests for false positives. - -### Suggested Jira Decomposition (under `JVNAUTOSCI-865`) -1. Subtask A: Shared cartouche helper adoption across relation/table renderers. -2. Subtask B: Compact cartouche variant and settings harmonisation. -3. Subtask C: Extent renderer concept-id detection hardening. -4. Subtask D: Chat fallback parser for relation triple lines. -5. Subtask E: Test and diagnostics completion pass. diff --git a/docs/vontology/vontology_mcp_text_relations_design.md b/docs/vontology/vontology_mcp_text_relations_design.md deleted file mode 100644 index f3c93ed89..000000000 --- a/docs/vontology/vontology_mcp_text_relations_design.md +++ /dev/null @@ -1,467 +0,0 @@ -# Vontology MCP Text Relations Tools - Design Document - -## Problem Statement - -During JVNAUTOSCI-800, we needed to add a `hasContent` text relation to a prompt concept. The current MCP server has an `add_names` tool (specialized for `hasName` relations only), but lacks general-purpose text relation tools. This forced us into: - -- **Throwaway Python scripts** with import/path issues -- **Inline Python one-liners** with parameter name confusion -- **Direct MongoDB queries** bypassing service layer -- **Trial-and-error with HTTP endpoints** that returned 404 - -**What should have been simple**: "Add this text as hasContent to concept X" -**What it became**: 45+ minutes of debugging import errors, hanging processes, and MongoDB connection issues. - -## Root Cause - -The MCP server exposes `add_names()` (wraps `upsert_text_for_concept` with hardcoded `predicate="hasName"`), but doesn't expose the underlying `text_value_service` functions that handle **all text relations**: - -- `upsert_text_for_concept()` - Add/update ANY text relation -- `get_texts_for_concept()` - Retrieve text relations by predicate -- `update_text_relation_text()` - Modify existing text content -- `delete_text_relation_by_predicate_and_text()` - Remove specific text relations - -## Proposed Solution: General-Purpose Text Relation Tools - -Add 4 new MCP tools that expose the complete text_value_service API: - -### 1. `upsert_text_relation` (Replaces ad-hoc scripts) - -**Purpose**: Add or update ANY text relation (hasContent, hasDescription, hasNote, custom predicates, etc.) - -**Input Schema**: -```json -{ - "type": "object", - "properties": { - "concept_id": { - "type": "string", - "description": "The concept to attach text to (e.g., '#V#my_concept')" - }, - "predicate": { - "type": "string", - "description": "The text relation predicate (e.g., 'hasContent', 'hasDescription', 'hasNote')" - }, - "text": { - "type": "string", - "description": "The text content to attach" - }, - "language": { - "type": "string", - "default": "en-NZ", - "description": "Language code (ISO 639-1/BCP 47): en-NZ (default), en-US, fr, de, mi, zh, etc." - }, - "context": { - "type": "object", - "description": "Optional metadata (e.g., {'text_type': 'NL', 'author': 'system'})" - } - }, - "required": ["concept_id", "predicate", "text"] -} -``` - -**Output**: -```json -{ - "success": true, - "text_value_id": "694665bc61d19992de831b52", - "relation_id": "694665bc61d19992de831b53", - "relation_created": true, - "predicate": "hasContent", - "text_preview": "Analyze this LLM response and determine..." -} -``` - -**What it eliminates**: -- ❌ Creating `utilities/add_detector_content.py` throwaway scripts -- ❌ Debugging import paths and parameter names -- ❌ Inline Python with syntax escaping nightmares -- ✅ Simple MCP call: `upsert_text_relation('#V#prompt', 'hasContent', 'Prompt text...')` - ---- - -### 2. `get_text_relations` (Replaces direct MongoDB queries) - -**Purpose**: Retrieve text relations for a concept, optionally filtered by predicate/language - -**Input Schema**: -```json -{ - "type": "object", - "properties": { - "concept_id": { - "type": "string", - "description": "The concept to query text relations for" - }, - "predicate": { - "type": "string", - "description": "Optional: filter by predicate (e.g., 'hasContent', 'hasName')" - }, - "language": { - "type": "string", - "description": "Optional: filter by language code (e.g., 'en-NZ', 'fr')" - }, - "limit": { - "type": "integer", - "default": 50, - "description": "Maximum number of results to return" - } - }, - "required": ["concept_id"] -} -``` - -**Output**: -```json -{ - "concept_id": "#V#my_prompt", - "relations_found": 2, - "relations": [ - { - "predicate": "hasContent", - "text": "This is the prompt content...", - "language": "en-NZ", - "text_value_id": "abc123", - "relation_id": "def456", - "context": {"text_type": "NL"} - }, - { - "predicate": "hasDescription", - "text": "A helpful prompt for...", - "language": "en-NZ", - "text_value_id": "ghi789", - "relation_id": "jkl012", - "context": {} - } - ] -} -``` - -**What it eliminates**: -- ❌ `pdm run python -c "from src.backend.services.text_value_service import get_texts_for_concept; ..."` -- ❌ Direct MongoDB queries via mongo_client -- ✅ Simple: `get_text_relations('#V#prompt', predicate='hasContent')` - ---- - -### 3. `update_text_relation` (Replaces manual update scripts) - -**Purpose**: Modify the text content of an existing text relation by relation ID - -**Input Schema**: -```json -{ - "type": "object", - "properties": { - "concept_id": { - "type": "string", - "description": "The concept owning the text relation" - }, - "relation_id": { - "type": "string", - "description": "The ID of the text relation to update" - }, - "new_text": { - "type": "string", - "description": "The new text content" - }, - "new_language": { - "type": "string", - "description": "Optional: update the language code" - }, - "new_context": { - "type": "object", - "description": "Optional: update the context metadata" - } - }, - "required": ["concept_id", "relation_id", "new_text"] -} -``` - -**Output**: -```json -{ - "success": true, - "relation_id": "def456", - "old_text_preview": "Old prompt content...", - "new_text_preview": "Updated prompt content...", - "text_value_id": "abc123" -} -``` - -**What it eliminates**: -- ❌ Manual find → update → verify cycles -- ✅ Direct update by relation ID - ---- - -### 4. `delete_text_relation` (Completes CRUD operations) - -**Purpose**: Delete a specific text relation by predicate and text match, or by relation ID - -**Input Schema**: -```json -{ - "type": "object", - "properties": { - "concept_id": { - "type": "string", - "description": "The concept owning the text relation" - }, - "relation_id": { - "type": "string", - "description": "Optional: delete by relation ID (preferred method)" - }, - "predicate": { - "type": "string", - "description": "Optional: delete by predicate + text match" - }, - "text": { - "type": "string", - "description": "Optional: exact text to match (used with predicate)" - } - }, - "required": ["concept_id"] -} -``` - -**Output**: -```json -{ - "success": true, - "deleted_relation_id": "def456", - "deleted_text_preview": "Old content...", - "text_value_cleaned_up": true -} -``` - ---- - -## Migration Path for `add_names` - -**Current**: `add_names` is a specialized wrapper around `upsert_text_for_concept(predicate="hasName")` - -**Options**: - -1. **Keep both** (recommended): - - Keep `add_names` for ergonomic bulk name operations - - Add `upsert_text_relation` for general use - - `add_names` internally calls `upsert_text_for_concept` (no duplication) - -2. **Deprecate `add_names`**: - - Mark as deprecated in manifest - - Redirect to `upsert_text_relation` with `predicate="hasName"` - - Remove in future version - -**Recommendation**: Option 1. `add_names` has specific semantics (bulk operations, name type validation) that justify keeping it as a convenience tool. - ---- - -## Implementation Checklist - -### Phase 1: Core Tools (Immediate - JVNAUTOSCI-800) -- [ ] Add `upsert_text_relation` handler to `mcp_stdio_server.py` -- [ ] Add `upsert_text_relation` tool definition to manifest -- [ ] Add `get_text_relations` handler -- [ ] Add `get_text_relations` tool definition -- [ ] Update internal MCP catalogue with new tools -- [ ] Add HTTP routes to `mcp_server.py` (for parity) - -### Phase 2: Complete CRUD (Follow-up) -- [ ] Add `update_text_relation` handler -- [ ] Add `update_text_relation` tool definition -- [ ] Add `delete_text_relation` handler -- [ ] Add `delete_text_relation` tool definition -- [ ] Update all three MCP implementations (stdio, HTTP, internal) - -### Phase 3: Testing & Documentation -- [ ] Add unit tests for each new tool -- [ ] Add integration tests for text relation workflows -- [ ] Update MCP server documentation -- [ ] Add examples to USER_GUIDE.md -- [ ] Update AGENTS.md with new capabilities - ---- - -## Example Usage Scenarios - -### Scenario 1: Add prompt content (JVNAUTOSCI-800 use case) - -**Before** (45 minutes of pain): -```python -# utilities/add_detector_content.py -import sys -from pathlib import Path -project_root = Path(__file__).parent.parent -sys.path.insert(0, str(project_root)) -from src.backend.services.text_value_service import upsert_text_for_concept -# ... 20 more lines ... -``` - -**After** (30 seconds): -``` -mcp_vontology_upsert_text_relation( - concept_id="#V#missing_tool_call_detection_prompt", - predicate="hasContent", - text="Analyze this LLM response and determine...", - language="en-NZ" -) -``` - ---- - -### Scenario 2: Retrieve and verify prompt content - -**Before**: -```bash -pdm run python -c "from src.backend.services.text_value_service import get_texts_for_concept; result = get_texts_for_concept('#V#prompt', 'hasContent'); print(result)" -``` - -**After**: -``` -mcp_vontology_get_text_relations( - concept_id="#V#missing_tool_call_detection_prompt", - predicate="hasContent" -) -``` - ---- - -### Scenario 3: Add multilingual descriptions - -**Before**: Multiple script executions or inline Python escaping hell - -**After**: -``` -# English description -upsert_text_relation('#V#eu_member_state', 'hasDescription', - 'A country that is a member of the European Union', language='en-NZ') - -# French description -upsert_text_relation('#V#eu_member_state', 'hasDescription', - 'Un pays membre de l\'Union européenne', language='fr') - -# German description -upsert_text_relation('#V#eu_member_state', 'hasDescription', - 'Ein Land, das Mitglied der Europäischen Union ist', language='de') -``` - ---- - -## Security Considerations - -- ✅ **Access control**: All tools use `can_access_concept()` checks (existing pattern) -- ✅ **Namespace isolation**: Respects existing namespace security boundaries -- ✅ **Audit trail**: All operations create timestamped database records -- ⚠️ **Text content limits**: Add validation for text length (prevent abuse) -- ⚠️ **Predicate validation**: Consider allowlist for valid predicates (hasContent, hasDescription, hasName, hasNote, etc.) - ---- - -## Performance Considerations - -- **Bulk operations**: `upsert_text_relation` handles one relation at a time (unlike `add_names` which batches) -- **Future optimization**: Add `upsert_text_relations_batch` for bulk operations if needed -- **Caching**: Text values are content-addressed (fingerprint-based deduplication works) - ---- - -## Alternative Rejected Approaches - -### ❌ "Just improve the docs/examples" -- Problem: Still requires throwaway scripts, import debugging, parameter mapping -- Verdict: Doesn't solve root cause (missing MCP tool) - -### ❌ "Add more HTTP endpoints" -- Problem: Three separate implementations to maintain (HTTP, stdio, internal) -- Verdict: MCP is the right abstraction layer - expand it, don't bypass it - -### ❌ "Make add_names accept any predicate" -- Problem: Semantics change (names vs content), breaks existing usage -- Verdict: Better to add dedicated tools with clear purpose - ---- - -## Success Metrics - -**Immediate (Post-Implementation)**: -- ✅ Adding text relations takes <30 seconds instead of 45+ minutes -- ✅ Zero throwaway utility scripts needed for text operations -- ✅ No inline Python one-liners with escaping issues - -**Long-term (3 months)**: -- ✅ >80% of text relation operations use MCP tools instead of scripts -- ✅ Zero import/path debugging issues in JIRA tickets -- ✅ Text relation CRUD documented with examples - ---- - -## Related Issues - -- **JVNAUTOSCI-800**: Fix MCP orchestrator execution flow (prompted this design) -- **JVNAUTOSCI-303**: Internal MCP synchronization (context for tool parity requirements) - ---- - -## Design Decisions Summary - -| Decision | Rationale | -|----------|-----------| -| Add 4 new tools vs extend existing | Clear separation of concerns, avoid breaking changes | -| Keep `add_names` separate | Specialized semantics justify dedicated tool | -| Use same signatures as `text_value_service` | Direct mapping = easier maintenance | -| Language defaults to "en-NZ" | Project standard (NZ English) | -| Support optional context metadata | Future-proof for provenance, authorship, etc. | -| Implement in all 3 MCP servers | Consistency prevents JVNAUTOSCI-303-style bugs | - ---- - -## Appendix: text_value_service API Reference - -**Functions to expose**: - -```python -def upsert_text_for_concept( - subject_concept_id: str, - predicate: str, - text: str, - lang: str = "en", - provenance: Optional[Dict[str, Any]] = None, - context: Optional[Dict[str, Any]] = None, -) -> Dict[str, Any] - -def get_texts_for_concept( - subject_concept_id: str, - predicate: Optional[str] = None, - lang: Optional[str] = None, - limit: int = 50, -) -> List[Dict[str, Any]] - -def update_text_relation_text( - subject_concept_id: str, - relation_id: str, - new_text: str, - new_lang: Optional[str] = None, - new_context: Optional[Dict[str, Any]] = None, - provenance: Optional[Dict[str, Any]] = None, -) -> Dict[str, Any] - -def delete_text_relation_by_predicate_and_text( - subject_concept_id: str, - predicate: str, - text: str, - lang: Optional[str] = None, -) -> Dict[str, Any] - -def delete_text_relation( - subject_concept_id: str, - relation_id: str -) -> Dict[str, Any] -``` - ---- - -**Author**: AI Agent (Copilot) -**Date**: 2025-12-20 -**Status**: Proposed -**Reviewers**: @witbrock diff --git a/src/backend/workflows/durable/__init__.py b/src/backend/workflows/durable/__init__.py index 3b89c5387..e3aad8346 100644 --- a/src/backend/workflows/durable/__init__.py +++ b/src/backend/workflows/durable/__init__.py @@ -13,7 +13,7 @@ - Scheduled report generation - Multi-step research workflows -See docs/engineering/durable_workflow_system_design.md for full specification. +Historical design: https://github.com/Strong-AI-Lab/Von/blob/5b2f8b976069935da73b82bca433fb3cefffcd3d/docs/engineering/durable_workflow_system_design.md """ from .models import ( diff --git a/src/backend/workflows/durable/rag_sync_workflow.py b/src/backend/workflows/durable/rag_sync_workflow.py index e611379a1..27b5003e8 100644 --- a/src/backend/workflows/durable/rag_sync_workflow.py +++ b/src/backend/workflows/durable/rag_sync_workflow.py @@ -5,7 +5,7 @@ with a checkpointable, resumable background workflow. Phase 4 of JVNAUTOSCI-1075: Durable Workflow System Implementation. -Design doc: docs/engineering/durable_workflow_system_design.md +Historical design: https://github.com/Strong-AI-Lab/Von/blob/5b2f8b976069935da73b82bca433fb3cefffcd3d/docs/engineering/durable_workflow_system_design.md """ from __future__ import annotations