Skip to content

RECORDING_AI.md

Latest commit

 

History

History
171 lines (141 loc) · 8.38 KB

RECORDING_AI.md

File metadata and controls

171 lines (141 loc) · 8.38 KB
Raw

Recording Bundle and AI enhancement

BrowserWeave uses one recording-to-workflow contract for two AI surfaces:

  1. external AI clients use MCP to read a redacted Recording Bundle, author a workflow, validate it, and save it with optimistic revision protection;
  2. Studio's AI 整理录制 action sends the same Bundle through a configured server-side model connection and displays a reviewable Proposal.

Neither surface receives unrestricted raw CDP access. Recording attaches to an existing Chrome, BitBrowser, or other Chromium-compatible endpoint. Core owns only its recording tab and never starts or terminates the browser process.

Evidence contract

GET /api/recordings/{id}/bundle returns browserweave.io/recording-bundle/v1alpha1 with:

  • the revisioned, editable action timeline;
  • the deterministic baseline Workflow;
  • semantic Locator candidates and fingerprints;
  • a small DOM neighborhood for each target: tag, role, accessible name, bounded text, allowlisted attributes, up to four ancestors and eight siblings;
  • bounded semantic page state at action time and after a short settle window, plus action-to-network temporal correlations;
  • up to 2,000 network records containing redacted URL, method, resource type, status, MIME type, and timestamps;
  • an explicit redaction report.

DOM evidence never contains form values, raw HTML, script content, or arbitrary attributes. Network evidence never contains headers, cookies, authorization, request bodies, or response bodies. All query values are replaced with [redacted]. Secret input values and the CDP endpoint are never returned. Ordinary typed values require the explicit includeInputValues=true option. The built-in AI causality view is filtered by the same explicit endpoint/type allowlist as network evidence, so unselected URLs cannot re-enter through an action correlation.

Built-in enhancement

POST /api/recordings/{id}/enhance accepts:

{
  "modelConnectionId": "model-main",
  "goal": "登录后台并查询订单状态",
  "instructions": "验证码出现时等待人工处理",
  "expectedRevision": 7,
  "includeInputValues": false,
  "allowScripts": false,
  "networkEndpointIds": ["71f6e5c27e8a2a90"],
  "networkMaxRequests": 200
}

GET /api/recordings/{id}/network-endpoints returns the deduplicated, redacted Method + URL candidates captured by that recording. Each candidate has a server-derived stable ID, resource type, and occurrence count. Studio uses resource-type chips and URL search only to narrow this candidate list. A user must explicitly check endpoints; the default is to send no network evidence.

networkEndpointIds is the exact allowlist applied before model invocation. Unknown IDs are rejected instead of ignored, so callers cannot accidentally believe evidence was included. networkResourceTypes remains available to API clients as a broader explicit filter, but omission of both allowlists sends no network evidence. Matching records are deduplicated by method, redacted URL, resource type, status, and MIME type, carry an occurrences count, and are capped after deduplication. The cap defaults to 200 and may be set from 1 to 500. The Proposal reports total, matched, unique, sent, deduplicated, and omitted counts, so model evidence is auditable rather than silently truncated.

The public Recording Bundle remains complete (up to the recorder's 2,000 record limit). Filtering only changes the purpose-built AI invocation bundle; it does not destroy recording evidence or change external MCP authoring.

Model endpoint and timeout diagnostics

New model connections default to a 180 second timeout; the accepted range is 1 second to 60 minutes. Studio waits slightly longer than the server maximum, so the server can return a useful 504 instead of a generic browser abort. A timeout response includes only the safe endpoint, configured timeout, model input byte count, and sent/total evidence counts. It never contains an API key, prompt, input value, request header, or response body.

For the official Zhipu BigModel service, configure:

Base URL: https://open.bigmodel.cn/api/paas/v4
Protocol: Chat Completions
Effective endpoint: https://open.bigmodel.cn/api/paas/v4/chat/completions

This is an example, not a server-side restriction. Model URLs are stored as entered. A complete /responses or /chat/completions endpoint is called directly; otherwise the adapter adds the selected protocol suffix. Provider capabilities are configured explicitly rather than inferred from host or model name.

Built-in enhancement uses the V2 compiler-style agent in core/authoringagent. The model is not trusted to produce executable source directly. Complete DOM and recording evidence remains local. Core first asks for PlanV2 with a lightweight evidence index, computes capability closure, resolves only the refs selected by the Plan, and then sends selected node contracts and bounded fragments for GraphIR generation. Core alone compiles the executable Workflow and checks it. Every stage carries the same catalog hash; blocking evidence gaps stop in NEEDS_EVIDENCE instead of allowing the model to guess.

Before graph generation, the Agent also separates evidence into the normal business path, exceptional interrupt guards, parallel network listeners, data collection, notification, and human handoff. Thus a login or verification modal observed during recording is not automatically replayed as a normal step. A required workflow-scoped responsibility must compile to its exact resource ID; reusable group guards and durable human work remain explicit control-plane requirements rather than fake graph branches. See Business Intent Agent.

If a check fails, the next model call receives one diagnostic category and a bounded outline of the relevant GraphIR resources. It returns typed resource mutations fenced by the current candidate hash. Core applies them and runs every check again. This loop is bounded to four changed diagnostic categories and never resends the full DOM or recording bundle. See Workflow Authoring Agent.

Core accepts a generated Workflow only after it:

  1. parses and compiles against the canonical executable catalog;
  2. passes graph, port, field, template, and strict data-flow checks;
  3. maps every required business success criterion to an executable typed check that runs before end;
  4. restores the original attached-browser configuration and private pageUrl default, so model output cannot replace them;
  5. passes model-connection dependency and JavaScript permission checks;
  6. produces a measurable semantic improvement over the baseline recording;
  7. still matches the recording revision before the Proposal is returned.

The focused contract is derived from the canonical executable node catalog, Workflow AST schema, and relevant authoring recipes. Control-plane Automation, Job, and scheduling resources are intentionally outside this generation scope; the agent can author only a bounded Workflow draft.

Core compares the parsed baseline and Proposal rather than trusting model prose. An unchanged/reformatted Workflow is rejected. A valid Proposal must contain at least one measurable gain in runtime bindings, semantic locators, evidence-based synchronization, failure recovery, guards/listeners, declared outputs, or redundant-step reduction. The bounded repair attempt receives these diagnostics and must patch the AST into an actually improved Workflow.

The Proposal never mutates the recording or current Studio canvas. Studio shows the live planning/generation/repair/check stream, elapsed time, evidence counts, the selected node and AST contracts, every generation/patch attempt, before/after workflow shape, model usage, Core diagnostics, and the generated YAML. Applying a valid Proposal requires a separate review action and a final confirmation; publishing remains a separate action.

MCP authoring loop

The external AI path uses:

recording_start -> recording_get -> recording_stop
-> recording_get_bundle -> workflow_node_catalog
-> workflow_validate_draft -> workflow_create/workflow_update_draft
-> workflow_publish (only when explicitly requested)

workflow_get_draft returns complete source and its revision. workflow_update_draft requires expectedRevision; a conflict must be resolved by reading and merging the newest draft rather than blindly retrying. This is the same concurrency boundary used by Studio.