Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,7 @@ Type `/help` in the panel for full signatures and flags. The most useful ones:
| `/schedule [prompt]` | Create a scheduled task |
| `/watch [--keep] [--secs <30-120>] [--long \| --short] <condition and action> [/beep]` | Poll the current page and act when a condition is met |
| `/workflow` · `/workflow --save <name>` | Manage saved workflows or compile the last successful run |
| `/teach --start <name>` · `/teach --end` | Learn a reusable workflow from your demonstrated actions |
| `/memory --add <text>` | Save a user preference |
| `/screenshot [--full-page]` | Capture the tab, or the full scrollable page |
| `/record [--transcribe]` | Record the current tab, optionally saving a transcript |
Expand Down
8 changes: 8 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -531,6 +531,14 @@ action CSS selectors, coordinates, query strings, fragments, and typed values. E
value becomes a declared runtime parameter; unsupported or failed actions are
skipped and reported to the user as save warnings.

Teacher mode is a second, value-free compiler input for the same schema.
`content/teacher-capture.js` observes only trusted top-document clicks, field
completion, checkbox/radio changes, Enter submissions, and navigation while a
tab-scoped session is active. Field values are never read or sent; a field
action becomes a runtime parameter marker as soon as it crosses the capture
boundary. The sanitized session survives navigation in `storage.session` and
is removed when `/teach --end` compiles and stores the workflow.

Each compiled step contains semantic target metadata (role, accessible name,
label, field identity, link, or placeholder), an expected postcondition, and
the origin/path family observed before that action. `/workflow` renders a
Expand Down
3 changes: 3 additions & 0 deletions docs/fr/slash-commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ d'une espace ouvre l'autocomplétion de ses options disponibles.
| `/workflow --delete <id>` | Supprimer un workflow enregistré |
| `/workflow --export <id>` | Télécharger un fichier JSON portable `webbrain-workflow/1` assaini |
| `/workflow --import --file` | Importer un fichier de workflow portable comme nouveau workflow local |
| `/teach` | Afficher l’état du mode d’apprentissage et le nombre d’actions capturées dans l’onglet actuel |
| `/teach --start <nom>` | Commencer à apprendre un workflow à partir de vos clics et modifications de champs |
| `/teach --end` | Arrêter l’apprentissage et compiler les actions en workflow enregistré sans valeurs saisies |
| `/allow-api` | **Dérogation de mutation API par conversation.** Voir [plus bas](#allow-api). |
| `/foreground [invite]` | Exécuter une tâche locale au premier plan pour assurer la compatibilité visuelle |
| `/dangerously-skip-permissions` | **Contournement global des demandes d'autorisation.** Désactive `Ask before consequential actions` sans ouvrir les Paramètres. WebBrain agira sans demandes par site jusqu'à ce que vous réactiviez le réglage. |
Expand Down
9 changes: 9 additions & 0 deletions docs/privacy-and-data-flow.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,15 @@ postconditions, and parameter descriptors. It does not contain typed field
values, raw historical `ref_id` values, action CSS selectors, coordinates, URL query strings, or URL
fragments.

`/teach --start <name>` records a user demonstration in temporary,
tab-scoped session storage. The page capture code never reads field values:
it sends only the field's semantic identity, and the compiler immediately
represents that action as a runtime parameter. Click/field target labels and
sanitized URL origin/path families are retained because replay needs them to
find the same controls. `/teach --end` removes the temporary session whether
compilation succeeds or fails; a successful compilation writes the same
`webbrain-workflow/1` format described above.

`/workflow --run <id>` collects declared values in a temporary side-panel form
and sends them directly to the background replay executor. The values are not
written to the workflow, chat text, retry payload, user memory, replay trace,
Expand Down
3 changes: 3 additions & 0 deletions docs/slash-commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ for its available flags.
| `/workflow --delete <id>` | Delete a saved workflow |
| `/workflow --export <id>` | Download a sanitized portable `webbrain-workflow/1` JSON file |
| `/workflow --import --file` | Import a portable workflow file as a new local saved workflow |
| `/teach` | Show the current tab's Teacher mode status and captured action count |
| `/teach --start <name>` | Start learning a saved workflow from your demonstrated clicks and field edits |
| `/teach --end` | Stop teaching and compile the captured actions into a value-free saved workflow |
| `/allow-api` | **Per-conversation API mutation override.** See [below](#allow-api). |
| `/foreground [prompt]` | Run one local task in the foreground for visual compatibility |
| `/dangerously-skip-permissions` | **Global permission-prompt bypass.** Turns off `Ask before consequential actions` without opening Settings. WebBrain will act without per-site prompts until you re-enable the setting. |
Expand Down
3 changes: 3 additions & 0 deletions docs/zh-CN/slash-commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,9 @@ WebBrain 接受作为输入框某行开头的斜杠命令。在面板内输入 `
| `/workflow --delete <id>` | 删除已保存的工作流 |
| `/workflow --export <id>` | 下载经过清洗的可移植 `webbrain-workflow/1` JSON 文件 |
| `/workflow --import --file` | 将可移植工作流文件导入为新的本地工作流 |
| `/teach` | 显示当前标签页的示教状态和已捕获操作数 |
| `/teach --start <名称>` | 开始从您演示的点击和字段编辑中学习工作流 |
| `/teach --end` | 结束示教,并将捕获的操作编译为不含实际输入值的已保存工作流 |
| `/allow-api` | **按对话的 API 变更覆盖。** 参见[下文](#allow-api)。 |
| `/foreground [提示词]` | 为视觉兼容性在前台运行一次本地任务 |
| `/dangerously-skip-permissions` | **全局权限提示绕过。** 无需打开设置即可关闭「操作前询问」。在你重新启用该设置之前,WebBrain 将不再按站点弹出提示。 |
Expand Down
1 change: 1 addition & 0 deletions src/chrome/manifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@
"js": [
"src/content/rich-text-toolbar-heuristic.js",
"src/content/accessibility-tree.js",
"src/content/teacher-capture.js",
"src/content/content.js",
"src/content/agent-visual-indicator.js",
"src/content/ollama-launch-handoff.js"
Expand Down
167 changes: 114 additions & 53 deletions src/chrome/src/agent/agent.js
Original file line number Diff line number Diff line change
Expand Up @@ -558,6 +558,7 @@ export class Agent extends LoopDetector {
this._recentSubmitClicks = new Map(); // tabId -> recent submit click timestamps
this._formValidationBlocks = new Map(); // tabId -> validation state that must change before another submit
this._runningTabs = new Set(); // tabIds with an active processMessage/Stream in flight
this._runStartGuard = null;
// Ordinary runs capture and act in their original tab without activating
// it. Managed cloud runs and the explicit /foreground compatibility
// override retain the old Page.bringToFront behavior for their lifetime.
Expand Down Expand Up @@ -896,6 +897,38 @@ export class Agent extends LoopDetector {
this._conversationScopeChangeListener = typeof listener === 'function' ? listener : null;
}

setRunStartGuard(guard) {
this._runStartGuard = typeof guard === 'function' ? guard : null;
}

_runEntryKind(defaultKind, runOptions = {}) {
if (runOptions?.cloudRun === true) return 'cloud';
if (runOptions?.scheduledRun === true) return 'scheduled';
return defaultKind;
}

async assertRunStartAllowed(tabId, defaultKind = 'interactive', runOptions = {}) {
await this._runStartGuard?.(tabId, {
kind: this._runEntryKind(defaultKind, runOptions),
runOptions,
});
}

async _claimRunEntry(tabId, defaultKind, runOptions = {}) {
if (this._runningTabs.has(tabId)) {
throw new Error('An agent run is already in progress for this tab.');
}
// Claim synchronously before awaiting the external guard so teacher-mode
// startup cannot race a run whose persisted teacher-state check is pending.
this._runningTabs.add(tabId);
try {
await this.assertRunStartAllowed(tabId, defaultKind, runOptions);
} catch (error) {
this._runningTabs.delete(tabId);
throw error;
}
}

isRunning(tabId) {
return this._runningTabs.has(tabId);
}
Expand Down Expand Up @@ -15576,44 +15609,68 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
};
}

async _endSavedWorkflowTraceRun(runId, status, finalContent) {
await trace.endRun(runId, { status, finalContent });
}

async replaySavedWorkflow(tabId, workflow, parameters = {}, onUpdate = () => {}, runOptions = {}) {
if (this._runningTabs.has(tabId)) throw new Error('An agent run is already in progress for this tab.');
if (!workflow?.id || !Array.isArray(workflow.steps) || !workflow.steps.length) {
throw new Error('Saved workflow is missing or invalid.');
}
await this._hydrate(tabId);
// Deterministic workflow replay is an independent task even when it never
// falls back to processMessage. Clear stale selected-text grounding before
// any replay action so a successful replay cannot poison the next turn.
this._clearSelectionGroundingForIndependentRun(tabId, { ...runOptions, independentRun: true });
// Align pre-run cleanup with processMessage so a prior Act turn cannot
// leak click-AX CDP fallbacks, plan guards, or active-skill state into
// deterministic replay (or the reverse on the next turn).
this._resetActiveSkillsForRun(tabId, { refreshPrompt: false });
this._clearRunLoopState(tabId);
this._resetRichTextToolbarAudit(tabId);
this._clickAxCdpFallbacks?.delete(tabId);
this.abortFlags.delete(tabId);
this._prepareClarificationAuthorizationForRun(tabId);
this.permissions.beginTurn(tabId);
this.conversationModes.set(tabId, 'act');
const completionRunToken = this._beginCompletionInvariant(tabId);
this._runningTabs.add(tabId);
const previousForegroundCapture = this._configureCapturePolicyForRun(tabId, runOptions);
const startUrl = await this._currentUrl(tabId);
const conversationId = await this.ensureConversationId(tabId, 'act');
const traceRunId = await trace.startRun({
conversationId,
userMessage: `Run saved workflow: ${workflow.name}`,
tabUrl: startUrl,
mode: 'act',
model: this.providerManager?.getActive?.()?.model || '',
providerId: this.providerManager?.activeProviderId || '',
runtimeConfig: this._runtimeTraceConfig(this.providerManager?.getActive?.(), {
tabId,
await this._claimRunEntry(tabId, 'workflow', runOptions);
let completionRunToken = '';
let previousForegroundCapture = false;
let capturePolicyConfigured = false;
let startUrl = '';
let traceRunId = null;
try {
await this._hydrate(tabId);
// Deterministic workflow replay is an independent task even when it never
// falls back to processMessage. Clear stale selected-text grounding before
// any replay action so a successful replay cannot poison the next turn.
this._clearSelectionGroundingForIndependentRun(tabId, { ...runOptions, independentRun: true });
// Align pre-run cleanup with processMessage so a prior Act turn cannot
// leak click-AX CDP fallbacks, plan guards, or active-skill state into
// deterministic replay (or the reverse on the next turn).
this._resetActiveSkillsForRun(tabId, { refreshPrompt: false });
this._clearRunLoopState(tabId);
this._resetRichTextToolbarAudit(tabId);
this._clickAxCdpFallbacks?.delete(tabId);
this.abortFlags.delete(tabId);
this._prepareClarificationAuthorizationForRun(tabId);
this.permissions.beginTurn(tabId);
this.conversationModes.set(tabId, 'act');
completionRunToken = this._beginCompletionInvariant(tabId);
previousForegroundCapture = this._configureCapturePolicyForRun(tabId, runOptions);
capturePolicyConfigured = true;
startUrl = await this._currentUrl(tabId);
const conversationId = await this.ensureConversationId(tabId, 'act');
traceRunId = await trace.startRun({
conversationId,
userMessage: `Run saved workflow: ${workflow.name}`,
tabUrl: startUrl,
mode: 'act',
}),
});
model: this.providerManager?.getActive?.()?.model || '',
providerId: this.providerManager?.activeProviderId || '',
runtimeConfig: this._runtimeTraceConfig(this.providerManager?.getActive?.(), {
tabId,
mode: 'act',
}),
});
} catch (error) {
try {
if (capturePolicyConfigured) {
await this._restoreCapturePolicyAfterRun(tabId, previousForegroundCapture);
}
} finally {
try {
if (completionRunToken) this._clearCompletionInvariant(tabId, completionRunToken);
} finally {
this._runningTabs.delete(tabId);
}
}
throw error;
}
let traceStatus = 'workflow_stopped';
let finalContent = '';
let matchedSteps = 0;
Expand Down Expand Up @@ -15859,16 +15916,28 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
healings: verifiedHealings,
};
} finally {
await trace.endRun(traceRunId, { status: traceStatus, finalContent });
this.currentCostState.delete(tabId);
this._planExecutionGuards.delete(tabId);
this._resetActiveSkillsForRun(tabId);
await this._restoreCapturePolicyAfterRun(tabId, previousForegroundCapture);
this._runningTabs.delete(tabId);
this._clearRunLoopState(tabId);
if (traceStatus !== 'workflow_fallback') this._resetRichTextToolbarAudit(tabId);
this._clickAxCdpFallbacks?.delete(tabId);
this._clearCompletionInvariant(tabId, completionRunToken);
try {
try {
await this._endSavedWorkflowTraceRun(traceRunId, traceStatus, finalContent);
} finally {
this.currentCostState.delete(tabId);
this._planExecutionGuards.delete(tabId);
this._resetActiveSkillsForRun(tabId);
this._clearRunLoopState(tabId);
if (traceStatus !== 'workflow_fallback') this._resetRichTextToolbarAudit(tabId);
this._clickAxCdpFallbacks?.delete(tabId);
}
} finally {
try {
await this._restoreCapturePolicyAfterRun(tabId, previousForegroundCapture);
} finally {
try {
this._clearCompletionInvariant(tabId, completionRunToken);
} finally {
this._runningTabs.delete(tabId);
}
}
}
}
}

Expand Down Expand Up @@ -21948,10 +22017,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
}

async processMessage(tabId, userMessage, onUpdate = () => {}, mode = 'ask', attachments = [], runOptions = {}) {
if (this._runningTabs.has(tabId)) {
throw new Error('An agent run is already in progress for this tab.');
}
this._runningTabs.add(tabId);
await this._claimRunEntry(tabId, 'interactive', runOptions);
try {
// Hydration has to run before the toolbar-ledger reset below, so a
// persisted obligation cannot outlive the run that cleared it. It is
Expand All @@ -21972,7 +22038,6 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
if (runOptions?.trustedContinuation !== true) this._continuationExecutionEvidence.delete(tabId);
this._clickAxCdpFallbacks.delete(tabId);
const completionRunToken = this._beginCompletionInvariant(tabId);
this._runningTabs.add(tabId);
if (runOptions?.trustedContinuation !== true) this.persistenceDegradedTabs.delete(tabId);
this._runUpdateCallbacks.set(tabId, onUpdate);
const previousForegroundCapture = this._configureCapturePolicyForRun(tabId, runOptions);
Expand Down Expand Up @@ -22874,10 +22939,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
* Process a message with streaming output.
*/
async processMessageStream(tabId, userMessage, onUpdate = () => {}, mode = 'ask', runOptions = {}) {
if (this._runningTabs.has(tabId)) {
throw new Error('An agent run is already in progress for this tab.');
}
this._runningTabs.add(tabId);
await this._claimRunEntry(tabId, 'interactive', runOptions);
try {
// Hydration has to run before the toolbar-ledger reset below, so a
// persisted obligation cannot outlive the run that cleared it. It is
Expand All @@ -22898,7 +22960,6 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
if (runOptions?.trustedContinuation !== true) this._continuationExecutionEvidence.delete(tabId);
this._clickAxCdpFallbacks.delete(tabId);
const completionRunToken = this._beginCompletionInvariant(tabId);
this._runningTabs.add(tabId);
if (runOptions?.trustedContinuation !== true) this.persistenceDegradedTabs.delete(tabId);
this._runUpdateCallbacks.set(tabId, onUpdate);
const previousForegroundCapture = this._configureCapturePolicyForRun(tabId, runOptions);
Expand Down
47 changes: 40 additions & 7 deletions src/chrome/src/agent/scheduler.js
Original file line number Diff line number Diff line change
Expand Up @@ -585,6 +585,10 @@ export class ScheduledJobManager {
this.restoreAlarms().catch((e) => console.warn('[WebBrain] restore scheduled alarms failed:', e));
}

isRunning(tabId) {
return this._runningTabs.has(tabId);
}

async _getJobs() {
const stored = await this.api.storage.local.get(SCHEDULED_JOBS_KEY);
const jobs = stored?.[SCHEDULED_JOBS_KEY];
Expand Down Expand Up @@ -1523,20 +1527,49 @@ export class ScheduledJobManager {
}

let tabId;
let reservedTabId = null;
const releaseReservation = () => {
if (reservedTabId == null) return;
this._runningTabs.delete(reservedTabId);
reservedTabId = null;
};
const reserveTab = async (candidateTabId) => {
if (candidateTabId == null) return;
const numericTabId = Number(candidateTabId);
const resolvedTabId = Number.isFinite(numericTabId) ? numericTabId : candidateTabId;
if (this._runningTabs.has(resolvedTabId) || this.agent.isRunning(resolvedTabId)) {
throw new Error('An agent run is already in progress for this tab.');
}
this._runningTabs.add(resolvedTabId);
reservedTabId = resolvedTabId;
try {
await this.agent.assertRunStartAllowed?.(resolvedTabId, 'scheduled', { scheduledRun: true });
} catch (error) {
releaseReservation();
throw error;
}
};
try {
const candidateTabId = job.kind === 'resume' || job.target?.type === 'current_tab'
? (job.tabId || job.target?.tabId)
: job.target?.tabId;
await reserveTab(candidateTabId);
tabId = await this._resolveTab(job);
if (reservedTabId !== tabId) {
releaseReservation();
await reserveTab(tabId);
}
await this._validateConversation(job, tabId);
await this._validateTaskTarget(job, tabId);
} catch (e) {
await this._markFailed(job, e.message);
return;
}

if (this._runningTabs.has(tabId) || this.agent.isRunning(tabId)) {
await this._requeue(job, 'The target tab already has an active WebBrain run.');
releaseReservation();
if (isActiveRunError(e) || e?.code === 'teacher_mode_active') {
await this._requeue(job, 'The target tab already has an active WebBrain run.');
} else {
await this._markFailed(job, e.message);
}
return;
}
this._runningTabs.add(tabId);

const running = await this._updateJobIf(job.id, (prev) => (
['pending', 'queued'].includes(prev.status)
Expand Down
Loading
Loading