Skip to content

fix: roll back incomplete archive schema creation - #335

Merged
vincentkoc merged 1 commit into
mainfrom
improve/profile-archive-startup
Nov 21, 2026
Merged

vincentkoc merged 1 commit into
mainfrom
improve/profile-archive-startup

Conversation

@vincentkoc

@vincentkoc vincentkoc commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

What Problem This Solves

A failed first archive initialization can leave earlier tables, indexes and FTS objects committed. Retrying then sees a partial archive. Creating those objects through separate commits also adds startup work.

User Impact

Fresh schema creation now rolls back its DDL when a schema statement fails, while preserving pre-existing data. After the conflict is repaired, initialization can be retried. Existing-archive repair, migrations and journal initialization keep their current behavior.

Why This Change Was Made

Apply connection pragmas before one transaction for fresh schema objects. Keep the same schema SQL for existing archives. Regression tests cover early failure, failure after FTS creation, preserved data, successful retry and connection settings.

Evidence

  • Baseline leaves 9 and 28 new schema objects in the two failure cases; the candidate leaves none and passes both retries.

  • Five alternating local benchmark pairs: fresh-open median 5.068 ms → 3.599 ms (29% lower), with one slower candidate pair. Reopen measurements were noisy and do not establish an improvement.

  • Fresh baseline ns/op: 5068210, 10243985, 4749745, 4137878, 5672416. Candidate: 3144358, 3599025, 3430797, 6358074, 5410978.

  • Independent P0–P3 review: scoped-clean.

  • Full local tests, vet and module checks passed with GOWORK=off and CGO_ENABLED=0 in isolated temporary home/config directories. Hosted Linux race and native macOS validation remain required.

  • These timings measure synthetic archive startup, not whole-CI duration or billing savings.

  • Hosted CI run 44767958752 passed on cea7bcb3f4c034200e6d5a9fae92a0c37bf82a9f, including Linux race, native macOS, lint, dependencies and snapshot builds. Docker, CodeQL and both secret scans also passed.

@clawsweeper

clawsweeper Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@vincentkoc
vincentkoc marked this pull request as ready for review November 20, 2026 16:03
@clawsweeper clawsweeper Bot added clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. P2 Normal priority bug or improvement with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. labels Nov 20, 2026
@clawsweeper

clawsweeper Bot commented Nov 20, 2026

Copy link
Copy Markdown

Codex review: needs maintainer review before merge.

What this changes

The PR wraps fresh SQLite schema creation in one transaction so failed initialization rolls back newly created objects.

Example: An unversioned archive contains a table named idx_sync_state_updated with a sentinel value of 'retained'.

  • Before: Initialization fails at the conflicting index name after committing earlier schema objects.
  • After: Initialization fails without retaining new schema objects, preserves 'retained', and succeeds after the conflicting table is removed.

Review scores

Measure Result What it means
Overall readiness 🧂 unranked krab (1/6) The patch is focused and source-correct with meaningful regression coverage, but the required real-behavior evidence is not supplied.
Proof confidence 🧂 unranked krab (1/6) Real behavior proof is not required for maintainer- or bot-authored pull requests.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Product

Kind: Bug fix · Worth it: Yes · Fix scope: Complete
User problem: A failed first archive open can leave a partial schema that complicates retry.
Reason: Atomic fresh DDL directly addresses the reported failure, and the maintainer-authored PR establishes direction.

Merge readiness

✅ Ready for maintainer review

Keep open: this maintainer-authored PR fixes a concrete initialization defect still present on main, and source review found no introduced correctness bug.

Priority: P2
Reviewed head: cea7bcb3f4c034200e6d5a9fae92a0c37bf82a9f

Before merge

None.

Findings

None.

Tests

  • Missing end-to-end proof: Provide redacted terminal output from a built CLI showing failed creation with a conflicting idx_sync_state_updated table, preserved sentinel data and no new objects, then successful retry after removing the conflict.
Agent review details

How this fits together

Database-backed CLI commands open the archive through Crawlkit; Slacrawl selects fresh creation, migration, or repair and returns a usable SQLite store or an error.

flowchart TD
  A[Database backed CLI command] --> B[Crawlkit SQLite connection]
  B --> C[Slacrawl schema version check]
  C --> D[Fresh schema transaction]
  C --> E[Existing archive migration or repair]
  D --> F[Journal and search initialization]
  E --> F
  F --> G[Usable archive store]
Loading

Technical review

Best possible solution:

Keep the focused transaction around fresh schema objects while preserving connection pragmas and existing-archive migration and repair behavior.

Do we have a high-confidence way to reproduce the issue?

The current-main execution path and concrete conflicting-table fixtures establish the failure mechanism; the author reports baseline leftovers, but this read-only review did not execute target code.

Is this the best way to solve the issue?

The patch fixes the transaction boundary in the existing schema owner, preserves the existing SQL and archive paths, and adds no competing initialization mechanism.

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning medium; reviewed against 35493fb5f717.

Provenance checked

  • Fresh creation versus existing-archive migration keeps the original intent (9e107d8: The historical change separates empty databases from legacy archives so older schemas migrate instead of receiving only current CREATE IF NOT EXISTS statements.)
  • Schema SQL placement and definitions keeps the original intent (refactor: split persistence by schema, writes, queries and snapshots #180: The persistence refactor moved declarations without behavior, migration, archive-format, or public API changes.)
  • Journal initialization after application schema creation keeps the original intent (b8a8fd1: Native archive journaling initializes its objects and writes the fresh schema version in its own transaction.)

Testing

Proof path: in-process harness.

Security

None.

Evidence

What I checked:

  • Pinned change and current-main defect: Main executes the fresh schema without an explicit transaction; the introduced diff changes only that branch, separates pragmas from otherwise unchanged DDL, and adds rollback coverage. (internal/store/store.go:50, cea7bcb3f4c0)
  • Distinct rollback cases: The added test injects an early message_files column conflict and a late index-name conflict after FTS creation, then checks absence of new objects, preserved sentinel data, unchanged version, successful retry, and connection settings. (internal/store/startup_test.go:13, cea7bcb3f4c0)
  • Verified no-migration case: The schema version, table definitions, indexes, FTS definition, migration branch, existing-archive repair branch, and journal initialization remain unchanged; only fresh DDL transaction boundaries change, so existing stored archives require no new migration. (internal/store/schema.go:61, cea7bcb3f4c0)
  • Imported connection contract and applicable boundary policy: Slacrawl imports this pinned store implementation. Its default pool keeps one open and idle connection, and its DSN applies SQLite pragmas; its AGENTS.md and boundary document retain application schema ownership downstream. This supports applying pragmas before the transaction without changing the shared API. (store/store.go, 5276b56d7422)
  • Hosted validation: The linked CI run completed successfully on the reviewed head, including Linux race tests, native macOS tests, CLI smoke, lint, dependency checks, and snapshot builds: https://github.com/openclaw/slacrawl/actions/runs/44767958752. (.github/workflows/ci.yml:85, cea7bcb3f4c0)
  • Cancelled dispatch did not execute: The cancelled dispatch job has no steps; its annotation attributes cancellation to a higher-priority waiting dispatch request, rather than this diff. (cea7bcb3f4c0)

Likely related people:

  • Vincent Koc: Raw commit b8a8fd1 adds internal/store/store.go:65 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: b8a8fd15d7b6; files: internal/store/store.go)
  • Peter Steinberger: Raw commit 9e107d8 adds internal/store/store.go:448 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: 9e107d896e54; files: internal/store/store.go)

Preflight source

Source unknown. Source comparison at the observed PR snapshot only. Completion is self-reported; body, policy, tests, execution, and report bytes are not verified.

Review metrics

Metric Value Why it matters
Change composition Production +23/-2; tests and benchmarks +101; changelog +3 The small production change is supported by two distinct failure cases and startup measurements.

Labels

Label changes:

  • add P2: A schema statement failure can leave an incomplete archive, with impact limited to failed fresh initialization.
  • add rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Label justifications:

  • P2: A schema statement failure can leave an incomplete archive, with impact limited to failed fresh initialization.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed November 20, 2026, 11:16 AM ET / 16:16 UTC.

@vincentkoc
vincentkoc merged commit 5b1fc3c into main Nov 21, 2026
28 of 29 checks passed
@vincentkoc
vincentkoc deleted the improve/profile-archive-startup branch November 21, 2026 01:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix other P2 Normal priority bug or improvement with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant