fix: harden WAL flush workers and merge-scan teardown under load - #111
Open
jamals86 wants to merge 25 commits into
Open
fix: harden WAL flush workers and merge-scan teardown under load#111jamals86 wants to merge 25 commits into
jamals86 wants to merge 25 commits into
Conversation
Prevent backend abort after merge-scan portal ERROR, page pending flush jobs with a stable keyset cursor, and keep recyclable WAL moving on flush fences without claiming apply progress in an uncommitted txn. Expand autonomous flush / teardown / journey coverage and retry slot-lock races in stress and passive convergence. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
Storage bench (PG 16, 10k rows)Comparison against base commit Results for commit
♻️ This comment has been updated with latest results. |
E2E (PG 16)207 tests +12 207 ✅ +12 14m 5s ⏱️ + 6m 0s Results for commit 87f0dfc. ± Comparison against base commit 43c2395. This pull request removes 1 and adds 13 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
E2E (PG 17)38 tests 37 ✅ 46s ⏱️ For more details on these failures, see this check. Results for commit 4b9ecc9. ♻️ This comment has been updated with latest results. |
E2E (PG 18)207 tests +12 207 ✅ +12 13m 45s ⏱️ + 5m 30s Results for commit 7f58b84. ± Comparison against base commit 43c2395. This pull request removes 1 and adds 13 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
E2E (PG 15)207 tests +12 207 ✅ +12 13m 46s ⏱️ + 5m 29s Results for commit 87f0dfc. ± Comparison against base commit 43c2395. This pull request removes 1 and adds 13 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
Prevent same-named tables in different schemas from sharing mirrors. Rehome mirror artifacts after source renames and refuse unsafe legacy shared-mirror teardown.
Move PG-free merge decisions into koldstore-merge, keep mirror/guard/index names within PostgreSQL's 63-byte limit, and update dependent tests plus release notice packaging so the full verification suite stays green. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
… roots. Ship dual Docker tags (PG18 multi-arch as latest, PG16 amd64, optional PG17 amd64) and fail closed when register_storage checks a non-empty filesystem base_path unless check => false. Co-authored-by: Cursor <cursoragent@cursor.com>
CI Debian package builds failed when curling the signing key timed out and left gpg with empty input. Prefer the vendored ACCC4CF8 key (with HTTPS fallbacks) in build/Docker paths. Co-authored-by: Cursor <cursoragent@cursor.com>
force_stop the async worker before DROP so supervisor relaunch cannot deadlock AccessExclusiveLock, and retry CASCADE on transient deadlock. Co-authored-by: Cursor <cursoragent@cursor.com>
Refresh the docs and SQL contract to distinguish a committed-WAL fence from read-your-own-uncommitted semantics, and to clarify that cold Parquet rows are not PostgreSQL heap tuples. The update documents current preview limits around DDL renames, backup/recovery expectations, security assumptions, and unsupported hot+cold behaviors. It also aligns the mirror apply comments and architecture docs with the current committed-change fence semantics.
Keep the WAL apply path off JSON: builtin int/bool keys stay native for in-batch identity, and order text is taken before PK cells so migration_order_by can reuse the same tuple. Co-authored-by: Cursor <cursoragent@cursor.com>
Switch rust-toolchain.toml to nightly and add .cargo/config.toml to use Cranelift codegen-backend and -Z threads=8 for faster local debug builds. Profile.dev/test use line-tables-only DWARF. CI and Docker set CARGO_ENCODED_RUSTFLAGS="" so stable rustc never receives -Z flags. Add E2E footprint tests: wal_applier_footprint (idle RSS vs sibling backend, restart SLO) and flush_executor_footprint (startup SLO, peak RSS, concurrent hot PK latency during encode). Introduce WorkerFootprintBudget with env-var overrides. Include both suites in run_memory_checks.sh. Document background worker process lifecycle in jobs-and-scheduler.md (what the 30-second intervals actually mean, fork model, dispatch path) and clarify that async_apply_watchdog_interval_ms is registered but not currently read by the applier loop.
Move the dev-only Cranelift setup out of committed Cargo config and document it as an optional local environment override, while pinning GitHub Actions and release jobs to `RUSTUP_TOOLCHAIN=1.96.0` so they do not pick up the repo's nightly toolchain or `-Z` flags. Also refresh a few shared Rust dependencies and apply small cleanup simplifications in flush planning, object-read stats, and pgoutput tag tests.
The WAL applier try-locks and yields instead of re-taking the advisory apply lock, so encode/upload stay concurrent with user commits. Also install clippy on the Examples + SQL CI job. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
scripts/run-all-tests.sh(fmt, clippy, unit, pg_test, e2e, examples, storage, SQL, memory, stress, benchmarks)merge_scan_teardown_crash_safe, auto-flush scheduler e2e, chat-penetration stress smokeMade with Cursor