Skip to content

duckdb_pglake: harden parallel partition scans - #733

Open
sfc-gh-dachristensen wants to merge 1 commit into
marcoslot/partitioned-postgres-scanfrom
pgguru/partitioned-postgres-scan-fixes
Open

sfc-gh-dachristensen wants to merge 1 commit into
marcoslot/partitioned-postgres-scanfrom
pgguru/partitioned-postgres-scan-fixes

Conversation

@sfc-gh-dachristensen

Copy link
Copy Markdown
Collaborator

Summary

Stacks on #653 to preserve PostgreSQL behavior while parallelizing scans of partitioned and inheritance tables.

  • Refresh member discovery during scan initialization in the scan transaction, and use ONLY when scanning members directly.
  • Query the parent when direct member reads could change grant or row-level security behavior, or when a pushed filter can benefit from partition pruning.
  • If worker connections cannot share an exported snapshot, run one parent query so the scan does not split across connections with different member snapshots.
  • Include pg_inherits membership in catalog staleness detection and reduce repeated partition metadata lookups.

Regression coverage

  • LIMIT/OFFSET across partition tasks, including distinct-row assertions.
  • Parent-only SELECT grants, parent row-level security, and filters on a partition key.
  • Reset pg_pages_per_task after tests that change it.

Validation

  • make release -j16: passed.
  • test_postgres_scanner.py: 37 passed, 12 warnings (Pydantic deprecations).
  • Focused LIMIT/OFFSET test: 1 passed.

🤖 Generated with Claude Code

Fall back to parent-table scans when grants or row-level security make direct member queries unsafe, and when filters can benefit from partition pruning. Refresh member discovery at scan start, track partition membership in the catalog staleness check, and use a parent scan if workers cannot share a snapshot.

Add regression coverage for LIMIT/OFFSET, parent-only grants, row-level security, partition-key filters, and session-setting cleanup.

Rebased onto #653 after the inheritance-parent split was dropped there (follow-up in #735). The member split applies to partitioned tables only again, and the inheritance-parent tests are gone.

Signed-off-by: David Christensen <david.christensen@snowflake.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@sfc-gh-dachristensen
sfc-gh-dachristensen force-pushed the pgguru/partitioned-postgres-scan-fixes branch from fdbdfb0 to 748198b Compare October 12, 2026 16:44
@sfc-gh-dachristensen

Copy link
Copy Markdown
Collaborator Author

I rebased this onto the new #653 head (f42e965), which no longer has the inheritance-parent split (follow-up in #735). Changes from the previous version:

  • Splitting by member is limited to partitioned tables again (relkind = 'p'). IsScannedByMember and the has_subclass column are gone, and the member queries go back to UNION ALL.
  • Member COPYs no longer add ONLY. Members are leaf partitions now, and those can't have inheritance children.
  • I removed the inheritance-parent fixtures and test_inheritance_parent_*. The LIMIT/OFFSET, grant, RLS and partition-key tests are unchanged.

Local checks: the regenerated patch set applies cleanly to the pinned duckdb-postgres, and the changed scanner sources compile. I haven't run the pytest suite locally; CI will cover it.

🤖 Generated with Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant