RP SQL on GCP#636
Conversation
✅ Deploy Preview for rp-cloud ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
📝 WalkthroughWalkthroughThe documentation expands Redpanda SQL BYOC support guidance from AWS to AWS or GCP. It adds GCS storage configuration, service-account and ADC authentication details, updated cloud-storage examples, and GCP endpoint guidance. Iceberg catalog documentation now supports GCP authentication with project IDs and includes a BigLake example. Related system table and prerequisite descriptions were updated, with TODOs marking pending GCP-specific documentation. Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
modules/reference/pages/sql/sql-statements/create-storage.adoc (1)
20-40: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winDocument the S3 options used by the how-to.
modules/sql/pages/query-data/query-iceberg-topics.adocusesendpointandpath_style, but this S3 options table lists onlyregion,access_key_id, andsecret_access_key. Either document those options here or remove them from the how-to so readers can distinguish supported from unsupported configuration.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@modules/reference/pages/sql/sql-statements/create-storage.adoc` around lines 20 - 40, Update the S3 options table in the create-storage documentation to include the endpoint and path_style options used by query-iceberg-topics.adoc, with their types, required status, and descriptions, so all demonstrated configuration is documented as supported.
🧹 Nitpick comments (1)
modules/sql/pages/get-started/deploy-sql-cluster.adoc (1)
96-99: 🎯 Functional Correctness | 🔵 TrivialResolve the GCP existing-cluster prerequisite before publishing this path.
The
//comments are not rendered, and the only visible prerequisite covers AWS on Line 99. If older GCP BYOC clusters require an agent-permission refresh, users following this new GCP path will fail without actionable instructions. Confirm the requirement and document the exact command, or explicitly state that existing GCP clusters are not yet supported.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@modules/sql/pages/get-started/deploy-sql-cluster.adoc` around lines 96 - 99, Resolve the existing-GCP-cluster prerequisite in the deploy SQL cluster documentation: confirm whether clusters created before Redpanda SQL launched on GCP require an agent-permission refresh, then either add a rendered NOTE with the exact required command alongside the AWS prerequisite or explicitly state that existing GCP BYOC clusters are not yet supported. Remove the unresolved TODO once the user-facing guidance is complete.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@modules/reference/pages/sql/sql-statements/copy-to.adoc`:
- Around line 179-185: Update the AWS_CRED example in the COPY documentation to
use a genuinely custom endpoint, such as an explicit <custom-endpoint>
placeholder or emulator URL, instead of https://storage.googleapis.com; leave
the surrounding guidance unchanged.
In `@modules/reference/pages/sql/sql-statements/create-iceberg-catalog.adoc`:
- Around line 44-47: Update the `gcp_project_id` description in the catalog
parameters table to explicitly state that the GCP project identifies where
BigLake API requests are attributed for billing and quota.
In `@modules/reference/pages/sql/sql-statements/create-storage.adoc`:
- Around line 79-101: Use distinct storage names in the GCS examples: rename the
service-account example to a unique name such as gcs_key_storage and the ADC
example to gcs_adc_storage, avoiding archive_storage reused by the S3 example.
Update each corresponding CREATE STORAGE statement consistently.
In `@modules/sql/pages/get-started/deploy-sql-cluster.adoc`:
- Around line 9-11: Update the Iceberg-related sentence in the deploy SQL
cluster page to qualify the cross-topic and Iceberg-history workflow by
supported cloud provider, excluding GCP until the referenced deployment and
BigLake support ship. Keep the general BYOC SQL availability statement unchanged
and preserve the existing Iceberg query reference for supported providers.
---
Outside diff comments:
In `@modules/reference/pages/sql/sql-statements/create-storage.adoc`:
- Around line 20-40: Update the S3 options table in the create-storage
documentation to include the endpoint and path_style options used by
query-iceberg-topics.adoc, with their types, required status, and descriptions,
so all demonstrated configuration is documented as supported.
---
Nitpick comments:
In `@modules/sql/pages/get-started/deploy-sql-cluster.adoc`:
- Around line 96-99: Resolve the existing-GCP-cluster prerequisite in the deploy
SQL cluster documentation: confirm whether clusters created before Redpanda SQL
launched on GCP require an agent-permission refresh, then either add a rendered
NOTE with the exact required command alongside the AWS prerequisite or
explicitly state that existing GCP BYOC clusters are not yet supported. Remove
the unresolved TODO once the user-facing guidance is complete.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: d7453a96-dfbd-45dd-8f0e-5a8e2e826754
📒 Files selected for processing (8)
modules/reference/pages/sql/sql-statements/copy-to.adocmodules/reference/pages/sql/sql-statements/create-iceberg-catalog.adocmodules/reference/pages/sql/sql-statements/create-storage.adocmodules/reference/pages/sql/system-virtual-tables.adocmodules/sql/pages/connect-to-sql/authenticate.adocmodules/sql/pages/get-started/deploy-sql-cluster.adocmodules/sql/pages/get-started/sql-quickstart.adocmodules/sql/pages/query-data/query-iceberg-topics.adoc
| `GCS_CRED` has no option to override the GCS endpoint. To target a custom endpoint (for example, a GCS emulator or the XML API), use `AWS_CRED` instead, since Google Cloud Storage supports an S3-compatible API: | ||
|
|
||
| [source,sql] | ||
| ---- | ||
| COPY default_redpanda_catalog=>project TO 'gs://your-bucket/project_file' | ||
| COPY default_redpanda_catalog=>project TO 'gs://<bucket-name>/<file-name>' | ||
| WITH (AWS_CRED(AWS_REGION 'region1', AWS_KEY_ID 'key_id', AWS_PRIVATE_KEY 'access_key', ENDPOINT 'https://storage.googleapis.com')); | ||
| ---- |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Use a genuinely custom endpoint in the custom-endpoint example.
The text describes targeting an emulator or custom endpoint, but the example uses the standard https://storage.googleapis.com endpoint. Use a placeholder such as <custom-endpoint> or an emulator URL so the example matches the stated use case.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@modules/reference/pages/sql/sql-statements/copy-to.adoc` around lines 179 -
185, Update the AWS_CRED example in the COPY documentation to use a genuinely
custom endpoint, such as an explicit <custom-endpoint> placeholder or emulator
URL, instead of https://storage.googleapis.com; leave the surrounding guidance
unchanged.
| // TODO: Drop the "On AWS" scoping once RPSQL Deployment with Iceberg on GCP | ||
| // (CIAINFRA-3422/M9) and BigLake REST catalog support (OXLA-9522, cloudv2 PR #27525) ship. | ||
| Enable Redpanda SQL on a Bring Your Own Cloud (BYOC) cluster so you can query streaming data in Redpanda topics using standard PostgreSQL syntax. You can also run queries that span both the streaming topic and its Iceberg history for Iceberg-enabled topics. See xref:sql:query-data/query-iceberg-topics.adoc[Query Iceberg-enabled topics] for that workflow. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Qualify the Iceberg claim by cloud provider.
The page now states that SQL is available on GCP, while the TODO indicates that GCP Iceberg deployment and BigLake support are not yet shipped. The unconditional claim on Line 11 therefore tells GCP users that the cross-topic/Iceberg workflow is available. Scope it to supported providers or remove it until GCP support is live.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@modules/sql/pages/get-started/deploy-sql-cluster.adoc` around lines 9 - 11,
Update the Iceberg-related sentence in the deploy SQL cluster page to qualify
the cross-topic and Iceberg-history workflow by supported cloud provider,
excluding GCP until the referenced deployment and BigLake support ship. Keep the
general BYOC SQL availability statement unchanged and preserve the existing
Iceberg query reference for supported providers.
1 participant
Description
This pull request expands Redpanda SQL's cloud support and storage integrations, adds support for Google Cloud Platform (GCP) in several workflows, and improves documentation for authentication and storage options. The most important changes are summarized below.
GCP Support and Documentation Updates
CREATE STORAGEstatement now supports both S3 and GCS types, with new options and examples for Google Cloud Storage, including authentication via service account key or Application Default Credentials (ADC). [1] [2] [3]Iceberg Catalog Enhancements
CREATE ICEBERG CATALOGstatement and system tables now support a newauth_type = 'gcp'option for connecting to BigLake REST catalogs on GCP, requiring agcp_project_id. Documentation and examples for this new authentication method have been added. [1] [2] [3]SQL COPY Command Improvements
COPY TOcommand documentation for exporting data to cloud storage has been updated for clarity and security. Placeholders are now used for credentials, and guidance is improved for authenticating with AWS, GCS (including ADC), and Azure. The use ofAWS_CREDfor GCS with HMAC keys is clarified. [1] [2]Other Improvements
These changes collectively make Redpanda SQL more accessible and secure for users on GCP, streamline authentication workflows, and enhance documentation for multi-cloud deployments.
Resolves https://github.com/redpanda-data/documentation-private/issues/
Review deadline: 24 July
Page previews
Checks