Skip to content

RP SQL on GCP #636

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
kbatuigas wants to merge 5 commits into
base: main
Choose a base branch
from
Open

RP SQL on GCP #636

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
63c0692
Add GCP support for RP SQL (BYOC)
kbatuigas Jul 15, 2026
2ea7fb7
TODOs for Iceberg/BigLake
kbatuigas Jul 15, 2026
9a826db
Review pass
kbatuigas Jul 15, 2026
80528b7
Update from source verification
kbatuigas Jul 22, 2026
1b20cb3
Apply suggestions from automated review
kbatuigas Jul 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions modules/reference/pages/sql/sql-statements/copy-to.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,38 +155,38 @@ To export data to cloud storage, use the `COPY TO` command with the appropriate

[source,sql]
----
COPY default_redpanda_catalog=>film TO 's3://your-bucket/file_name'
WITH (AWS_CRED(AWS_REGION 'us-west-1', AWS_KEY_ID 'key_id', AWS_PRIVATE_KEY 'access_key', ENDPOINT 's3.us-west-1.amazonaws.com'),
COPY default_redpanda_catalog=>film TO 's3://<bucket-name>/<file-name>'
WITH (AWS_CRED(AWS_REGION 'us-west-1', AWS_KEY_ID '<aws-key-id>', AWS_PRIVATE_KEY '<access-key>', ENDPOINT 's3.us-west-1.amazonaws.com'),
FORMAT CSV, HEADER ON, NULL 'unknown');
----

==== Google Cloud Storage

Pass the path to your credentials JSON file:
Pass the contents of your service account key JSON file to `GCS_CRED`:

[source,sql]
----
COPY default_redpanda_catalog=>film TO 'gs://your-bucket/file_name' WITH (GCS_CRED('/path/to/credentials.json'));
COPY default_redpanda_catalog=>film TO 'gs://<bucket-name>/<file-name>' WITH (GCS_CRED('<credentials-json-contents>'));
----

If you cannot use a path to the credentials file, pass its contents as a string:
`GCS_CRED` is optional. If you omit it, Redpanda SQL authenticates using Application Default Credentials (ADC), which includes Workload Identity Federation on Google Kubernetes Engine (GKE):

[source,sql]
----
COPY default_redpanda_catalog=>project TO 'gs://your-bucket/project_file' WITH (GCS_CRED('<contents of the credentials.json file>'));
COPY default_redpanda_catalog=>project TO 'gs://<bucket-name>/<file-name>';
----

You can also use `AWS_CRED` with GCS by setting the endpoint:
`GCS_CRED` has no option to override the GCS endpoint. To target a custom endpoint (for example, a GCS emulator or the XML API), use `AWS_CRED` instead, since Google Cloud Storage supports an S3-compatible API:

[source,sql]
----
COPY default_redpanda_catalog=>project TO 'gs://your-bucket/project_file'
COPY default_redpanda_catalog=>project TO 'gs://<bucket-name>/<file-name>'
WITH (AWS_CRED(AWS_REGION 'region1', AWS_KEY_ID 'key_id', AWS_PRIVATE_KEY 'access_key', ENDPOINT 'https://storage.googleapis.com'));
----
Comment on lines +179 to 185

@coderabbitai coderabbitai Bot Jul 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Use a genuinely custom endpoint in the custom-endpoint example.

The text describes targeting an emulator or custom endpoint, but the example uses the standard https://storage.googleapis.com endpoint. Use a placeholder such as <custom-endpoint> or an emulator URL so the example matches the stated use case.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@modules/reference/pages/sql/sql-statements/copy-to.adoc` around lines 179 -
185, Update the AWS_CRED example in the COPY documentation to use a genuinely
custom endpoint, such as an explicit <custom-endpoint> placeholder or emulator
URL, instead of https://storage.googleapis.com; leave the surrounding guidance
unchanged.


[TIP]
====
For Google Cloud Storage, use HMAC keys for authentication. See the link:https://cloud.google.com/storage/docs/authentication/hmackeys[HMAC keys - Cloud Storage^] page for details.
The `AWS_CRED` approach requires HMAC keys for authentication. See the link:https://cloud.google.com/storage/docs/authentication/hmackeys[HMAC keys - Cloud Storage^] page for details.
====

==== Azure Blob Storage
Expand All @@ -197,6 +197,6 @@ For Google Cloud Storage, use HMAC keys for authentication. See the link:https:/

[source,sql]
----
COPY default_redpanda_catalog=>taxi_data TO 'wasbs://container-name/your_blob'
COPY default_redpanda_catalog=>taxi_data TO 'wasbs://<container-name>/<blob-name>'
WITH (AZURE_CRED(TENANT_ID 'your_tenant_id' CLIENT_ID 'your_client_id', CLIENT_SECRET 'your_client_secret'));
----
22 changes: 21 additions & 1 deletion modules/reference/pages/sql/sql-statements/create-iceberg-catalog.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,12 @@ NOTE: Catalogs are created in the current schema (`public` by default). To creat
|`auth_type`
|STRING
|No
|Authentication type for the REST catalog. One of `oauth2`, `basic`, or `aws_sigv4`. If omitted, the catalog connects without authentication. Providing an auth-specific option (such as `username` or `aws_region`) without `auth_type` is rejected.
|Authentication type for the REST catalog. One of `oauth2`, `basic`, `aws_sigv4`, or `gcp`. If omitted, the catalog connects without authentication. Providing an auth-specific option (such as `username`, `aws_region`, or `gcp_project_id`) without `auth_type` is rejected.

|`gcp_project_id`
|STRING
|Required when `auth_type = 'gcp'`
|GCP project ID for the BigLake catalog. Attributes API calls for billing and quota.
Comment thread
kbatuigas marked this conversation as resolved.
Outdated

|`oauth2_client_id`
|STRING
Expand Down Expand Up @@ -215,6 +220,21 @@ CREATE ICEBERG CATALOG glue_cat STORAGE iceberg_storage
);
----

=== Create an Iceberg catalog for BigLake on GCP

Use `auth_type = 'gcp'` for a Google Cloud BigLake REST catalog. Redpanda SQL authenticates with Application Default Credentials (ADC), which on Google Kubernetes Engine (GKE) resolves to Workload Identity, so no static key is required. The `gcp_project_id` option is required. Set `uri` and `warehouse` to the values for your BigLake catalog.

[source,sql]
----
CREATE ICEBERG CATALOG biglake_cat STORAGE iceberg_storage
WITH (
uri = '<biglake-rest-catalog-uri>',
warehouse = '<warehouse>',
auth_type = 'gcp',
gcp_project_id = '<gcp-project-id>'
);
----

=== Create an Iceberg catalog scoped to specific namespaces (nested backend)

For backends that support nested namespaces (such as Polaris), use multi-segment dotted paths in `allowed_namespaces`.
Expand Down
57 changes: 53 additions & 4 deletions modules/reference/pages/sql/sql-statements/create-storage.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,22 +2,22 @@
:description: The CREATE STORAGE statement creates a connection to external object storage for use with Redpanda catalogs.
:page-topic-type: reference

The `CREATE STORAGE` statement creates a named connection to external object storage such as Amazon S3.
The `CREATE STORAGE` statement creates a named connection to external object storage such as Amazon S3 or Google Cloud Storage (GCS).

== Syntax

[source,sql]
----
CREATE STORAGE [IF NOT EXISTS] storage_name
TYPE = S3
TYPE = S3 | GCS
WITH (option = 'value' [, ...]);
----

* `storage_name`: Name for the new storage connection.
* `TYPE`: Storage type. Redpanda SQL currently supports only `S3`.
* `TYPE`: Storage type. Redpanda SQL supports `S3` and `GCS`.
* `IF NOT EXISTS`: Optional. Prevents an error if a storage connection with the same name already exists.

== Options
== S3 options

[cols="<30%,<15%,<10%,<45%",options="header"]
|===
Expand All @@ -39,8 +39,32 @@ WITH (option = 'value' [, ...]);
|AWS secret access key.
|===

== GCS options

[cols="<30%,<15%,<10%,<45%",options="header"]
|===
|Option |Type |Required |Description

|`url`
|STRING
|No
|GCS bucket URL, in the form `gs://<bucket>/<path>`. If omitted, the connection holds credentials only and has no associated bucket.

|`service_account_key`
|STRING
|No
|Contents of a GCP service account key JSON file. If omitted, Redpanda SQL authenticates using Application Default Credentials (ADC), which includes Workload Identity Federation on Google Kubernetes Engine (GKE).

|`endpoint`
|STRING
|No
|Override the default GCS endpoint.
|===

== Examples

=== Create an S3 storage connection

[source,sql]
----
CREATE STORAGE archive_storage
Expand All @@ -51,3 +75,28 @@ WITH (
secret_access_key = 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY'
);
----

=== Create a GCS storage connection with a service account key

[source,sql]
----
CREATE STORAGE archive_storage
TYPE = GCS
WITH (
url = 'gs://archive-bucket/redpanda-sql',
service_account_key = '{"type": "service_account", "project_id": "my-project", ...}'
);
----

=== Create a GCS storage connection using Application Default Credentials

Omit `service_account_key` to authenticate with Application Default Credentials (ADC). On GKE, this includes Workload Identity Federation.

[source,sql]
----
CREATE STORAGE archive_storage
TYPE = GCS
WITH (
url = 'gs://archive-bucket/redpanda-sql'
);
Comment thread
kbatuigas marked this conversation as resolved.
----
2 changes: 1 addition & 1 deletion modules/reference/pages/sql/system-virtual-tables.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ Iceberg catalog connections, with their REST endpoint and authentication details
|`name` |`text` |No |Iceberg catalog name.
|`uri` |`text` |No |REST catalog endpoint URI.
|`warehouse` |`text` |No |Iceberg warehouse identifier or location.
|`auth_type` |`text` |No |Authentication type for the REST catalog. One of `oauth2`, `basic`, `aws_sigv4`, or empty if the catalog connects without authentication.
|`auth_type` |`text` |No |Authentication type for the REST catalog. One of `oauth2`, `basic`, `aws_sigv4`, `gcp`, or empty if the catalog connects without authentication.
|`namespace_name` |`text` |No |Schema containing the catalog.
|`database_name` |`text` |No |Database containing the catalog.
|===
Expand Down
2 changes: 1 addition & 1 deletion modules/sql/pages/connect-to-sql/authenticate.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ After completing these steps, you will be able to:

== Prerequisites

* A Redpanda BYOC cluster on AWS with Redpanda SQL enabled. See xref:sql:get-started/deploy-sql-cluster.adoc[Enable Redpanda SQL].
* A Redpanda BYOC cluster on AWS or GCP with Redpanda SQL enabled. See xref:sql:get-started/deploy-sql-cluster.adoc[Enable Redpanda SQL].
* https://www.postgresql.org/download/[`psql`^] or another PostgreSQL-compatible client.
* A user or service account assigned a role with the *SQL: Access* or *SQL: Manage* permission in Redpanda Cloud's data-plane RBAC. Redpanda Cloud provisions a corresponding user in the SQL engine when the role is assigned. To assign roles, go to *Organization IAM > Roles*; SQL permissions are under the *Data Plane* tab when you create or edit a role. See xref:security:authorization/rbac/rbac_dp.adoc[Configure RBAC in the Data Plane].

Expand Down
16 changes: 11 additions & 5 deletions modules/sql/pages/get-started/deploy-sql-cluster.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,17 @@
:learning-objective-2: Scale or disable the SQL engine
:learning-objective-3: Verify that the SQL engine is running and ready to accept connections

Enable Redpanda SQL on a Bring Your Own Cloud (BYOC) cluster so you can query streaming data in Redpanda topics using standard PostgreSQL syntax. For Iceberg-enabled topics, queries can span both the streaming topic and its Iceberg history. See xref:sql:query-data/query-iceberg-topics.adoc[Query Iceberg-enabled topics] for that workflow.
// TODO: Drop the "On AWS" scoping once RPSQL Deployment with Iceberg on GCP
// (CIAINFRA-3422/M9) and BigLake REST catalog support (OXLA-9522, cloudv2 PR #27525) ship.
Enable Redpanda SQL on a Bring Your Own Cloud (BYOC) cluster so you can query streaming data in Redpanda topics using standard PostgreSQL syntax. You can also run queries that span both the streaming topic and its Iceberg history for Iceberg-enabled topics. See xref:sql:query-data/query-iceberg-topics.adoc[Query Iceberg-enabled topics] for that workflow.
Comment on lines +9 to +11

@coderabbitai coderabbitai Bot Jul 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Qualify the Iceberg claim by cloud provider.

The page now states that SQL is available on GCP, while the TODO indicates that GCP Iceberg deployment and BigLake support are not yet shipped. The unconditional claim on Line 11 therefore tells GCP users that the cross-topic/Iceberg workflow is available. Scope it to supported providers or remove it until GCP support is live.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@modules/sql/pages/get-started/deploy-sql-cluster.adoc` around lines 9 - 11,
Update the Iceberg-related sentence in the deploy SQL cluster page to qualify
the cross-topic and Iceberg-history workflow by supported cloud provider,
excluding GCP until the referenced deployment and BigLake support ship. Keep the
general BYOC SQL availability statement unchanged and preserve the existing
Iceberg query reference for supported providers.


After reading this page, you will be able to:

* [ ] {learning-objective-1}
* [ ] {learning-objective-2}
* [ ] {learning-objective-3}

NOTE: Redpanda SQL is currently available only on BYOC clusters running on AWS.
NOTE: Redpanda SQL is available for BYOC clusters on AWS and GCP.

TIP: If you manage your infrastructure with Terraform, see xref:manage:terraform-provider.adoc#enable-redpanda-sql-on-a-byoc-cluster[Enable Redpanda SQL on a BYOC cluster] in the Terraform provider guide.

Expand Down Expand Up @@ -43,7 +45,7 @@ Cloud Console::
+
--
. Log in to https://cloud.redpanda.com[Redpanda Cloud^].
. Start creating a new BYOC cluster on AWS. For details and prerequisites, see xref:get-started:cluster-types/byoc/aws/create-byoc-cluster-aws.adoc[].
. Start creating a new BYOC cluster on AWS or GCP. For details and prerequisites, see xref:get-started:cluster-types/byoc/aws/create-byoc-cluster-aws.adoc[] or xref:get-started:cluster-types/byoc/gcp/create-byoc-cluster-gcp.adoc[].
. In the cluster creation form, locate the *Redpanda SQL engine* card. Toggle the engine on and use the *RPU* slider to set the compute size.
+
For more on RPUs, compute, and cost calculations, see xref:billing:billing.adoc#redpanda-sql-billing-metrics[Redpanda SQL billing metrics].
Expand All @@ -64,7 +66,7 @@ curl -X POST "https://api.redpanda.com/v1/clusters" \
-d '{
"cluster": {
"name": "<cluster-name>",
"cloud_provider": "CLOUD_PROVIDER_AWS",
"cloud_provider": "<cloud-provider>",
"type": "TYPE_BYOC",
"region": "<region>",
"zones": [ <zones> ],
Expand All @@ -79,8 +81,9 @@ curl -X POST "https://api.redpanda.com/v1/clusters" \
}'
----
+
Replace the `rpsql` placeholders with your own values:
Replace the placeholders with your own values:
+
* `<cloud-provider>`: `CLOUD_PROVIDER_AWS` or `CLOUD_PROVIDER_GCP`.
* `<compute-nodes>`: Set the initial number of SQL compute nodes (minimum 1).
* `<sql-az>` (optional): For multi-AZ clusters, specify which of the cluster's availability zones to deploy for the SQL engine. Provide a single zone string from the cluster's `zones` list.
+
Expand All @@ -90,6 +93,9 @@ Replace the `rpsql` placeholders with your own values:

=== On an existing cluster

// TODO: Confirm with SME whether GCP BYOC clusters created before Redpanda SQL launched on GCP
// (M7/CIAINFRA-3420) need an equivalent agent-permission refresh step before enabling SQL, similar
// to the AWS `rpk cloud byoc aws apply` requirement below.
NOTE: If your BYOC cluster on AWS was created before May 28, 2026, rerun `rpk cloud byoc aws apply --redpanda-id=<cluster-id>` to update the agent's IAM permissions before enabling Redpanda SQL.

[tabs]
Expand Down
2 changes: 1 addition & 1 deletion modules/sql/pages/get-started/sql-quickstart.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ After reading this page, you will be able to:

== Prerequisites

* A Redpanda BYOC cluster on AWS with Redpanda SQL enabled. See xref:sql:get-started/deploy-sql-cluster.adoc[].
* A Redpanda BYOC cluster on AWS or GCP with Redpanda SQL enabled. See xref:sql:get-started/deploy-sql-cluster.adoc[].
* Admin access to your cluster in the Redpanda Cloud Console, or a role with the *SQL: Manage* permission. You need one of these to view SQL connection details and to create catalogs, tables, and grants in the SQL engine. For more information on authorization in Redpanda Cloud, see xref:security:authorization/rbac/index.adoc[].
* A Redpanda topic with a schema registered in Schema Registry. If you don't have one, follow the optional <<optional-produce-sample-data,Produce sample data>> section below to create a sample `orders` topic.
* xref:manage:rpk/rpk-install.adoc[`rpk` v26.1.6] or later installed on your local machine to generate an authentication token.
Expand Down
2 changes: 2 additions & 0 deletions modules/sql/pages/query-data/query-iceberg-topics.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ After reading this page, you will be able to:

== Prerequisites

// TODO: Update prerequisites to include GCP once RPSQL Deployment with Iceberg on GCP
// (CIAINFRA-3422/M9) and BigLake REST catalog support (OXLA-9522, cloudv2 PR #27525) ship.
* A Redpanda BYOC cluster on AWS with Redpanda SQL enabled. See xref:sql:get-started/deploy-sql-cluster.adoc[Enable Redpanda SQL].
* The cluster's xref:reference:properties/cluster-properties.adoc#iceberg_catalog_type[`iceberg_catalog_type`] property is set to `rest`. The `object_storage` catalog type does not support Iceberg queries from Redpanda SQL.
* The cluster's Iceberg REST catalog is configured. See xref:manage:iceberg/rest-catalog/index.adoc[Integrate with REST Catalogs] for the supported REST catalog options (AWS Glue, Snowflake, Databricks Unity, Polaris, and others) and their configuration steps.
Expand Down
Loading