Repository navigation
iceberg: add lake_iceberg.relocate_table() to rebuild a table at a new location - #718
Draft
sfc-gh-dachristensen wants to merge 4 commits into
Draft
sfc-gh-dachristensen wants to merge 4 commits into
sfc-gh-dachristensen wants to merge 4 commits into
Conversation
…ocation Copies all data files to the new prefix via parallel DuckDB copy_file, rewrites manifests/manifest-lists with updated paths, writes new metadata.json, and updates the catalog pointer. History is preserved across all snapshots. Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
MaxParallelFileUploads is defined in pg_lake_engine.so and not available at pg_lake_iceberg.so load time, causing an undefined symbol error in CI.
sfc-gh-mslot
reviewed
Oct 9, 2026
| * regenerated with rewritten paths in the next pass. | ||
| */ | ||
| static HTAB * | ||
| RelocateDataFiles(IcebergTableMetadata * metadata, |
Collaborator
There was a problem hiding this comment.
what does this do with the position delete files
sfc-gh-mslot
reviewed
Oct 9, 2026
| StringInfoData command; | ||
|
|
||
| initStringInfo(&command); | ||
| appendStringInfo(&command, "SELECT * FROM pg_lake_copy_file(%s,%s);", |
Collaborator
There was a problem hiding this comment.
given the way things are implemented, it might actually be a lot faster to do a CTAS type operation to maximize parallelization and throw away stuff we don't really need
pg_lake_copy_file does not parallelize, though there could be a possible optimization of using the S3 copy future
Copying the existing data and delete files to the new location carried position deletes and the old snapshot history along, and left the foreign table location and pg_lake's file catalog pointing at the old paths. Instead, create a new Iceberg table at the target location and fill it with INSERT ... SELECT from the source, which is what CTAS does internally and applies deletes on read. The column definitions come from the source catalog, since CTAS would drop NOT NULL, defaults and generated expressions. The owner, table and column privileges, comments and serial sequence ownership move to the new table before the source is dropped and the new table takes its name. Tables with properties that DROP TABLE would silently remove (triggers, policies, inheritance, extended statistics, publications, identity columns) are refused. The function moves to pg_lake_table, which owns CTAS and the catalog hooks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: David Christensen <david.christensen@snowflake.com>
sfc-gh-dachristensen
changed the title
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
There is no way to move a pg_lake Iceberg table to a different storage location. Copying the existing files and rewriting metadata paths would carry over position delete files and the full snapshot history, and it would leave the foreign table
locationoption and pg_lake's file catalog pointing at the old paths.Solution
lake_iceberg.relocate_table(table_name regclass, new_location text)rebuilds the table at the new location from its visible rows. It creates a new Iceberg table there and fills it withINSERT ... SELECTfrom the source. This is the same thing CTAS does internally, and reading the source applies its deletes. The new table starts with one snapshot and no delete files. The source is then dropped and the new table takes its name, so the normal create, insert and drop hooks keep the pg_lake catalogs correct.Plain CTAS would lose NOT NULL, defaults and generated expressions, so the column definitions come from the source catalog. Table options, owner, table and column privileges, comments and serial sequence ownership also move to the new table. Tables with triggers, policies, inheritance, extended statistics, publications or identity columns are refused, since
DROP TABLEwould remove those silently. Objects that depend on the table, such as views, make the drop fail, which rolls the whole call back.Only tables in the postgres catalog are supported. The function lives in
pg_lake_table, which owns CTAS and the catalog hooks.Test plan
locationoptionmake check-indent🤖 Generated with Claude Code