Skip to content

Shared datasets integration - #202

Draft
jonaraphael wants to merge 77 commits into
mainfrom
feat/shared-datasets
Draft

jonaraphael wants to merge 77 commits into
mainfrom
feat/shared-datasets

Conversation

@jonaraphael

Copy link
Copy Markdown
Collaborator

Draft PR to run CI on feat/shared-datasets.

This branch integrates shared-datasets-1 as the source for AOI datasets (catalog-driven accessors, SDK-based fetches, user-AOI child-table geometry).

Latest commits worth reviewer attention:

  • CI now installs PostgreSQL/PostGIS in the test job (same step as feat/aoi_backfill). Locally the full suite passes under real PostGIS (79 passed / 0 failed); the no-PostGIS compat fallback cannot run the slick_to_aoi tests, since slick's computed columns call ST_* functions.
  • skytruth-shared-datasets pin bumped f9e1c5e (May 3) → c69850b (Aug 6, current main tip). fetch_dataset/DatasetRef surface unchanged; suite green locally against the new pin.
  • Spatial-compat model table gains the zoom_level/scale columns it was missing.

🤖 Generated with Claude Code

ancientpi3 and others added 30 commits April 10, 2026 13:48
GOOGLE_APPLICATION_CREDENTIALS not needed
# Conflicts:
#	.claude/skills/README.md
#	.claude/skills/cerulean-cloud-test-env/SKILL.md
#	.claude/skills/cerulean-cloud-test-env/agents/openai.yaml
#	.claude/skills/cerulean-ui-frontend-sql/SKILL.md
#	.claude/skills/cerulean-ui-frontend-sql/references/frontend-sql-inventory.md
#	.claude/skills/skytruth-intent-engineering/SKILL.md
#	AGENTS.md
jonaraphael and others added 28 commits May 3, 2026 01:48
# Conflicts:
#	stack/cloud_run_tipg.py
# Conflicts:
#	cerulean_cloud/database_schema.py
The no-PostGIS fallback schema in conftest was missing the model
table's computed zoom_level/scale columns. Also gitignore the
repo-local .conda/ test env.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
f9e1c5e (May 3) -> c69850b (Aug 6, current main tip; SDK last
changed at 64cc7f9). fetch_dataset/DatasetRef surface unchanged;
full test suite passes locally against the new pin.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same step the backfill branch runs. Locally the full suite passes
under real PostGIS (79 passed / 0 failed), while the no-PostGIS
compat fallback cannot run the slick_to_aoi tests (computed
columns call ST_* functions), so real PostGIS in CI is required
for a green, meaningful run on this branch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

🍹 preview on cerulean-cloud-images/test

Pulumi report
   Previewing update (test):
@ previewing update........

   pulumi:pulumi:Stack cerulean-cloud-images-test  /opt/hostedtoolcache/Python/3.11.16/x64/lib/python3.11/site-packages/google/auth/transport/grpc.py:44: FutureWarning: grpcio < 1.83.0 does not support Post-Quantum Cryptography (PQC). Support for non-PQC environments is deprecated. In October 2026, google-auth will raise its minimum requirements to enforce grpcio >= 1.83.0. For more details on Google Cloud's post-quantum security migration, visit: https://cloud.google.com/security/resources/post-quantum-cryptography
   pulumi:pulumi:Stack cerulean-cloud-images-test    warnings.warn(
@ previewing update...........
   pulumi:pulumi:Stack cerulean-cloud-images-test running   warnings.warn(
@ previewing update.....
   gcp:artifactregistry:Repository cerulean-cloud-images-test-registry  
@ previewing update....
   docker:index:Image cerulean-cloud-images-test-cr-orchestrator-image  Building your image for linux/amd64 architecture.
~  docker:index:Image cerulean-cloud-images-test-cr-orchestrator-image update [diff: ~build]; Building your image for linux/amd64 architecture.
   docker:index:Image cerulean-cloud-images-test-cr-infer-image  Building your image for linux/amd64 architecture.
~  docker:index:Image cerulean-cloud-images-test-cr-infer-image update [diff: ~build]; Building your image for linux/amd64 architecture.
   docker:index:Image cerulean-cloud-images-test-cr-tipg-image  Building your image for linux/amd64 architecture.
~  docker:index:Image cerulean-cloud-images-test-cr-tipg-image update [diff: ~build]; Building your image for linux/amd64 architecture.
-  docker:index:Image cerulean-cloud-images-test-cr-aoi-backfill-image delete 
   pulumi:pulumi:Stack cerulean-cloud-images-test  2 messages
Diagnostics:
 docker:index:Image (cerulean-cloud-images-test-cr-infer-image):
   Building your image for linux/amd64 architecture.
   To ensure you are building for the correct platform, consider explicitly setting the `platform` field on ImageBuildOptions.

 docker:index:Image (cerulean-cloud-images-test-cr-tipg-image):
   Building your image for linux/amd64 architecture.
   To ensure you are building for the correct platform, consider explicitly setting the `platform` field on ImageBuildOptions.

 pulumi:pulumi:Stack (cerulean-cloud-images-test):
   /opt/hostedtoolcache/Python/3.11.16/x64/lib/python3.11/site-packages/google/auth/transport/grpc.py:44: FutureWarning: grpcio < 1.83.0 does not support Post-Quantum Cryptography (PQC). Support for non-PQC environments is deprecated. In October 2026, google-auth will raise its minimum requirements to enforce grpcio >= 1.83.0. For more details on Google Cloud's post-quantum security migration, visit: https://cloud.google.com/security/resources/post-quantum-cryptography
     warnings.warn(

 docker:index:Image (cerulean-cloud-images-test-cr-orchestrator-image):
   Building your image for linux/amd64 architecture.
   To ensure you are building for the correct platform, consider explicitly setting the `platform` field on ImageBuildOptions.

Resources:
   ~ 3 to update
   - 1 to delete
   4 changes. 2 unchanged

   

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

🍹 preview on cerulean-cloud/test

Pulumi report
   Previewing update (test):

   pulumi:pulumi:Stack cerulean-cloud-test running 
   pulumi:providers:docker cerulean-cloud-images-test-ar  
@ previewing update.......
   pulumi:pulumi:Stack cerulean-cloud-test running /opt/hostedtoolcache/Python/3.11.16/x64/lib/python3.11/site-packages/google/auth/transport/grpc.py:44: FutureWarning: grpcio < 1.83.0 does not support Post-Quantum Cryptography (PQC). Support for non-PQC environments is deprecated. In October 2026, google-auth will raise its minimum requirements to enforce grpcio >= 1.83.0. For more details on Google Cloud's post-quantum security migration, visit: https://cloud.google.com/security/resources/post-quantum-cryptography
   pulumi:pulumi:Stack cerulean-cloud-test running   warnings.warn(
@ previewing update.....
   docker:index:RemoteImage cerulean-cloud-images-test-remote-offset  
   docker:index:RemoteImage cerulean-cloud-images-test-remote-orchestrator  
   docker:index:RemoteImage cerulean-cloud-images-test-remote-tipg  
@ previewing update.....
   gcp:serviceaccount:Account cerulean-cloud-test-cr-infer  
@ previewing update.....
   gcp:sql:DatabaseInstance cerulean-cloud-test-database-instance  warning: Type checking failed: 
   gcp:sql:DatabaseInstance cerulean-cloud-test-database-instance  warning: Unexpected type at field "deletionProtection": 
   gcp:sql:DatabaseInstance cerulean-cloud-test-database-instance  warning: Type checking is still experimental. If you believe that a warning is incorrect,
   gcp:sql:DatabaseInstance cerulean-cloud-test-database-instance  warning: Type checking is still experimental. If you believe that a warning is incorrect,
   gcp:storage:Bucket cerulean-cloud-test-cf-alerts-bucket  
   gcp:vpcaccess:Connector test-vpca  
   gcp:cloudtasks:Queue cerulean-cloud-test-queue-cloud-tasks-asa-analysis  
   gcp:serviceaccount:Account cerulean-cloud-test-cf-alerts-service-account  
   gcp:serviceaccount:Account cerulean-cloud-test-cr-tipg  
   gcp:storage:Bucket cerulean-cloud-test-bucket-cf-asa  
   gcp:projects:IAMMember cerulean-cloud-test-cr-infer-secretmanagerSecretAccessor  
   gcp:projects:IAMMember cerulean-cloud-test-cr-infer-cloudSqlClient  
   gcp:serviceaccount:Account cerulean-cloud-test-cf-asa  
   gcp:secretmanager:SecretIamMember cerulean-cloud-test-cr-infer-secret-accessor-binding  
   gcp:sql:User cerulean-cloud-test-database-users  
-- gcp:storage:BucketObject cerulean-cloud-test-cf-alerts-source delete original 
+- gcp:storage:BucketObject cerulean-cloud-test-cf-alerts-source replace [diff: ~detectMd5hash,name,source]
++ gcp:storage:BucketObject cerulean-cloud-test-cf-alerts-source create replacement [diff: ~detectMd5hash,name,source]
   gcp:projects:IAMMember cerulean-cloud-test-cr-tipg-cloudSqlClient  
   gcp:projects:IAMMember cerulean-cloud-test-cr-tipg-secretmanagerSecretAccessor  
   gcp:secretmanager:SecretIamMember cerulean-cloud-test-cr-tipg-secret-accessor-binding  
-- gcp:storage:BucketObject cerulean-cloud-test-source-cf-asa delete original 
+- gcp:storage:BucketObject cerulean-cloud-test-source-cf-asa replace [diff: ~detectMd5hash,name,source]
++ gcp:storage:BucketObject cerulean-cloud-test-source-cf-asa create replacement [diff: ~detectMd5hash,name,source]
   gcp:projects:IAMMember cerulean-cloud-test-cf-asa-iam  
~  gcp:cloudrun:Service cerulean-cloud-test-cr-infer update [diff: -metadata~template]
   gcp:sql:Database cerulean-cloud-test-database  
~  gcp:cloudrun:Service cerulean-cloud-test-cr-tipg update [diff: -metadata~template]
~  gcp:cloudfunctionsv2:Function cerulean-cloud-test-cf-asa update [diff: ~buildConfig,serviceConfig]
   gcp:cloudrun:IamPolicy cerulean-cloud-test-cr-noauth-iam-policy-offset  
   gcp:cloudfunctionsv2:FunctionIamMember cerulean-cloud-test-cf-asa-invoker  
   gcp:cloudrun:IamMember cf-asa-run-invoker  
~  gcp:cloudfunctionsv2:Function cerulean-cloud-test-cf-alerts update [diff: ~buildConfig]
   gcp:cloudrun:IamPolicy cerulean-cloud-test-cr-noauth-iam-policy-tipg  
   gcp:cloudscheduler:Job cerulean-cloud-test-cf-alerts-scheduler  
   gcp:cloudfunctionsv2:FunctionIamMember cerulean-cloud-test-cf-alerts-invoker  
   gcp:cloudrun:IamMember cerulean-cloud-test-cf-alerts-cr-invoker  
@ previewing update......
   aws:iam:Role cerulean-cloud-test-lambda-titiler-role  
   aws:s3:Bucket cerulean-cloud-test-titiler-lambda-archive  
   aws:iam:Policy cerulean-cloud-test-lambda-titiler-policy  
   aws:apigatewayv2:Api cerulean-cloud-test-lambda-titiler-api  
   gcp:storage:Bucket cerulean-cloud-test-bucket-cf-sr  
   aws:sns:Topic cerulean-cloud-test-lambda-APIAbuseAlert  
   gcp:cloudtasks:Queue cerulean-cloud-test-queue-cr-orchestrator  
   gcp:serviceaccount:Account cerulean-cloud-test-cr-orchestrator  
   gcp:serviceaccount:Account cerulean-cloud-test-cf-sr  
   aws:iam:Role cerulean-cloud-test-lambda-sentinel1-iam  
   aws:iam:RolePolicyAttachment cerulean-cloud-test-lambda-titiler-attachment2  
   aws:iam:RolePolicyAttachment cerulean-cloud-test-lambda-titiler-attachment  
-- gcp:storage:BucketObject cerulean-cloud-test-source-cf-sr delete original 
+- gcp:storage:BucketObject cerulean-cloud-test-source-cf-sr replace [diff: ~detectMd5hash,name,source]
++ gcp:storage:BucketObject cerulean-cloud-test-source-cf-sr create replacement [diff: ~detectMd5hash,name,source]
-- gcp:storage:BucketObject cerulean-cloud-test-source-cf-historical-run delete original 
+- gcp:storage:BucketObject cerulean-cloud-test-source-cf-historical-run replace [diff: ~detectMd5hash,name,source]
++ gcp:storage:BucketObject cerulean-cloud-test-source-cf-historical-run create replacement [diff: ~detectMd5hash,name,source]
   aws:sns:TopicSubscription cerulean-cloud-test-lambda-titiler-email-tech  
   aws:sns:TopicSubscription cerulean-cloud-test-lambda-titiler-email-jona  
   aws:sns:TopicSubscription cerulean-cloud-test-lambda-titiler-email-aemon  
   aws:sns:TopicSubscription cerulean-cloud-test-lambda-titiler-email-jason  
   gcp:projects:IAMMember cerulean-cloud-test-cr-orchestrator-cloudTasksEnqueuer  
   gcp:projects:IAMMember cerulean-cloud-test-cr-orchestrator-cloudSqlClient  
   gcp:projects:IAMMember cerulean-cloud-test-cr-orchestrator-secretmanagerSecretAccessor  
   gcp:projects:IAMMember cerulean-cloud-test-cf-sr-iam  
   gcp:secretmanager:SecretIamMember cerulean-cloud-test-cr-orchestrator-secret-accessor-binding  
   aws:iam:RolePolicyAttachment cerulean-cloud-test-lambda-sentinel1-basic-execution  
~  gcp:cloudrun:Service cerulean-cloud-test-cr-orchestrator update [diff: -metadata~template]
   gcp:cloudrun:IamPolicy cerulean-cloud-test-cr-noauth-iam-policy-orchestrator  
~  gcp:cloudfunctionsv2:Function cerulean-cloud-test-cf-sr update [diff: ~buildConfig,serviceConfig]
~  gcp:cloudfunctionsv2:Function cerulean-cloud-test-cf-historical-run update [diff: ~buildConfig,serviceConfig]
   aws:lambda:Function cerulean-cloud-test-lambda-sentinel1-sub  
   gcp:cloudfunctionsv2:FunctionIamMember cerulean-cloud-test-cf-sr-invoker  
   gcp:cloudrun:IamMember cf-sr-run-invoker  
   gcp:cloudrun:IamMember cf-historical-run-run-invoker  
   gcp:cloudfunctionsv2:FunctionIamMember cerulean-cloud-test-cf-historical-run-invoker  
   aws:sns:TopicSubscription cerulean-cloud-test-sentinel1-subscription  
   aws:lambda:Permission cerulean-cloud-test-lambda-sentinel1-permission  
@ previewing update......................................................................................................
~  aws:s3:BucketObject cerulean-cloud-test-titiler-lambda-archive update [diff: ~source]
~  aws:lambda:Function cerulean-cloud-test-lambda-titiler-sentinel update [diff: ~sourceCodeHash]
   aws:apigatewayv2:Integration cerulean-cloud-test-lambda-titiler-integration  
   aws:lambda:Permission cerulean-cloud-test-lambda-titiler-permission  
   aws:cloudwatch:MetricAlarm cerulean-cloud-test-lambda-titiler-alarm  
   aws:apigatewayv2:Route cerulean-cloud-test-lambda-titiler-route  
   aws:apigatewayv2:Stage cerulean-cloud-test-lambda-titiler-stage  
   pulumi:pulumi:Stack cerulean-cloud-test running Creating lambda package in [/home/runner/work/cerulean-cloud/cerulean-cloud] [running in Docker]...
   pulumi:pulumi:Stack cerulean-cloud-test running Checking Docker is available...
   pulumi:pulumi:Stack cerulean-cloud-test running Building container image...
   pulumi:pulumi:Stack cerulean-cloud-test running Sucessfully built container image with id sha256:f2cd55d4c0a2ffeeaacff70dd764d1a82dafd7776cb7b355c1ae819f68427393
   pulumi:pulumi:Stack cerulean-cloud-test running Creating installation package.zip ...
   pulumi:pulumi:Stack cerulean-cloud-test running Sucessfully created package.zip at /home/runner/work/cerulean-cloud/cerulean-cloud/package.zip
-  gcp:cloudrun:IamPolicy cerulean-cloud-test-cr-noauth-iam-policy-aoi-backfill delete 
-  gcp:cloudrun:Service cerulean-cloud-test-cr-aoi-backfill delete 
-  gcp:projects:IAMMember cerulean-cloud-test-cr-aoi-backfill-cloudSqlClient delete 
-  gcp:projects:IAMMember cerulean-cloud-test-cr-aoi-backfill-cloudTasksEnqueuer delete 
-  gcp:projects:IAMMember cerulean-cloud-test-cr-aoi-backfill-secretmanagerSecretAccessor delete 
-  gcp:secretmanager:SecretIamMember cerulean-cloud-test-cr-aoi-backfill-secret-accessor-binding delete 
-  gcp:projects:IAMMember cerulean-cloud-test-cr-aoi-backfill-storageObjectViewer delete 
-  gcp:serviceaccount:Account cerulean-cloud-test-cr-aoi-backfill delete 
-  docker:index:RemoteImage cerulean-cloud-images-test-remote-aoi-backfill delete 
-  gcp:cloudtasks:Queue cerulean-cloud-test-queue-cr-aoi-backfill delete 
   pulumi:pulumi:Stack cerulean-cloud-test  8 messages
Diagnostics:
 pulumi:pulumi:Stack (cerulean-cloud-test):
   /opt/hostedtoolcache/Python/3.11.16/x64/lib/python3.11/site-packages/google/auth/transport/grpc.py:44: FutureWarning: grpcio < 1.83.0 does not support Post-Quantum Cryptography (PQC). Support for non-PQC environments is deprecated. In October 2026, google-auth will raise its minimum requirements to enforce grpcio >= 1.83.0. For more details on Google Cloud's post-quantum security migration, visit: https://cloud.google.com/security/resources/post-quantum-cryptography
     warnings.warn(

   Creating lambda package in [/home/runner/work/cerulean-cloud/cerulean-cloud] [running in Docker]...
   Checking Docker is available...
   Building container image...
   Sucessfully built container image with id sha256:f2cd55d4c0a2ffeeaacff70dd764d1a82dafd7776cb7b355c1ae819f68427393
   Creating installation package.zip ...
   Sucessfully created package.zip at /home/runner/work/cerulean-cloud/cerulean-cloud/package.zip

 gcp:sql:DatabaseInstance (cerulean-cloud-test-database-instance):
   warning: Type checking failed: 
   warning: Unexpected type at field "deletionProtection":
              expected boolean type, got string type
   warning: Type checking is still experimental. If you believe that a warning is incorrect,
   please let us know by creating an issue at https://github.com/pulumi/pulumi-terraform-bridge/issues.
   This will become a hard error in the future.

Outputs:
 - cloud_run_aoi_backfill_url: "https://cerulean-cloud-test-cr-aoi-backfill-5qkjkyomta-ew.a.run.app"

Resources:
   ~ 9 to update
   - 10 to delete
   +-4 to replace
   23 changes. 65 unchanged

   

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants