Allow configuring cascade deletion in DABs

NEXT_CHANGELOG.md

@@ -7,6 +7,7 @@
 ### CLI
 
 ### Bundles
+* Added a `cascade_on_destroy` field to the pipeline resource to control whether destroying a pipeline also deletes its datasets (MVs, STs, Views). When unset, the server default applies; set `cascade_on_destroy: false` to retain the datasets on destroy. Supported with the direct deployment engine.
 
 ### Dependency updates
 

acceptance/bundle/config-remote-sync/cli_defaults/output.txt

@@ -57,7 +57,7 @@ The following resources will be deleted:
   delete resources.pipelines.pipeline1
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.pipeline1
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default

acceptance/bundle/config-remote-sync/job_pipeline_task/output.txt

@@ -41,7 +41,7 @@ The following resources will be deleted:
   delete resources.pipelines.my_pipeline
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.my_pipeline
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default

acceptance/bundle/config-remote-sync/pipeline_fields/output.txt

@@ -60,7 +60,7 @@ The following resources will be deleted:
   delete resources.pipelines.my_pipeline
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.my_pipeline
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default

acceptance/bundle/config-remote-sync/resolve_variables/output.txt

@@ -143,7 +143,7 @@ The following resources will be deleted:
   delete resources.pipelines.my_pipeline
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.my_pipeline
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default

acceptance/bundle/config-remote-sync/validation_errors/output.txt

@@ -51,7 +51,7 @@ The following resources will be deleted:
   delete resources.pipelines.my_pipeline
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.my_pipeline
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default

acceptance/bundle/deploy/pipeline-config-dots/output.txt

@@ -23,7 +23,7 @@ This action will result in the deletion of the following UC schemas. Any underly
   delete resources.schemas.my_schema
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.my_pipeline
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default

acceptance/bundle/deploy/snapshot-comparison/output.txt

@@ -43,7 +43,7 @@ The following resources will be deleted:
   delete resources.pipelines.test_pipeline
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.test_pipeline
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/snapshot-test-1-[UNIQUE_NAME]/default
@@ -58,7 +58,7 @@ The following resources will be deleted:
   delete resources.pipelines.test_pipeline
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.test_pipeline
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/snapshot-test-2-[UNIQUE_NAME]/default

acceptance/bundle/destroy/all-resources/output.txt

@@ -14,7 +14,7 @@ This action will result in the deletion of the following UC schemas. Any underly
   delete resources.schemas.my_schema
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.my_pipeline
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default

acceptance/bundle/destroy/jobs-and-pipeline/output.txt

@@ -53,7 +53,7 @@ The following resources will be deleted:
   delete resources.pipelines.bar
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.bar
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]

acceptance/bundle/migrate/runas/output.txt

@@ -138,7 +138,7 @@ The following resources will be deleted:
   delete resources.pipelines.foo
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.foo
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dabs_revenue-[UNIQUE_NAME]/production

acceptance/bundle/refschema/out.fields.txt

@@ -2273,6 +2273,7 @@ resources.models.*.permissions[*].service_principal_name	string	ALL
 resources.models.*.permissions[*].user_name	string	ALL
 resources.pipelines.*.allow_duplicate_names	bool	ALL
 resources.pipelines.*.budget_policy_id	string	ALL
+resources.pipelines.*.cascade_on_destroy	*bool	ALL
 resources.pipelines.*.catalog	string	ALL
 resources.pipelines.*.cause	string	REMOTE
 resources.pipelines.*.channel	string	ALL

acceptance/bundle/resource_deps/pipelines_recreate/output.txt

@@ -108,7 +108,7 @@ The following resources will be deleted:
   delete resources.pipelines.foo
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.foo
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default

acceptance/bundle/resource_deps/remote_app_url/output.txt

@@ -90,7 +90,7 @@ The following resources will be deleted:
   delete resources.pipelines.mypipeline
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.mypipeline
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default

acceptance/bundle/resources/permissions/pipelines/current_can_manage/output.txt

@@ -30,7 +30,7 @@ The following resources will be deleted:
   delete resources.pipelines.foo
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.foo
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default

acceptance/bundle/resources/permissions/pipelines/current_is_owner/output.txt

@@ -18,7 +18,7 @@ The following resources will be deleted:
   delete resources.pipelines.foo
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.foo
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default

acceptance/bundle/resources/permissions/pipelines/empty_list/output.txt

@@ -13,7 +13,7 @@ The following resources will be deleted:
   delete resources.pipelines.foo
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.foo
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default

acceptance/bundle/resources/permissions/pipelines/other_can_manage/output.txt

@@ -30,7 +30,7 @@ The following resources will be deleted:
   delete resources.pipelines.foo
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.foo
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default

acceptance/bundle/resources/permissions/pipelines/other_is_owner/output.txt

@@ -22,7 +22,7 @@ The following resources will be deleted:
   delete resources.pipelines.foo
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.foo
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default

acceptance/bundle/resources/permissions/pipelines/update/output.txt

@@ -123,7 +123,7 @@ The following resources will be deleted:
   delete resources.pipelines.foo
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.foo
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/permissions-test/default

acceptance/bundle/resources/pipelines/allow-duplicate-names/output.txt

@@ -47,7 +47,7 @@ The following resources will be deleted:
   delete resources.pipelines.pipeline_one
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.pipeline_one
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/acc-bundle-deploy-pipeline-duplicate-names-[UNIQUE_NAME]/default

acceptance/bundle/resources/pipelines/auto-approve/output.txt

@@ -63,7 +63,7 @@ The following resources will be deleted:
   delete resources.pipelines.bar
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.bar
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]

acceptance/bundle/resources/pipelines/lakeflow-pipeline/output.txt

@@ -43,7 +43,7 @@ The following resources will be deleted:
   delete resources.pipelines.foo
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.foo
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-pipeline-lakeflow-pipeline-[UNIQUE_NAME]/default

acceptance/bundle/resources/pipelines/recreate-keys/change-ingestion-definition/output.txt

@@ -150,7 +150,7 @@ The following resources will be deleted:
   delete resources.pipelines.my
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.my
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/acc-[UNIQUE_NAME]/default

acceptance/bundle/resources/pipelines/recreate-keys/change-storage/output.txt

@@ -134,7 +134,7 @@ The following resources will be deleted:
   delete resources.pipelines.my
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.my
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/acc-[UNIQUE_NAME]/default

acceptance/bundle/resources/pipelines/recreate/output.txt

@@ -65,7 +65,7 @@ This action will result in the deletion of the following UC schemas. Any underly
   delete resources.schemas.bar
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.foo
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]

acceptance/bundle/resources/pipelines/update/output.txt

@@ -72,7 +72,7 @@ The following resources will be deleted:
   delete resources.pipelines.my
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.my
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/acc-[UNIQUE_NAME]/default

acceptance/bundle/resources/schemas/auto-approve/output.txt

@@ -72,7 +72,7 @@ This action will result in the deletion of the following UC schemas. Any underly
   delete resources.schemas.bar
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.foo
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]

acceptance/bundle/summary/modified_status/output.txt

@@ -189,7 +189,7 @@ This action will result in the deletion of the following UC schemas. Any underly
   delete resources.schemas.my_schema
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.my_pipeline
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle/default

acceptance/bundle/templates/default-python/integration_classic/output.txt

@@ -93,7 +93,7 @@ The following resources will be deleted:
   delete resources.pipelines.project_name_[UNIQUE_NAME]_etl
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.project_name_[UNIQUE_NAME]_etl
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/project_name_[UNIQUE_NAME]/dev
@@ -491,7 +491,7 @@ The following resources will be deleted:
   delete resources.pipelines.project_name_[UNIQUE_NAME]_etl
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.project_name_[UNIQUE_NAME]_etl
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/project_name_[UNIQUE_NAME]/prod

acceptance/pipelines/destroy/auto-approve/output.txt

@@ -19,7 +19,7 @@ The following resources will be deleted:
   delete resources.pipelines.my_pipeline
 
 This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the
-Streaming Tables (STs) and Materialized Views (MVs) managed by them:
+Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain its datasets:
   delete resources.pipelines.my_pipeline
 
 All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-pipeline-destroy/default

acceptance/pipelines/destroy/cascade-on-destroy-terraform-error/databricks.yml

@@ -0,0 +1,7 @@
+bundle:
+  name: test-cascade-on-destroy-tf-error
+resources:
+  pipelines:
+    my_pipeline:
+      name: test-pipeline-cascade-tf
+      cascade_on_destroy: false

acceptance/pipelines/destroy/cascade-on-destroy-terraform-error/output.txt

@@ -0,0 +1,8 @@
+
+=== pipelines deploy fails with cascade_on_destroy on terraform engine
+>>> errcode [CLI] pipelines deploy
+Error: cascade_on_destroy is only supported in direct deployment mode
+  in databricks.yml:7:27
+
+
+Exit code: 1

acceptance/pipelines/destroy/cascade-on-destroy-terraform-error/script

@@ -0,0 +1,2 @@
+title "pipelines deploy fails with cascade_on_destroy on terraform engine"
+trace errcode $CLI pipelines deploy

acceptance/pipelines/destroy/cascade-on-destroy-terraform-error/test.toml

@@ -0,0 +1,8 @@
+Local = true
+RecordRequests = false
+
+# Static validation only; no need to run on cloud.
+Cloud = false
+
+# cascade_on_destroy is direct-only; assert the terraform engine rejects it.
+EnvMatrix.DATABRICKS_BUNDLE_ENGINE = ["terraform"]