Repository navigation
Add Comprehensive Guide to Snowflake Data Quality Monitoring - #3542
sfc-gh-yostrinsky wants to merge 7 commits into
Conversation
Example-led guide: a small shared setup plus self-contained recipes, so a reader can run any single example without working through the others. Part 1 only (overview, setup, profiling, anatomy of a check). All SQL verified against a live account; result tables are real output. Covers two behaviours that are easy to get wrong: time-relative metrics reject TIMESTAMP_NTZ, and UNSET DATA_METRIC_SCHEDULE reverts to the default hourly cadence rather than suspending monitoring. .... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-Authored-By: Cortex Code <noreply@snowflake.com>
Adds the metric catalogue and decision table, twelve failure-mode recipes, custom metrics, schema-wide monitoring, AI-suggested checks, anomaly detection, remediation, notifications, the circuit breaker, layer strategy, quality as code, cost and cleanup. Every SQL block runs against a live account and the result tables are real output. Findings worth noting in the prose: - FRESHNESS can return a negative value when a future-dated row is the newest timestamp, so a freshness check alone can be defeated by one bad timestamp. Paired with the clock-skew check. - SYSTEM$EVALUATE_DATA_QUALITY_EXPECTATIONS does not record its result; the _PERSIST_RESULT variant does. Anything reading monitoring history needs the latter. - The circuit breaker uses an ALERT because the native notification settings have no action hook. Proven end to end: alert TRIGGERED and the downstream task suspended. .... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-Authored-By: Cortex Code <noreply@snowflake.com>
🔗 Staging Preview ReadyQuickstart:
Generated by GitHub Actions |
…pes into themed sections Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
🔗 Staging Preview ReadyQuickstart:
Generated by GitHub Actions |
Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
🔗 Staging Preview ReadyQuickstart:
Generated by GitHub Actions |
…x Code its own block, and move cost to the end Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
🔗 Staging Preview ReadyQuickstart:
Generated by GitHub Actions |
…metric categories, and replace the deprecated model allowlist with model RBAC Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
🔗 Staging Preview ReadyQuickstart:
Generated by GitHub Actions |
…le, drop the Covered in column, and link Checks as code to the DCM pipelines guide Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
🔗 Staging Preview ReadyQuickstart:
Generated by GitHub Actions |
This branch has not been deployed
1 participant
Summary
Comprehensive guide to Snowflake Data Quality Monitoring, built on Data Metric Functions — system and custom DMFs, scheduling, expectations, and alerting on violations.
The guide was deployed and live-validated end to end against a real database before submission, rather than written from documentation.
Why this replaces #3506
Same content, moved from a fork branch to a branch in this repository so the staging preview workflow can run. Fork-based PRs do not get the
AEM staging deployjob, so #3506 could never produce a preview URL for reviewers.This branch is also rebased onto current
master, which fixes a second problem: the original branch predatedscripts/requirements.txt(added 2026-09-04 in "Harden workflows" #3507), sovalidate_and_preparefailed withNo file ... matched to [scripts/requirements.txt or **/pyproject.toml]. Validation passes on this branch.Supersedes #3506.
Verification
validate_and_preparepassing.Note for reviewers
Opening as a draft. There is an outstanding question to the Data Quality team about roadmap direction that may affect one section; I will mark this ready once that is settled. Feedback on the rest is welcome in the meantime.