fix(optimizer)!: annotate type for databricks REGR_AVGY, REGR_COUNT, REGR_INTERCEPT, REGR_R2, REGR_SLOPE by fivetran-amrutabhimsenayachit · Pull Request #7820 · tobymao/sqlglot

fivetran-amrutabhimsenayachit · 2026-07-01T19:17:13Z

Summary

Adds Databricks type inference support for REGR_AVGY (DOUBLE), REGR_COUNT (BIGINT), REGR_INTERCEPT (DOUBLE), REGR_R2 (DOUBLE), and REGR_SLOPE (DOUBLE), plus fixture coverage for all five functions.

Issue: REGR_FUNC(DISTINCT col1, col2) raised a parse error in Databricks because the base parser's DISTINCT handler consumed all comma-separated arguments into a single node, leaving the second required argument missing.

Fix: Added a custom parser method in DatabricksParser that reads only the first argument under DISTINCT, then parses the rest normally.

Tickets

RD-1229638 (REGR_AVGY) — DOUBLE
RD-1229639 (REGR_COUNT) — BIGINT
RD-1229640 (REGR_INTERCEPT) — DOUBLE
RD-1229641 (REGR_R2) — DOUBLE
RD-1229642 (REGR_SLOPE) — DOUBLE

Test plan

python3 -c "import sqlglot; print(repr(sqlglot.parse_one('SELECT REGR_AVGY(DISTINCT tbl.double_col, tbl.double_col) FROM tbl', dialect='databricks').expressions[0]))"


RegrAvgy(
  this=Distinct(
    expressions=[
      Column(
        this=Identifier(this=double_col, quoted=False),
        table=Identifier(this=tbl, quoted=False))]),
  expression=Column(
    this=Identifier(this=double_col, quoted=False),
    table=Identifier(this=tbl, quoted=False))

make style — PASS
make unit — PASS

github-actions · 2026-07-01T19:56:10Z

SQLGlot Integration Test Results

✅ All tests passed

Comparing:

this branch (sqlglot:type-inference-batch-3 @ sqlglot 4cbf184)
baseline (main @ sqlglot e9b0140)

Overall

main: 192416 total, 153530 passed (pass rate: 79.8%)

sqlglot:type-inference-batch-3: 180222 total, 142385 passed (pass rate: 79.0%)

Transitions:
No change

Dialect pair changes: 0 previous results not found, 3 current results not found

✅ All tests passed

geooo109

Left a comment, also we should add roundtrip tests for ALL/DISTINCT if they are missing.

geooo109 · 2026-07-08T10:25:12Z

In the tests, let's also check where the DISTINCT is applied to for the case of REGR_AVGX (check for the ast)

…E] (#7823) `Resolver._get_column_type_from_scope` resolves the element type of an unnested/exploded column by recursing over the sources of each scope. It had no memoization, so when the same CTE is reachable through several paths of the scope graph, its subtree was re-walked once per path. Qualifying a query whose CTEs reconverge on a shared upstream CTE was therefore exponential in the depth of the graph, for a single UNNEST/EXPLODE. On a real query this reached ~3.8M calls (~4s) into the trace. The scope and schema are immutable during qualification, so the type of a column under a given source depends only on `(source, column name)`. Cache the result on that key so each source is walked once, which makes the trace roughly linear in the size of the scope graph. Qualified output is unchanged. Reached through both BigQuery `UNNEST(col)` and Spark/Hive `LATERAL VIEW EXPLODE(col)`. Closes #7821 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…* functions in databricks [CLAUDE]

fivetran-amrutabhimsenayachit force-pushed the type-inference-batch-3 branch from 3e9a43d to 3bd5b3f Compare July 1, 2026 19:20

fivetran-amrutabhimsenayachit self-assigned this Jul 1, 2026

geooo109 self-assigned this Jul 2, 2026

geooo109 reviewed Jul 2, 2026

View reviewed changes

geooo109 changed the title ~~feat(typing): add databricks type inference for REGR_AVGY, REGR_COUNT, REGR_INTERCEPT, REGR_R2, REGR_SLOPE~~ fix(optimizer)!: annotate type for databricks REGR_AVGY, REGR_COUNT, REGR_INTERCEPT, REGR_R2, REGR_SLOPE Jul 2, 2026

fivetran-amrutabhimsenayachit force-pushed the type-inference-batch-3 branch from 780a37e to d73df08 Compare July 2, 2026 17:14

fivetran-amrutabhimsenayachit requested a review from geooo109 July 2, 2026 17:54

geooo109 reviewed Jul 3, 2026

View reviewed changes

Comment thread sqlglot/parsers/databricks.py Outdated

geooo109 reviewed Jul 3, 2026

View reviewed changes

Comment thread sqlglot/parsers/databricks.py Outdated

fivetran-amrutabhimsenayachit force-pushed the type-inference-batch-3 branch 2 times, most recently from 731d1ef to 7a265fb Compare July 6, 2026 15:07

georgesittas requested a review from geooo109 July 6, 2026 16:54

geooo109 reviewed Jul 6, 2026

View reviewed changes

Comment thread sqlglot/parsers/databricks.py Outdated

Comment thread sqlglot/parsers/databricks.py Outdated

Comment thread sqlglot/parsers/databricks.py Outdated

fivetran-amrutabhimsenayachit force-pushed the type-inference-batch-3 branch 2 times, most recently from d0e7e96 to 8df4873 Compare July 7, 2026 17:49

fivetran-amrutabhimsenayachit requested a review from geooo109 July 7, 2026 18:26

geooo109 reviewed Jul 8, 2026

View reviewed changes

Comment thread sqlglot/parsers/databricks.py Outdated

dvryaboy and others added 3 commits July 8, 2026 12:05

feat(typing): add DISTINCT/ALL/OVER fixtures and parser fix for REGR_…

5c05391

…* functions in databricks [CLAUDE]

fix: update aggregate expressions and databricks parser

69d59e1

fivetran-amrutabhimsenayachit force-pushed the type-inference-batch-3 branch 2 times, most recently from 823a114 to ca93ea1 Compare July 8, 2026 16:22

fix: update databricks parser to have a generic function for regr_avgy/x

02cbca0

fivetran-amrutabhimsenayachit force-pushed the type-inference-batch-3 branch from ca93ea1 to 02cbca0 Compare July 8, 2026 17:06

geooo109 approved these changes Jul 8, 2026

View reviewed changes

fivetran-amrutabhimsenayachit merged commit 77c7814 into main Jul 8, 2026
8 checks passed

fivetran-amrutabhimsenayachit deleted the type-inference-batch-3 branch July 8, 2026 17:42

geooo109 mentioned this pull request Jul 10, 2026

feat(optimizer)!: type annotation for databricks REGR_SXX, REGR_SXY, REGR_SYY #7851

Open

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(optimizer)!: annotate type for databricks REGR_AVGY, REGR_COUNT, REGR_INTERCEPT, REGR_R2, REGR_SLOPE#7820

fix(optimizer)!: annotate type for databricks REGR_AVGY, REGR_COUNT, REGR_INTERCEPT, REGR_R2, REGR_SLOPE#7820
fivetran-amrutabhimsenayachit merged 4 commits into
mainfrom
type-inference-batch-3

fivetran-amrutabhimsenayachit commented Jul 1, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 1, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

geooo109 left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

geooo109 commented Jul 8, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

fivetran-amrutabhimsenayachit commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tickets

Test plan

Uh oh!

github-actions Bot commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

SQLGlot Integration Test Results

✅ All tests passed

Overall

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

geooo109 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

geooo109 commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fivetran-amrutabhimsenayachit commented Jul 1, 2026 •

edited

Loading

github-actions Bot commented Jul 1, 2026 •

edited

Loading

geooo109 left a comment •

edited

Loading

geooo109 commented Jul 8, 2026 •

edited

Loading