fix(optimizer)!: normalization of synthesized output aliases by fivetran-kwoodbeck · Pull Request #7816 · tobymao/sqlglot

fivetran-kwoodbeck · 2026-06-29T18:31:00Z

When the optimizer generates an output alias for an unaliased projection (in qualify_outputs), the synthesized alias does not conform to the dialect's NORMALIZATION_STRATEGY

The patch calls normalize_identifier to ensure the alias follow the dialect's casing rules (e.g. upper-folding in Snowflake). An update to star expansion (_expand_stars) was needed so that columns expanded from quoted source projections carry the quoted flag, keeping star output consistent with an explicit reference to the same column.

Quoted columns keep their exact spelling, since folding them would change which column they resolve to.

Example: Snowflake (upper-folding)

SELECT OBJECT_CONSTRUCT(*) FROM (SELECT a, b FROM x) AS t;

Before

SELECT OBJECT_CONSTRUCT(*) AS _col_0 FROM (SELECT a AS a, b AS b FROM x AS x) AS t;

After

SELECT OBJECT_CONSTRUCT(*) AS _COL_0 FROM (SELECT a AS A, b AS B FROM x AS x) AS t;

georgesittas · 2026-06-30T12:55:00Z

@fivetran-kwoodbeck can you remind me what the motivation behind this was? Was this a bug, like changing the semantics of the output compared to the input? Or is it just an aesthetic/consistency improvement?

fivetran-kwoodbeck · 2026-06-30T14:15:51Z

@fivetran-kwoodbeck can you remind me what the motivation behind this was? Was this a bug, like changing the semantics of the output compared to the input? Or is it just an aesthetic/consistency improvement?

@georgesittas The motivation is to ensure the optimizer both idempotent and to make sure the generated aliases are consistent with each dialect's NORMALIZATION_STRATEGY. If idempotency is a requirement for the optimizer, then yes, it's a bug.

fivetran-kwoodbeck · 2026-06-30T18:01:50Z

        # Databricks
        sql = _parse_and_optimize("SELECT col:A.a, col:a.A FROM t", dialect="databricks")
-        assert sql == "SELECT `t`.`col`:A.a AS `a`, `t`.`col`:a.A AS `A` FROM `t` AS `t`"
+        assert sql == "SELECT `t`.`col`:A.a AS `a`, `t`.`col`:a.A AS `a` FROM `t` AS `t`"


Just flagging this change, I'm not sure how Databricks would behave given this query (no access to Databricks to test). The Snowflake query below had an incorrect assert.

Yeah, this is a regression:

WITH t AS ( SELECT PARSE_JSON('{"a": {"a": 1, "A": 2}, "A": {"a": 3, "A": 4}}') AS col ) -- SELECT col:a.a, col:a.A, col:A.a, col:A.A FROM t -- a:1, A:2, a:3, A:4 -- SELECT col:A.a, col:a.A FROM t -- a:3, A:2 -- SELECT `t`.`col`:A.a AS `a`, `t`.`col`:a.A AS `A` FROM `t` AS `t` -- a:3 A:2 SELECT `t`.`col`:A.a AS `a`, `t`.`col`:a.A AS `a` FROM `t` AS `t` -- a:3 a:2

Actually, @fivetran-kwoodbeck, unresolving this again because I realized that the resulting projections are ambiguous after the fact:

WITH t1 AS ( SELECT PARSE_JSON('{"a": {"a": 1, "A": 2}, "A": {"a": 3, "A": 4}}') AS col ), t2 AS ( SELECT col:a.a, col:a.A, col:A.a, col:A.A FROM t1 -- a:1, A:2, a:3, A:4 -- SELECT col:A.a, col:a.A FROM t1 -- a:3, A:2 -- SELECT `t`.`col`:A.a AS `a`, `t`.`col`:a.A AS `A` FROM `t1` AS `t` -- a:3 A:2 -- SELECT `t`.`col`:A.a AS `a`, `t`.`col`:a.A AS `a` FROM `t1` AS `t` -- a:3 a:2 ) SELECT a FROM t2 -- All cases for t2 result in an ambiugous reference error

So I'm not sure this is a problem, after all.

georgesittas · 2026-07-01T12:10:34Z

+            quoted_columns = (
+                {s.output_name: _output_identifier_quoted(s) for s in source_expression.selects}
+                if isinstance(source_expression, exp.Query)
+                else {}
+            )


Nit: I'd make this a set.

Suggested change

quoted_columns = (

{s.output_name: _output_identifier_quoted(s) for s in source_expression.selects}

if isinstance(source_expression, exp.Query)

else {}

)

quoted_columns = {

s.output_name

for s in source_expression.selects

if isinstance(source_expression, exp.Query) and _output_identifier_quoted(s)

}

By the way, is this costly to construct within the for table in tables loop instead of once? Doesn't this repeat work? Is there a better way to implement this?

I'm not sure there's a better way, for example, I don't see if (or how) to reliably access this in the for name in columns: loop.

Yeah feel free to ignore my last comment above. It's fine.

That snippet crashes the test suite. The if isinstance(source_expression, exp.Query) needs to run outside of the loop, not inside.

Yeah, I missed that. You can still rewrite it via a ternary operator similarly.

georgesittas · 2026-07-01T12:46:45Z

        # Databricks
        sql = _parse_and_optimize("SELECT col:A.a, col:a.A FROM t", dialect="databricks")
-        assert sql == "SELECT `t`.`col`:A.a AS `a`, `t`.`col`:a.A AS `A` FROM `t` AS `t`"
+        assert sql == "SELECT `t`.`col`:A.a AS `a`, `t`.`col`:a.A AS `a` FROM `t` AS `t`"


Yeah, this is a regression:

WITH t AS ( SELECT PARSE_JSON('{"a": {"a": 1, "A": 2}, "A": {"a": 3, "A": 4}}') AS col ) -- SELECT col:a.a, col:a.A, col:A.a, col:A.A FROM t -- a:1, A:2, a:3, A:4 -- SELECT col:A.a, col:a.A FROM t -- a:3, A:2 -- SELECT `t`.`col`:A.a AS `a`, `t`.`col`:a.A AS `A` FROM `t` AS `t` -- a:3 A:2 SELECT `t`.`col`:A.a AS `a`, `t`.`col`:a.A AS `a` FROM `t` AS `t` -- a:3 a:2

…schema

georgesittas · 2026-07-06T10:19:08Z

-                    "id": "text",
+                    '"ID"': "text",


Why did this change?

To make it consistent. If you look at the query below, it refers to "ID" but the schema specifies id (as well as "name" and "payload").

The id form was intentional, this test is covering schema normalization as well.

georgesittas · 2026-07-06T10:28:56Z

        # Databricks
        sql = _parse_and_optimize("SELECT col:A.a, col:a.A FROM t", dialect="databricks")
-        assert sql == "SELECT `t`.`col`:A.a AS `a`, `t`.`col`:a.A AS `A` FROM `t` AS `t`"
+        assert sql == "SELECT `t`.`col`:A.a AS `a`, `t`.`col`:a.A AS `a` FROM `t` AS `t`"


Actually, @fivetran-kwoodbeck, unresolving this again because I realized that the resulting projections are ambiguous after the fact:

WITH t1 AS ( SELECT PARSE_JSON('{"a": {"a": 1, "A": 2}, "A": {"a": 3, "A": 4}}') AS col ), t2 AS ( SELECT col:a.a, col:a.A, col:A.a, col:A.A FROM t1 -- a:1, A:2, a:3, A:4 -- SELECT col:A.a, col:a.A FROM t1 -- a:3, A:2 -- SELECT `t`.`col`:A.a AS `a`, `t`.`col`:a.A AS `A` FROM `t1` AS `t` -- a:3 A:2 -- SELECT `t`.`col`:A.a AS `a`, `t`.`col`:a.A AS `a` FROM `t1` AS `t` -- a:3 a:2 ) SELECT a FROM t2 -- All cases for t2 result in an ambiugous reference error

So I'm not sure this is a problem, after all.

georgesittas · 2026-07-06T10:29:55Z

+                        # if it has characters that the dialect would have changed, infer that it was quoted.
+                        isinstance(source, exp.Table) and dialect.case_sensitive(name)


Can you explain a bit more why we need case_sensitive here? I wasn't expecting such a "low-level" check to surface in this code.

This is to handle the case when we're working with a schema, which does not preserve quotations (as the column names are normalized). So we use dialect.case_sensitive(name) as a heuristic: if a column name contains characters the dialect would normalize away, then the column name must have been quoted originally.

The other alternative would be to preserve quotation information in the schema. Would that be better?

georgesittas · 2026-07-06T10:33:53Z

+                    )
+                    if (
+                        quoted
+                        and isinstance(selection_expr, exp.Column)


I suppose that you can either have a Column or an Alias here. Is the assumption that the alias is already properly quoted and that's why we only handle Column?

That was the original reason, yes. But the last commit actually made this block redundant (it's all correctly handled with the quoted variable), so I'll remove this block.

georgesittas · 2026-07-06T10:36:37Z

+            if dialect and not (dialect.PRESERVE_ORIGINAL_OUTPUT_NAME_CASE):
+                dialect.normalize_identifier(selection.args["alias"])


Doesn't this kind of defeat the purpose? I don't think we should have to gate normalization behind a flag. I think given my latest reply re: databricks' ambiguity errors we should be able to simplify it once again.

fivetran-kwoodbeck commented Jun 29, 2026

View reviewed changes

Comment thread sqlglot/optimizer/qualify_columns.py

georgesittas reviewed Jun 30, 2026

View reviewed changes

Comment thread sqlglot/optimizer/qualify_columns.py Outdated

georgesittas reviewed Jun 30, 2026

View reviewed changes

Comment thread sqlglot/optimizer/qualify_columns.py Outdated

georgesittas reviewed Jun 30, 2026

View reviewed changes

Comment thread sqlglot/optimizer/qualify_columns.py Outdated

Comment thread sqlglot/optimizer/qualify_columns.py Outdated

Comment thread sqlglot/optimizer/qualify_columns.py Outdated

georgesittas changed the title ~~fix(optimizer): normalization of synthesized output aliases~~ fix(optimizer)!: normalization of synthesized output aliases Jun 30, 2026

fivetran-kwoodbeck changed the title ~~fix(optimizer)!: normalization of synthesized output aliases~~ fix(optimizer): normalization of synthesized output aliases Jun 30, 2026

fivetran-kwoodbeck force-pushed the optimizer/bug-normalizealiases branch from 99348c9 to 9654789 Compare June 30, 2026 16:38

fivetran-kwoodbeck commented Jun 30, 2026

View reviewed changes

Comment thread tests/test_optimizer.py

fivetran-kwoodbeck commented Jun 30, 2026

View reviewed changes

Repository owner deleted a comment from github-actions Bot Jun 30, 2026

This comment was marked as outdated.

Sign in to view

georgesittas requested changes Jul 1, 2026

View reviewed changes

georgesittas requested review from Copilot and tobymao July 1, 2026 12:50

Copilot started reviewing on behalf of georgesittas July 1, 2026 12:50 View session

This comment was marked as duplicate.

Sign in to view

georgesittas reviewed Jul 1, 2026

View reviewed changes

Comment thread sqlglot/optimizer/qualify_columns.py Outdated

georgesittas changed the title ~~fix(optimizer): normalization of synthesized output aliases~~ fix(optimizer)!: normalization of synthesized output aliases Jul 1, 2026

fivetran-kwoodbeck added 8 commits July 3, 2026 11:04

Normalization of synthesized output aliases

96a6b13

PR tweaks / updates

314b105

tweak to deal with dialects that preserve case

762eb85

adjusted comment

b854e22

updated tests

a76206e

updated tests

44caa96

fixed test schema, made dialect non optional, and infers quotes from …

cf071d4

…schema

updated tests

eb6c093

fivetran-kwoodbeck force-pushed the optimizer/bug-normalizealiases branch from 478351a to eb6c093 Compare July 3, 2026 15:15

minor tweak

4bccdf6

fivetran-kwoodbeck added 4 commits July 3, 2026 12:53

fixed tests

13a847c

updated tests

e4f77f8

fixed other branch

1c765e1

formatting

3003df7

georgesittas reviewed Jul 6, 2026

View reviewed changes

		# if it has characters that the dialect would have changed, infer that it was quoted.
		isinstance(source, exp.Table) and dialect.case_sensitive(name)

		if dialect and not (dialect.PRESERVE_ORIGINAL_OUTPUT_NAME_CASE):
		dialect.normalize_identifier(selection.args["alias"])

Conversation

fivetran-kwoodbeck commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

georgesittas commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fivetran-kwoodbeck commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as duplicate.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fivetran-kwoodbeck Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fivetran-kwoodbeck Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fivetran-kwoodbeck Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fivetran-kwoodbeck commented Jun 29, 2026 •

edited

Loading

georgesittas commented Jun 30, 2026 •

edited

Loading

fivetran-kwoodbeck commented Jun 30, 2026 •

edited

Loading

fivetran-kwoodbeck Jul 6, 2026 •

edited

Loading

fivetran-kwoodbeck Jul 6, 2026 •

edited

Loading

fivetran-kwoodbeck Jul 6, 2026 •

edited

Loading