Skip to content

fix(viewer): treat DECIMAL columns as quantitative in field stats - #290

Open
kwy404 wants to merge 1 commit into
apple:mainfrom
kwy404:fix-decimal-field-stats
Open

kwy404 wants to merge 1 commit into
apple:mainfrom
kwy404:fix-decimal-field-stats

Conversation

@kwy404

@kwy404 kwy404 commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

Anyone whose data has a DECIMAL column, common in Parquet files exported from databases or Spark, cannot color the embedding by that column: the viewer throws Error: invalid data type, and charts for the column get no field stats.

Root cause: computeFieldStats (packages/viewer/src/charts/common/aggregate.ts) only computes quantitative stats for the types in its quantitativeTypes set, which has no DECIMAL types, so it returns undefined for a column that DuckDB reports as DECIMAL(18,3). The rest of the viewer already treats DECIMAL as numeric and continuous (jsTypeFromDBType and isFloatingPointDBType in utils/db_types.ts), so makeCategoryColumn sends such a column to makeBinnedNumericColumn, which needs those stats and throws when they are missing.

Fix: also treat the types matched by isFloatingPointDBType as quantitative, which adds DECIMAL(p,s) and NUMERIC(p,s). The stats query already casts the field to DOUBLE, so nothing else changes. It is imported from db_types.js, which has no database or browser dependencies.

Test: added packages/viewer/test/aggregate.test.ts, which runs computeFieldStats against a stub coordinator that reports the column type as DECIMAL(18,3). Before the fix it returns undefined; after the fix it returns quantitative stats. npx vitest run in packages/viewer (145 tests) and prettier -c pass, and svelte-check reports nothing in the changed files.

computeFieldStats only treats the types in its quantitativeTypes list as
numbers, and that list has no DECIMAL types, so it returns no stats for
a column such as DECIMAL(18,3). Coloring the embedding by such a column
goes through makeBinnedNumericColumn, which needs those stats and throws
"invalid data type". Also accept the types that isFloatingPointDBType
matches, which include DECIMAL(p,s) and NUMERIC(p,s).

@Manishram-ai Manishram-ai left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this matches what I saw. I ran it against a real DuckDB (duckdb-wasm in Node) with a table holding DECIMAL(18,3), NUMERIC, DECIMAL(38,2), a low-cardinality DECIMAL(4,1) and a DECIMAL(9,2) with some nulls, and called makeCategoryColumn on each column. On main every one throws Error: invalid data type. On this branch they all get quantitative stats and a binned legend, and the nulls land in the (null / nan / inf) bucket. DuckDB reports NUMERIC as DECIMAL(18,3), so the regex covers it either way. The viewer tests pass (145) and the new test fails on main.

quantitativeTypes here has exactly the same entries as numberTypes in db_types.ts, so this condition ends up the same as jsTypeFromDBType(columnType) == "number". Using that would keep the numeric type list in one place, but the current version works fine too.

@kwy404

kwy404 commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for checking this against a real DuckDB. Good point on jsTypeFromDBType, it would keep the numeric types in one place. I left the condition as it is to keep this change small, but I can switch to it if a maintainer prefers that.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants