Repository navigation
Conversation
computeFieldStats only treats the types in its quantitativeTypes list as numbers, and that list has no DECIMAL types, so it returns no stats for a column such as DECIMAL(18,3). Coloring the embedding by such a column goes through makeBinnedNumericColumn, which needs those stats and throws "invalid data type". Also accept the types that isFloatingPointDBType matches, which include DECIMAL(p,s) and NUMERIC(p,s).
There was a problem hiding this comment.
Thanks, this matches what I saw. I ran it against a real DuckDB (duckdb-wasm in Node) with a table holding DECIMAL(18,3), NUMERIC, DECIMAL(38,2), a low-cardinality DECIMAL(4,1) and a DECIMAL(9,2) with some nulls, and called makeCategoryColumn on each column. On main every one throws Error: invalid data type. On this branch they all get quantitative stats and a binned legend, and the nulls land in the (null / nan / inf) bucket. DuckDB reports NUMERIC as DECIMAL(18,3), so the regex covers it either way. The viewer tests pass (145) and the new test fails on main.
quantitativeTypes here has exactly the same entries as numberTypes in db_types.ts, so this condition ends up the same as jsTypeFromDBType(columnType) == "number". Using that would keep the numeric type list in one place, but the current version works fine too.
|
Thanks for checking this against a real DuckDB. Good point on |
Anyone whose data has a DECIMAL column, common in Parquet files exported from databases or Spark, cannot color the embedding by that column: the viewer throws
Error: invalid data type, and charts for the column get no field stats.Root cause:
computeFieldStats(packages/viewer/src/charts/common/aggregate.ts) only computes quantitative stats for the types in itsquantitativeTypesset, which has no DECIMAL types, so it returnsundefinedfor a column that DuckDB reports asDECIMAL(18,3). The rest of the viewer already treats DECIMAL as numeric and continuous (jsTypeFromDBTypeandisFloatingPointDBTypeinutils/db_types.ts), somakeCategoryColumnsends such a column tomakeBinnedNumericColumn, which needs those stats and throws when they are missing.Fix: also treat the types matched by
isFloatingPointDBTypeas quantitative, which addsDECIMAL(p,s)andNUMERIC(p,s). The stats query already casts the field toDOUBLE, so nothing else changes. It is imported fromdb_types.js, which has no database or browser dependencies.Test: added
packages/viewer/test/aggregate.test.ts, which runscomputeFieldStatsagainst a stub coordinator that reports the column type asDECIMAL(18,3). Before the fix it returnsundefined; after the fix it returns quantitative stats.npx vitest runinpackages/viewer(145 tests) andprettier -cpass, andsvelte-checkreports nothing in the changed files.