Skip to content

fix(viewer): match columns by name when loading multiple files - #305

Merged
donghaoren merged 1 commit into
apple:mainfrom
0z5a:fix/import-match-columns-by-name
Nov 16, 2026
Merged

donghaoren merged 1 commit into
apple:mainfrom
0z5a:fix/import-match-columns-by-name

Conversation

@0z5a

@0z5a 0z5a commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

When more than one file is loaded in the file viewer (the upload page accepts multiple .csv, .parquet, .json and .jsonl files), the first file creates the dataset table and each further file is appended with INSERT INTO dataset SELECT *, ... AS filename. DuckDB's INSERT ... SELECT matches columns by position, so if a later file has the same columns in a different order, its values go into the wrong columns. When the swapped columns have compatible types this happens silently. For example, x/y coordinates or two text fields trade places for every row of the second file.

To reproduce, upload these two CSV files together:

a.csv            b.csv
x,y,text         y,x,text
1.5,10.5,a       30.5,3.5,c

The row for c ends up with x = 30.5, y = 3.5 instead of x = 3.5, y = 30.5. This is the relevant part of the new test's failure on main (- Expected, + Received):

      "filename": "b.csv",
      "text": "c",
-     "x": 3.5,
-     "y": 30.5,
+     "x": 30.5,
+     "y": 3.5,

The same happens with JSON/JSONL files whose objects list their keys in a different order. The command line tool loads the same two files correctly, because pandas.concat aligns columns by name.

This PR appends later files with INSERT INTO ... BY NAME. The added filename column goes through a subquery, so a file that already has its own filename column keeps working: the subquery renames the duplicate to filename_1, the same way the CREATE TABLE ... AS for the first file does. A plain BY NAME would fail with "Duplicate column name" in that case. As a side effect, a later file that is missing a column now loads with NULLs for that column instead of failing with a column count mismatch. A later file with an extra column still fails, as it does today.

The new test packages/viewer/test/import_data.test.ts runs importDataTable against DuckDB-Wasm in Node (blocking bindings) with two in-memory files. The reordered-columns case fails on main (x and y swapped for the second file) and passes with this change. A second case checks the existing filename column behavior, which is unchanged. npx vitest run in packages/viewer passes (12 files, 146 tests), and prettier is clean on the changed files. Tested on macOS arm64, Node 24.

This change was drafted with Claude Code (claude-opus-5-5); the test results above are from the runs described.

Later files were appended with a positional INSERT ... SELECT, so a file that lists the same columns in another order had its values written into the wrong columns. Use INSERT ... BY NAME, with the added filename column in a subquery so an existing filename column is still renamed to filename_1.
@donghaoren

Copy link
Copy Markdown
Collaborator

Thanks for the fix!

@donghaoren
donghaoren enabled auto-merge (squash) November 16, 2026 19:52
@donghaoren
donghaoren merged commit 5c5b5c0 into apple:main Nov 16, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants