Repository navigation
fix(viewer): match columns by name when loading multiple files - #305
Merged
Merged
Conversation
Later files were appended with a positional INSERT ... SELECT, so a file that lists the same columns in another order had its values written into the wrong columns. Use INSERT ... BY NAME, with the added filename column in a subquery so an existing filename column is still renamed to filename_1.
donghaoren
approved these changes
Nov 16, 2026
Collaborator
|
Thanks for the fix! |
donghaoren
enabled auto-merge (squash)
November 16, 2026 19:52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When more than one file is loaded in the file viewer (the upload page accepts multiple
.csv,.parquet,.jsonand.jsonlfiles), the first file creates thedatasettable and each further file is appended withINSERT INTO dataset SELECT *, ... AS filename. DuckDB'sINSERT ... SELECTmatches columns by position, so if a later file has the same columns in a different order, its values go into the wrong columns. When the swapped columns have compatible types this happens silently. For example,x/ycoordinates or two text fields trade places for every row of the second file.To reproduce, upload these two CSV files together:
The row for
cends up withx = 30.5, y = 3.5instead ofx = 3.5, y = 30.5. This is the relevant part of the new test's failure on main (- Expected,+ Received):The same happens with JSON/JSONL files whose objects list their keys in a different order. The command line tool loads the same two files correctly, because
pandas.concataligns columns by name.This PR appends later files with
INSERT INTO ... BY NAME. The addedfilenamecolumn goes through a subquery, so a file that already has its ownfilenamecolumn keeps working: the subquery renames the duplicate tofilename_1, the same way theCREATE TABLE ... ASfor the first file does. A plainBY NAMEwould fail with "Duplicate column name" in that case. As a side effect, a later file that is missing a column now loads with NULLs for that column instead of failing with a column count mismatch. A later file with an extra column still fails, as it does today.The new test
packages/viewer/test/import_data.test.tsrunsimportDataTableagainst DuckDB-Wasm in Node (blocking bindings) with two in-memory files. The reordered-columns case fails on main (xandyswapped for the second file) and passes with this change. A second case checks the existingfilenamecolumn behavior, which is unchanged.npx vitest runinpackages/viewerpasses (12 files, 146 tests), and prettier is clean on the changed files. Tested on macOS arm64, Node 24.This change was drafted with Claude Code (claude-opus-5-5); the test results above are from the runs described.