Reimplement DB2ToSqlite in Go - #1510
Open
hillerstorm wants to merge 11 commits into
Open
Conversation
… writer) First stage of docs/db2tool-migration-plan.md: everything downstream of file extraction, fed by pre-extracted .db2 files. - dbd/: BSD-3 port of DBDefsLib's DBDReader + the SQLite helpers' exact-build version selection (last block whose builds contain the trailing build number; fail-loud when WoWDBDefs lacks the build). - wdc/: MIT port of DBCD.IO's WDC5Reader/WDC4Row/BitReader — multi-section, all 6 compression modes, sparse offset-map, copy tables, trailing relations, negative-base string offsets, and the keyless encrypted-section skip (TactKeyLookup != 0 + zero-filled data). - sqlite/: port of this repo's SQLiteDbCreator/SqliteDataInserter via modernc.org/sqlite (delete-first lifecycle, generated [Col_i] VIRTUAL columns, table-less idx_ names in settings order, relation 0 stays 0, float32 marshaling, C# Array-declared JSON serialization semantics). - Temporary driver (main.go) decodes tools/DB2ToSqlite/dbfilesclient against DBDCache for an explicit --build; a follow-up commit replaces this with local CASC extraction. - internal/golden + tests: schema DDL gate, per-table row parity, the float-notation assertion from plan §5.5, modernc marshaling smoke test. Validation vs the Jul 11 dotnet reference (build 5.5.4.68571, with hotfixes): sqlite_master schema byte-identical (236 objects); row dumps byte-identical for 67/72 tables; the 9 differing lines are exactly the DBCache hotfix overlay (add item 272920/spell 1291315 'Spring Panda', delete item 277947, modify spell 1298412) plus the documented CurvePoint float-notation case (slack table). gen_db against the Go-built DB reproduces the committed assets/database/db.json and leftover_db.json byte-for-byte (.bin files are nondeterministic even from the reference DB).
The new tact/ package reads the full local install — a deliberate behavior change from the CDN-fed .NET tool, proven byte-equivalent (see docs/db2tool-migration-plan.md): - .build.info parse + entry-by-Product + trailing build number - build/CDN config from Data/config (vfs-* lines parsed, never consulted) - local .idx v7 buckets (XOR bucket select, packed archive/offset bits, 30-byte storage frame skip), data.NNN via os.ReadAt - encoding EN table (paged BE binary search, 40-bit sizes) - TSFM root (post-10.1.7 dfVersion 1/2, delta-encoded FDIDs, enUS) - BLTE N/Z; keyless 'E' chunks stay zero-filled (plan §7 C1/M5) - FDID resolution: static path map (80 configured paths) + listfile fallback - listfile.csv + .dbd HTTP fetch/cache (Last-Modified / 24h-mtime rules), with offline fallback to existing copies - main.go: local mode derives the build from .build.info; --build keeps the offline pre-extracted mode; settings-relative paths resolve against tools/db2tool (M4, incl. the raw-TargetDirectory FDID-key carve-out) - makefile: make db / make ptrdb now run the Go tool; the three gen_db listfile.csv literals repoint to tools/db2tool/listfile.csv (§9.2) Pre-flight + gates (live install, wow_classic 5.5.4.68571): - one-line BaseDir patch on the .NET tool with its 1.3GB CDN cache moved aside re-extracted everything byte-identically from local CASC - Go tool, local mode: all 72 .db2 and 8 basestats .txt byte-identical to the CDN-fed originals; wowsims.db vs the fresh no-hotfix dotnet reference: schema identical, 2 residual diff lines (known CurvePoint float-notation case, slack table) - full make db pipeline with no dotnet reproduces the committed assets/database/db.json and leftover_db.json byte-for-byte Reference captures (gitignored tools/db2tool/refs/): wowsims.nohotfix.db, wowsims.hotfix.db, DBCache.68571.bin. Current live hotfix overlay is 7 rows (Spring Panda add, Item 277947 delete, Spell 1298412 description) — the input for the hotfix-overlay commit.
Ports DBCD.IO's HotfixReader/HTFXReader (XFTH v9) plus wow.tools.local's HotfixManager cache scanning and SStrHash to wdc/hotfix.go. Local-CASC runs now scan tools/db2tool/caches/*.bin and <BaseDir>/**/DBCache.bin and apply the exact-build cache's records before insertion: stable ascending-PushId order, add/replace whole rows decoded from the byte-aligned blobs (DBD field metadata, non-inline id from RecordId, non-inline relations read at their DBD-declared MetaDataFieldType), deletes remove rows, Combine dedup on the full record identity including data bytes. --dbcache <file> pins specific caches for deterministic runs; --no-hotfixes disables the overlay; the offline --build mode stays hotfix-free by default. Gates (build 68571, refs/DBCache.68571.bin snapshot): - with hotfixes vs refs/wowsims.hotfix.db: 2 diff lines total across all 72 tables — the documented CurvePoint Id=236585 float-notation divergence - --no-hotfixes vs refs/wowsims.nohotfix.db: same 2 lines only - overlay delta: +Item/ItemSparse 272920, +Spell/SpellName 1291315, -Item 277947, Spell 1298412 AuraDescription_lang modified - go build/vet/gofmt clean; TestHotfixGoldenParity added (skips without refs/), existing golden/wdc/dbd/sqlite tests pass; SStrHash(upper(name)) verified == WDC5 header TableHash for all 72 tables
make db / make ptrdb have run the pure-Go extractor since the previous commits; the dotnet tool, its vendored DLLs (TACTSharp, DBCD, DBCD.IO, DBDefsLib), the CDN cache, and the solution file are no longer needed. - delete tools/DB2ToSqlite/ (csproj, Program.cs, Helpers/, references/*.dll, appsettings, launch configs) and wowsims-mop.sln (it contained only this project) - the gitignored artifact dirs (dbfilesclient/, DBDCache/, listfile.csv) already live under tools/db2tool/; add tools/db2tool/caches/ (optional hotfix-cache scan dir) to .gitignore - docs/commands.md: make db now needs a local WoW install (Settings.BaseDir) instead of dotnet 9
The effect generator wrote raw template output, so the gofmt'ed committed files picked up whitespace churn on every regeneration. Render through go/format.Source instead. Entry sorts also only compared the first variant's item ID, which ties for items with multiple effects (e.g. Blessing of Zuldazar 138925/138967) and left the order to map iteration — tie-break on SpellID so the tooltip comments stop swapping between runs. enchants_auto_gen.go picks up a one-time formatting diff (it was committed unformatted); all three generated files now reproduce byte-identically.
The binary protobuf encoding is nondeterministic (map ordering), so a no-op regeneration previously churned the .bin files on every run. The JSON output is deterministic and covers all UIDatabase fields, so compare the freshly built JSON against the file on disk and skip writing both files when identical.
Correctness: - WriteBinaryAndJson only skipped writing when db.json was unchanged, so a deleted or corrupt db.bin was never regenerated and make db still exited 0. Require the binary to exist before skipping. WriteBinary/WriteJson now fail loudly instead of discarding os.WriteFile's error, which that guard relies on. - DecodeRows clones a copy-table row's decoded values, which only matches upstream while no column is COMMON-compressed: common values are resolved by row id, so a copy row would inherit the source id's value. No table has both today; error out instead of silently emitting wrong rows if that changes. - Bound the .dbd and listfile HTTP requests. The listfile deliberately bounds connect and response-header time only, since a legitimate ~150 MB transfer must not be cut off mid-flight. - Warn when --dbcache is given but no pinned file holds the extracted build, instead of quietly running without the hotfix overlay. - --build now takes a bare number or a full version string and no longer leaks an internally synthesized "0.0.0."+v into its error message. Unused code: bitReader.clone, the value32 accessors, Build.BuildConfig/ CDNConfig plus the CDN config read the local-only reader never needed, AvailableBuild.CDNPath/CDNConfig, five unread WDC5 header fields (the cursor still reads past them), and Decoded.ColumnNames, which had become a second source of truth for column order next to the version definitions. Settings files and their bindings now carry only the keys that are read. Tests: the golden gates diffed against a .NET reference capture, and the .NET tool is gone, so that reference can never be regenerated again. Together with the build-68571 fixtures under a gitignored refs/, 8 of 9 tests could only ever skip. Replaced with 21 fixture-free tests that always run: exhaustive unaligned bit-reader reads against an independent reference implementation, a synthetic XFTH v9 builder covering parse/dedup/rejection paths, SStrHash pinned to two known table hashes, and a .dbd parser suite built on an inline definition. Also fixes the sqlite test's all-zero-array assertion, which was reading the row it had just upserted. Idiom: license headers no longer sit in the package-doc slot (godoc showed concatenated license text), one package doc per package, range-over-int, strings.SplitSeq, slices.Sort/Equal, and max where gopls' modernize analyzer flagged them. Verified: extracting build 68806 with the live DBCache.bin reproduces the committed wowsims.db exactly — 236 schema objects identical, 0 differing rows across all 72 tables.
hillerstorm
marked this pull request as ready for review
July 28, 2026 15:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Also ports any c# dependency to go, including proper license text in
tools/db2tool/NOTICES.mdand/or relevant files.