Release v0.3.10 (ingestion hardening + path-traversal fix + single-label preflight)

[divyasinghds]

Summary

Release v0.3.10. Bumps __version__ 0.3.9 → 0.3.10. Bundles the fixes merged to develop since v0.3.9.

Security

• Block path traversal via manifest filename/mask_id (security/bug: path traversal via unsanitised 'filename' column in file_transfer (read + write outside SRC_PATH/DEST_PATH) #239 → fix(security): block path traversal via manifest filename/mask_id (#239) #244)

Ingestion correctness & accounting

• Dropped records now fail the run; JSON read-layer fails fast (fix: propagate hard ingest failures in all modalities, send only inserted records #230, bug: records dropped by skip-paths don't fail the run — exit 0 / Job 'Succeeded' + self-contradictory summary (follow-up to #99/#230) #234, bug: CSV aborts the whole ingest on one bad cell while JSON silently skips the row — make the policy consistent (follow-up to #189) #235 → fix(accounting): dropped records fail the run; JSON read-layer fails fast (#234, #235) #243)
• Single source of truth for NA policy + int64 range handling (bug: huge integer in INT column raises cryptic numpy "ufunc 'isinf' not supported" instead of a clean overflow error #236, bug: NA/null/None tokens crash non-tabular CSV ingest but are NULL for tabular — validator passes them (validate-pass → ingest-crash) #237 → fix(coercion): single source of truth for NA policy + int64 range (#236, #237) #242)
• Real MySQL column types from get_table_schema (fix: report real MySQL column types from get_table_schema (dialect-type reflection) #241)
• CHAR(N) maps to a SQLAlchemy type (fix(database): CHAR(N) must map to a SQLAlchemy type in _get_sqlalchemy_type #249)
• Empty CSV fails fast with a clear input error (fix(csv): empty CSV must fail fast with clear input-error message #250)
• DataValidator: accept single-dict JSON / filter non-dict items (DataValidator rejects single-dict JSON inputs that JSONIngestor.read_data is designed to handle #232, fix(validators): accept single-dict JSON top-level shape (issue #232) #233)

Single-label classification preflight (#251 → #252)

• New LabelDiversityValidator catches unlearnable single-class datasets at preflight and surfaces the backend reason — instead of a misleading "Backend failed to prepare the dataset" after rows land in MySQL.
• Hardened through six Cursor Bugbot rounds so the validator reads the label column identically to CSVIngestor: fail-closed on read errors, pandas-native header parse, build_csv_na_values NA sentinels, string-dtype pin, full (unstripped) schema, and whitespace-insensitive column resolution.

UX / CLI / packaging

• Four ingestion validation/UX papercuts: delimiter hint, NUL truncation, table-name message, Config numeric coercion (bug: ingestion validation/UX papercuts (delimiter hint, NUL truncation, table-name message, Config numeric coercion) #238 → fix(ux): four ingestion validation/UX papercuts (#238) #245)
• Schema descriptions surfaced in validation errors instead of raw JSON-schema mechanics (fix(cli): surface schema descriptions in validation errors instead of raw JSON-schema mechanic #254)
• ConsoleRenderer extraction (refactor(P2): extract ingestion-summary renderer into reporting.ConsoleRenderer #248); runtime vs dev requirements split, numpy declared (refactor(packaging): split runtime vs dev requirements; declare numpy #246)
• Remove instance_segmentation from the schema enum + category congruence guard (fix: remove instance_segmentation zombie category, add dispatch-site congruence guard #240)

Test plan

• pytest green on release/v0.3.10 (local: 1043 passed, 1 xfailed)
• CI green
• python setup.py sdist bdist_wheel builds cleanly
• pip install dist/tracebloc_ingestor-0.3.10-*.whl then python -c "import tracebloc_ingestor; print(tracebloc_ingestor.__version__)" → 0.3.10

🤖 Generated with Claude Code

---

Note

High Risk
Changes touch security-sensitive path joining, core ingest/validate/DB/API paths, and packaging/CI install graphs across many modalities—high blast radius despite strong test coverage.

Overview
v0.3.10 bumps the package version and bundles correctness, security, and UX fixes with a large expansion of tests and CI wiring.

Security & file handling: Manifest filename/mask_id values are joined via _safe_join so reads/writes cannot escape SRC_PATH/DEST_PATH (#239).

Ingestion behavior: Dropped or invalid records and empty CSVs fail the run instead of silent success; JSON read/validate paths align with CSV (fail-fast, single-dict JSON, filter non-objects). Shared NA coercion and int64 overflow checks keep validator and ingest layers in agreement. get_table_schema maps reflected MySQL dialect types correctly; CHAR(N) is supported in DDL. Mid-batch DB failures send only inserted rows to the API; skipped_records count toward has_failures.

Preflight: New LabelDiversityValidator rejects single-class classification datasets locally; prepare_dataset stashes last_prepare_error for clearer errors (#251). instance_segmentation is removed from the schema; test_category_congruence guards full dispatch wiring.

Templates & packaging: All template scripts delegate to exported run_ingestion; summary UI moves to ConsoleRenderer. Runtime deps stay in requirements.txt (adds explicit numpy); test/lint tools move to requirements-dev.txt with setup.py filtering comment lines. CI installs requirements-dev.txt.

Tests: New e2e characterization harness over bundled modalities; expanded unit/e2e coverage for the above; CLI schema errors prefer rule descriptions over raw JSON Schema mechanics (#254).

^{Reviewed by Cursor Bugbot for commit aabfa20. Bugbot is set up for automated code reviews on this repo. Configure here.}

[LukasWodka]

👋 Heads-up — Code review queue is at 16 / 8

Above the WIP limit. The team convention is to review existing PRs before opening new work.

Open PRs currently in Code review (oldest first):

• averaging-service#106 — test(integration): pin XGBoost/LightGBM/CatBoost end-to-end averaging round-trips · author: @LukasWodka · no reviewer assigned
• averaging-service#111 — fix(averaging): correctness round-2 sweep — N5/N7/N8/N9/N10 + load_model except (Sync develop → master for v0.3.0 release #109) · author: @LukasWodka · no reviewer assigned
• client#254 — Sync develop → main for v1.7.1 chart release (egress-enforcement preflight helm test, inert) · author: @saadqbal · no reviewer assigned
• client-runtime#105 — chore(ci): drop unused mysqlclient dep; report test coverage (Phase 0) · author: @LukasWodka · no reviewer assigned
• client-runtime#106 — fix(schema): sync vendored ingest.v1.json — restore token_classification · author: @LukasWodka · no reviewer assigned
• data-ingestors#253 — refactor(P3a): move per-category behavior flags into a ModalityRegistry · author: @LukasWodka · no reviewer assigned
• data-ingestors#255 — refactor(P3b): move validator factories into the ModalityRegistry · author: @LukasWodka · no reviewer assigned
• data-ingestors#256 — refactor(P3c): move sidecar-transfer factories into the ModalityRegistry · author: @LukasWodka · no reviewer assigned
• data-ingestors#257 — refactor(P3d): source data_format from the ModalityRegistry · author: @LukasWodka · no reviewer assigned
• data-ingestors#258 — Develop to prod · author: @divyasinghds · no reviewer assigned

Pull from review before opening new work. (This is a nudge from the kanban WIP check, not a block.)

The base branch was changed.