test(e2e): characterization harness pinning clean-ingest behavior per modality

[LukasWodka]

Summary

Phase 0 of the refactor — the characterization harness (safety net). Adds e2e/test_characterization.py: for each bundled templates/ dataset it runs the real engine into real MySQL and pins the observable behavior the upcoming structural refactor must preserve. Validated locally against MySQL (docker-compose) across all 9 modalities.

What it pins (three dimensions, per modality)

1. MySQL rows — row count == source manifest; standard-column semantics (data_intent == configured intent, a single non-null ingestor_id, unique non-null data_id); and for the tabular family a feature-value round-trip (catches type corruption — leading zeros, NA handling, numeric coercion).
2. DEST_PATH file manifest — exactly the sidecar files copied for file-bearing categories. This directly catches the insert-rows-but-copy-no-files silent-half-ingest class (the instance_segmentation zombie pattern).
3. Backend payloads — the records + metadata the engine hands the APIClient. Since CLIENT_ENV=local short-circuits the HTTP call before serialising the payload, the harness spies on the APIClient method args (passed regardless of mode) — capturing the engine's intent to send.

Covers all 9 cleanly-ingesting modalities (tabular clf/reg, image clf, text clf, time-to-event, time-series, keypoint, object detection, MLM). Expectations are derived from the source files (row counts, sidecar listings, schema) — no hardcoded magic values — so the harness stays honest if a template changes.

Why it's stable on develop today

It characterises clean-input behavior. The in-flight fix PRs (#242–#245) only change malformed-input handling (bad cells, NA tokens, traversal filenames, dropped-record accounting), so these goldens hold now and become the contract the refactor is checked against. When those PRs merge, any golden that legitimately shifts documents the intended change.

It already earned its keep

Building it surfaced two behaviors worth pinning explicitly:

• MySQL FLOAT is 32-bit — a ~1e-6 feature round-trip delta is correct storage, not corruption (the round-trip tolerates float32 while still catching real corruption).
• The label column is mapped onto the standard label column — so e.g. regression's price is correctly absent from the physical feature schema.

Operational

Runs in the e2e job (real MySQL service). Auto-skipped in the unit suite when no MySQL is reachable (existing e2e/conftest.py collect_ignore_glob mechanism), so the default pytest + coverage gate is unaffected. New file only — no conflict with #242–#246.

🤖 Generated with Claude Code

[LukasWodka]

👋 Heads-up — Code review queue is at 15 / 8

Above the WIP limit. The team convention is to review existing PRs before opening new work.

Open PRs currently in Code review (oldest first):

• client#254 — Sync develop → main for v1.7.1 chart release (egress-enforcement preflight helm test, inert) · author: @saadqbal · no reviewer assigned
• client#255 — fix(egress-proxy): enforcement check probes TCP reachability, not HTTP code [image_validator falsely rejects images when target_size matches exactly #104] · author: @saadqbal · no reviewer assigned
• data-ingestors#233 — fix(validators): accept single-dict JSON top-level shape (issue DataValidator rejects single-dict JSON inputs that JSONIngestor.read_data is designed to handle #232) · author: @divyasinghds · no reviewer assigned
• data-ingestors#240 — fix: remove instance_segmentation zombie category, add dispatch-site congruence guard · author: @LukasWodka · no reviewer assigned
• data-ingestors#241 — fix: report real MySQL column types from get_table_schema (dialect-type reflection) · author: @LukasWodka · no reviewer assigned
• data-ingestors#242 — fix(coercion): single source of truth for NA policy + int64 range (bug: huge integer in INT column raises cryptic numpy "ufunc 'isinf' not supported" instead of a clean overflow error #236, bug: NA/null/None tokens crash non-tabular CSV ingest but are NULL for tabular — validator passes them (validate-pass → ingest-crash) #237) · author: @LukasWodka · no reviewer assigned
• data-ingestors#243 — fix(accounting): dropped records fail the run; JSON read-layer fails fast (bug: records dropped by skip-paths don't fail the run — exit 0 / Job 'Succeeded' + self-contradictory summary (follow-up to #99/#230) #234, bug: CSV aborts the whole ingest on one bad cell while JSON silently skips the row — make the policy consistent (follow-up to #189) #235) · author: @LukasWodka · no reviewer assigned
• data-ingestors#244 — fix(security): block path traversal via manifest filename/mask_id (security/bug: path traversal via unsanitised 'filename' column in file_transfer (read + write outside SRC_PATH/DEST_PATH) #239) · author: @LukasWodka · no reviewer assigned
• data-ingestors#245 — fix(ux): four ingestion validation/UX papercuts (bug: ingestion validation/UX papercuts (delimiter hint, NUL truncation, table-name message, Config numeric coercion) #238) · author: @LukasWodka · no reviewer assigned
• data-ingestors#246 — refactor(packaging): split runtime vs dev requirements; declare numpy · author: @LukasWodka · no reviewer assigned

Pull from review before opening new work. (This is a nudge from the kanban WIP check, not a block.)