Conversation
…ngs can be passed with types for MongoDB sources NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
…oggle in the TableMapping NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
…out table mapping checks. Add basic structured ingestion checks. NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
…ed mappings with structured ingestion NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
dtunikov
reviewed
Sep 4, 2026
…ns: Malformed data tracker. NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
…mplementation of the generic "flattening" logic. NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
… specific BSON->QValue conversions NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
…s structured use literal CH types for table creations NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
…ect against schema specificity (depend just on kind) NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
…n to provide the column QValue schema NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
…ion from CH types to QKInd. I think we can skip this by doing the checks in the other way around: QKind -> CHTypes through `ToDWHColumnType` NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
…a for CDC (including initial load) MongoDB flow. NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
…gging flag in `QRepConfig` for MongoDB QRep flows. NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
Contributor
❌ Test FailureAnalysis: Two new Mongo→ClickHouse structured-ingestion e2e tests fail deterministically on all three matrix legs because the destination table is never created with the expected schema (ClickHouse "Unknown table expression identifier 'test_structured_dst'" and "Unknown expression identifier |
Contributor
❌ Test FailureAnalysis: Deterministic real failure: the two new MongoDB structured-ingestion e2e tests fail identically across all three CI matrix legs because the ClickHouse destination table (test_structured_dst / test_structured_flow_dst) and its |
…tion` flag through `NameAndExclude` mapping summary passed to CDC PullRecords req. NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
… constructors NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
…nector: Factor out CHtype -> QKind conversion so it can be re-used NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
…n CH -> QKind conversion function NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
…nector: Factor out CHtype -> QKind conversion so it can be re-used NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
Contributor
❌ Test FailureAnalysis: A deterministic Go compile error ( |
…pected values tracker generating a JSON report to be ingested along with correct values NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
…zy iterator of QValues (like a walk through a JSON or a BSON object) into a flat array of column QValues including malformed data according to the schema NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
…be wrapped around a field named "malformed_data". The column name suffices. NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
…for CDC flow NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
…nector: Add support for nullable JSON columns. Required for `malformed_data`. NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
…-ingestion/abstractions' into DBI-1095/connectors/mongodb/typed-mappings
… for destination_type overrides in normalize NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
pfcoperez
added a commit
that referenced
this pull request
Sep 10, 2026
…or destination_type overrides in normalize (#4783) With a table mapping column setting both a `destination_type` override and nullability (table- or column-level `nullable_enabled`), the DDL generator creates the destination column as Nullable(<type>), but the normalize query still extracted it as plain <type>. JSONExtract to a non-nullable type turns JSON nulls into the type's default, so NULL values silently landed as `0`, `" "`, etc. instead of NULL. This PR makes the normalize query generator mirror the DDL: - Wraps the override in Nullable(...) under the same conditions. - Guards both generators against double wrapping when the override is already spelled Nullable(...), which previously produced invalid Nullable(Nullable(<type>)) DDL. Part of: https://linear.app/clickhouse/issue/DBI-1096 Related to: - #4781 - #4774
…p_unexpected_values to structured ingestion CDC and QRep Replaces the temporary cdc/qrepStructuredRecordMalformedValues constants: the projector records offending values in the malformed data reports unless the mapping sets drop_unexpected_values. The flag travels to CDC through NameAndExclude and to QRep mirrors through the QRepConfig field mirroring it. NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
…/mongodb/typed-mappings
… type conbinations with Nullable and LowCardinality
Contributor
🔄 Flaky Test DetectedAnalysis: Not a test failure at all — both matrix jobs died during environment setup when Docker Hub returned "pull access denied" for the digest-pinned minio/minio image across 3 retries, an infrastructure/registry issue unrelated to the PR's MongoDB typed-mappings changes. ✅ Automatically retrying the workflow |
Contributor
❌ Test FailureAnalysis: Not a test failure or a flake: all 3 matrix jobs died during infrastructure bring-up because the digest-pinned |
…onnectors/mongodb/typed-mappings
…/mongodb/typed-mappings
pfcoperez
force-pushed
the
DBI-1095/connectors/mongodb/typed-mappings
branch
from
September 14, 2026 10:57
7f76e56 to
826a652
Compare
pfcoperez
added a commit
that referenced
this pull request
Sep 14, 2026
Expand the converter interface with QValueFromBsonValue, which dispatches on the BSON type to per-type converters. No pre-existing behaviour change. NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
… in MongoDB structured ingestion instead of landing as JSON columns
…ersion in flow/pkg This enables ClickPipes' Discovery MongoDB schema inference. Part of: https://linear.app/clickhouse/issue/DBI-1098
pfcoperez
added a commit
that referenced
this pull request
Sep 16, 2026
…ter (#4802) Expand the converter interface with QValueFromBsonValue, which dispatches on the BSON type to per-type converters. No pre-existing behaviour change. :warning: These conversions are bound to evolve as we align schema inference with desired destination types. NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated. Part of: https://linear.app/clickhouse/issue/DBI-1096
pfcoperez
force-pushed
the
DBI-1095/connectors/mongodb/typed-mappings
branch
from
September 17, 2026 09:01
826a652 to
fd2f258
Compare
pfcoperez
added a commit
that referenced
this pull request
Sep 17, 2026
…n normalization (#4813) JSONExtractString yields `''` both for a JSON null and for a missing field. Casting `''` or `NULL` with `::JSON` (behaviour prior to these changes) fails. These changes route NULL for Nullable(JSON) columns in the normalize query casting to `Nullable(JSON)`. This is required by structured ingestion, whose `_peerdb_malformed_data` column is a `Nullable(JSON)`. NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated. Part of: https://linear.app/clickhouse/issue/DBI-1096
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
These changes introduce the table mappings options for structured ingestion.
Opening as draft until it addresses the actual document flattening at ingestion.
Closes: https://linear.app/clickhouse/issue/DBI-1095
Part of: https://linear.app/clickhouse/issue/DBI-91