Skip to content

MongoDB: Per-type BSON to QValue converters behind BsonToQValueConverter - #4802

Merged
pfcoperez merged 4 commits into
mainfrom
DBI-91/connectors/mongodb/structured-ingestion/1-bson-converters
Sep 16, 2026
Merged

pfcoperez merged 4 commits into
mainfrom
DBI-91/connectors/mongodb/structured-ingestion/1-bson-converters

Conversation

@pfcoperez

@pfcoperez pfcoperez commented Sep 14, 2026

Copy link
Copy Markdown
Member

Expand the converter interface with QValueFromBsonValue, which dispatches on the BSON type to per-type converters.

No pre-existing behaviour change.

⚠️ These conversions are bound to evolve as we align schema inference with desired destination types.

NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.

Part of: https://linear.app/clickhouse/issue/DBI-1096

Expand the converter interface with QValueFromBsonValue, which dispatches on the
BSON type to per-type converters.

No pre-existing behaviour change.

NOTE: This is a split up from a feature branch (#4774) where all pieces are integrated.
@pfcoperez
pfcoperez requested a review from a team as a code owner September 14, 2026 15:50
@claude

claude Bot commented Sep 14, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

Comment thread flow/connectors/mongo/qvalue_convert.go Outdated

case bsoncore.TypeTimestamp:
t, i := v.Timestamp()
return c.QValueJSONFromTimestamp(t, i), nil

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that bsoncore.TypeTimestamp is not a direct translation to timestamp but https://www.mongodb.com/docs/manual/reference/bson-types/#timestamps

With an extra field for sequential order within the same second.

@pfcoperez

Copy link
Copy Markdown
Member Author

@claude fix linting issues

@claude

claude Bot commented Sep 14, 2026

Copy link
Copy Markdown

Claude finished @pfcoperez's task in 1m 43s —— View job


Lint fixes pushed

  • Gather context (lint config, changed files)
  • Pull the failing GolangCI-Lint job log (golangci-lint 2.13.2)
  • Fix reported issues
  • Commit & push (465d792)

What CI reported

All three failures were in flow/connectors/mongo/qvalue_convert_test.go:

Line Linter Issue
812 lll line 149 chars, max 144
872 testifylint (compares) require.True(t, a == b)require.Equal
873 testifylint (compares) same

Changes

-		{desc: "Object", input: bson.D{{Key: "x", Value: int64(1)}, {Key: "y", Value: bson.A{}}}, expected: types.QValueJSON{Val: `{"x":1,"y":[]}`}},
+		{
+			desc:     "Object",
+			input:    bson.D{{Key: "x", Value: int64(1)}, {Key: "y", Value: bson.A{}}},
+			expected: types.QValueJSON{Val: `{"x":1,"y":[]}`},
+		},
-			require.True(t, math.IsNaN(value) == math.IsNaN(result.Value().(float64)))
-			require.True(t, math.IsInf(value, 0) == math.IsInf(result.Value().(float64), 0))
+			require.Equal(t, math.IsNaN(value), math.IsNaN(result.Value().(float64)))
+			require.Equal(t, math.IsInf(value, 0), math.IsInf(result.Value().(float64), 0))

Formatting-only and assertion-form-only; no test semantics changed. I also scanned both changed files for any remaining lines over the 144-char lll limit — none.

Not verified locally: go and golangci-lint invocations are not in this job's allowed tool list, so I could not run the linter or the test suite here. The pushed commit will be checked by the GolangCI-Lint and Flow build and test workflows.
· branch DBI-91/connectors/mongodb/structured-ingestion/1-bson-converters

- Split the "Object" table entry across lines to satisfy lll (144 cols)
- Use require.Equal instead of require.True on a comparison (testifylint)

Co-authored-by: Pablo Francisco Pérez Hidalgo <273379+pfcoperez@users.noreply.github.com>
@pfcoperez

Copy link
Copy Markdown
Member Author

@claude fix linting issues

@claude

claude Bot commented Sep 14, 2026

Copy link
Copy Markdown

Claude finished @pfcoperez's task in 1m 26s —— View job


No lint issues remain — the failure is an infra flake

  • Check CI status
  • Pull the failing GolangCI-Lint job log
  • Determine root cause
  • Push a commit to re-trigger CI (25ae3a4)

What the latest lint run actually reported

Run 34870500962 (on 465d792, i.e. after the previous lint fixes) failed with 2 typecheck issues — neither in this PR's files:

flow/shared/aws_common/config.go:6:2: could not import github.com/aws/aws-sdk-go-v2/aws
  (internal/env.go:15:2: google.golang.org/genproto@... requires
   cloud.google.com/go/bigtable@v1.42.0 requires
   google.golang.org/api@v0.259.0: verifying go.mod:
   reading https://sum.golang.org/tile/8/0/x190/421:
   stream error: stream ID 1877; INTERNAL_ERROR; received from peer) (typecheck)

That is a transient checksum-database fetch failure against sum.golang.org — module verification never completed, so golangci-lint could not type-check the package and reported it as a typecheck issue. It is unrelated to flow/connectors/mongo/*, and no linter flagged either changed file.

The earlier real issues (lll at line 812, two testifylint compares at 872–873) were fixed in 465d792 and did not reappear.

What I pushed

Since a re-run needed a push, I bundled a genuine (if tiny) fix rather than an empty commit — a duplicated word in a comment added by this PR:

-		// Nested documents are are encoded as `QValueJSON` ...
+		// Nested documents are encoded as `QValueJSON` ...

This should clear the lint job on the next run assuming the module proxy behaves.

Also worth knowing

Flow build and test on 465d792 failed in one matrix leg only — flow_test (pg16, mysql-gtid, maria-11, mongo6.0, ch-lts, crdb-v24) — while the pg17/mongo7.0 and pg18/mongo8.0 legs passed. I did not investigate it since you asked specifically about linting; say the word and I'll dig into that job.

Not verified locally: go and golangci-lint are not in this job's allowed tool list, so I could not run the linter here. I did review both changed files by hand against flow/.golangci.yml (including the 144-char lll limit and the testifylint/funcorder/gocritic settings) and found nothing outstanding.
· branch DBI-91/connectors/mongodb/structured-ingestion/1-bson-converters

Co-authored-by: Pablo Francisco Pérez Hidalgo <273379+pfcoperez@users.noreply.github.com>
@CLAassistant

CLAassistant commented Sep 15, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Comment thread flow/connectors/mongo/qvalue_convert.go
Comment thread flow/connectors/postgres/client.go
@pfcoperez
pfcoperez force-pushed the DBI-91/connectors/mongodb/structured-ingestion/1-bson-converters branch 2 times, most recently from 08f95af to 25ae3a4 Compare September 16, 2026 08:30
…onValue

Remove the QValueXFromY methods that were only called from the type
dispatch switch, constructing types.QValueX directly instead. Only
QValueStringFromObjectID and QValueStringFromString remain, as they are
also used by QValueStringFromId.

Co-authored-by: Pablo Francisco Pérez Hidalgo <273379+pfcoperez@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@pfcoperez
pfcoperez merged commit 619fc6e into main Sep 16, 2026
20 checks passed
@pfcoperez
pfcoperez deleted the DBI-91/connectors/mongodb/structured-ingestion/1-bson-converters branch September 16, 2026 09:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants