Skip to content

Return no-op instead of SQL error when ObservationLog has no valid entries - #2706

Open
pujitha24 wants to merge 3 commits into
kubeflow:masterfrom
pujitha24:auto/issue-2705
Open

Return no-op instead of SQL error when ObservationLog has no valid entries#2706
pujitha24 wants to merge 3 commits into
kubeflow:masterfrom
pujitha24:auto/issue-2705

Conversation

@pujitha24

Copy link
Copy Markdown

What this PR does / why we need it:

RegisterObservationLog in pkg/db/v1beta1/mysql/mysql.go and pkg/db/v1beta1/postgres/postgres.go builds an INSERT ... VALUES query by appending a (?, ?, ?, ?), placeholder group per MetricLog entry that has a non-empty TimeStamp, then strips the trailing comma with sqlQuery[0:len(sqlQuery)-1]. When every entry in the ObservationLog is skipped (empty MetricLogs, or every entry has an empty TimeStamp), no placeholder groups are appended, and the truncation instead chops the trailing space off "...VALUES ", producing a malformed statement. Prepare() then fails with a SQL syntax error, and that error is returned as-is by the ReportObservationLog gRPC handler (cmd/db-manager/v1beta1/main.go), so a call that should be a harmless no-op fails the RPC instead. This is per-RPC only — it does not crash or otherwise affect the db-manager process itself, just that one ReportObservationLog call.

Approach: Add an early if len(values) == 0 { return nil } guard right after the loop that builds values, before the query string is truncated, in both mysql.go and postgres.go. This is the primary fix the issue proposed. RegisterObservationLog's only caller passes its error straight through as the gRPC error, so returning nil when there is nothing to insert is a safe no-op with no skipped side effects.

Validation:

  • Added TestRegisterObservationLogNoValidEntries to pkg/db/v1beta1/mysql/mysql_test.go and pkg/db/v1beta1/postgres/postgres_test.go, each registering an ObservationLog with a single MetricLog whose TimeStamp is empty (so it is skipped), asserting RegisterObservationLog returns nil.
  • Confirmed these tests reproduce the exact bug: reverting only the two-line fix (keeping the new tests) makes both fail with Prepare SQL statement failed: ... call to Prepare 'INSERT INTO observation_logs (trial_name, time, metric_name, value) VALUES' query was not expected — sqlmock rejecting the malformed truncated query from the issue report. Re-applying the fix makes them pass.
  • go build ./pkg/db/... — passes.
  • go test ./pkg/db/v1beta1/mysql/... ./pkg/db/v1beta1/postgres/... — all tests pass, including the two new ones and the pre-existing TestRegisterObservationLog.
  • gofmt -l on the changed files — clean.
  • go vet ./pkg/db/v1beta1/mysql/... ./pkg/db/v1beta1/postgres/... — clean.
  • golangci-lint run ./pkg/db/v1beta1/mysql/... ./pkg/db/v1beta1/postgres/... reports 7 pre-existing issues (errcheck on stmt.Close()/rows.Close(), staticcheck ST1005 on capitalized error strings); verified via git stash that all 7 are present identically on unmodified master and none are introduced by this change. Note: the locally available golangci-lint is v2.12.2, while this repo's make lint pins v1.64.7 — a version mismatch, disclosed here rather than resolved, since the flagged lines are all pre-existing and outside this diff.
  • Did not run against a live MySQL or PostgreSQL instance; validation relies on this package's existing sqlmock-based unit test harness (github.com/DATA-DOG/go-sqlmock), the same harness the file's pre-existing RegisterObservationLog test uses.
  • Branch is cut from upstream/master tip (dec5030ef9791add2a7224d8ba3cb9b699bf70da).

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Checklist:

  • Docs included if any changes are user facing

Fixes #2705

…tries

Motivation:
RegisterObservationLog in pkg/db/v1beta1/mysql/mysql.go and
pkg/db/v1beta1/postgres/postgres.go builds an INSERT ... VALUES query by
appending a placeholder group per MetricLog entry with a non-empty
TimeStamp, then strips the trailing comma via
sqlQuery[0:len(sqlQuery)-1]. When every entry is skipped (empty
MetricLogs, or every entry has an empty TimeStamp), no placeholder
groups are appended, and the truncation instead chops the trailing
space off "...VALUES ", producing a malformed statement that fails
Prepare() with a SQL syntax error. That error is returned as-is by the
ReportObservationLog gRPC handler, so a call that should be a harmless
no-op fails the RPC instead. This is per-RPC only; it does not affect
the db-manager process itself.

Approach:
Add an early "if len(values) == 0 { return nil }" guard right after
the loop that builds values, before the query string is truncated, in
both mysql.go and postgres.go.

Validation:
Added TestRegisterObservationLogNoValidEntries to
pkg/db/v1beta1/mysql/mysql_test.go and
pkg/db/v1beta1/postgres/postgres_test.go, each registering an
ObservationLog with a single MetricLog whose TimeStamp is empty,
asserting RegisterObservationLog returns nil. Confirmed the tests
reproduce the exact bug: reverting only the two-line fix (keeping the
new tests) makes both fail with "Prepare SQL statement failed: ...
call to Prepare 'INSERT INTO observation_logs (trial_name, time,
metric_name, value) VALUES' query was not expected"; re-applying the
fix makes them pass.

Ran:
- go build ./pkg/db/... -> passes
- go test ./pkg/db/v1beta1/mysql/... ./pkg/db/v1beta1/postgres/... -> all pass
- gofmt -l on changed files -> clean
- go vet ./pkg/db/v1beta1/mysql/... ./pkg/db/v1beta1/postgres/... -> clean
- golangci-lint run on the changed packages reports 7 pre-existing
  issues (errcheck on stmt.Close()/rows.Close(), staticcheck ST1005 on
  capitalized error strings); verified via git stash that all 7 exist
  identically on unmodified master, none introduced by this change.
  Locally available golangci-lint is v2.12.2 vs this repo's pinned
  v1.64.7 for make lint, a version mismatch disclosed here rather than
  resolved since the flagged lines are outside this diff.
- Did not run against a live MySQL or PostgreSQL instance; validation
  relies on this package's existing sqlmock-based unit test harness,
  the same one the file's pre-existing RegisterObservationLog test
  uses.

Report: kubeflow#2705
Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
Assisted-by: claude-sonnet-5 (via Claude Code)
@github-actions

Copy link
Copy Markdown

🎉 Welcome to the Kubeflow Katib repo! 🎉

Thanks for opening your first PR! We're excited to have you onboard 🚀

Next steps:

Feel free to ask questions in the comments. Thanks again for contributing! 🙏

@google-oss-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign electronic-waste for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@AdeshDeshmukh

Copy link
Copy Markdown

Hi @pujitha24 , thanks for jumping on this so quickly! For transparency — I filed #2705 and also have a fix ready for this (same approach as proposed in the issue), which I was holding until maintainer feedback before submitting a PR.

Since your PR is already up and looks correct, I'd suggest the maintainers move forward with it. I've left my review with a couple of small suggestions below.
Then follow up on that same PR with a review comment containing the suggestions (this is what actually adds value):

  1. Add a test case for fully-empty MetricLogs (MetricLogs: []*api_pb.MetricLog{}) — currently only empty-timestamp entries are covered.
  2. Optional nit: add a brief comment in mysql.go/postgres.go noting the no-op path deliberately skips Prepare/Exec.

Add a test covering a fully-empty MetricLogs slice (distinct from the
existing empty-timestamp case) and a short comment explaining why the
no-op path skips Prepare/Exec, per review feedback.

Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>

@vjkumar2756 vjkumar2756 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pujitha24
Great fix Returning early when no valid metric entries exist prevents malformed SQL syntax errors and avoids unnecessary DB execution.

Some Suggestions

  1. Add nil Guard (mysql.go & postgres.go)
    If observationLog is nil, iterating over observationLog.MetricLogs will cause a runtime panic (nil pointer dereference). Consider checking observationLog == nil at the top of RegisterObservationLog:
    if observationLog == nil || len(observationLog.MetricLogs) == 0 {
        return nil
    }

Add an early nil check for observationLog in mysql.go and postgres.go
to avoid a nil pointer dereference when iterating MetricLogs, per
review feedback from vjkumar2756.

Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
@pujitha24

Copy link
Copy Markdown
Author

Thanks both for the suggestions — pushed two follow-up commits addressing them:

  • @AdeshDeshmukh: added TestRegisterObservationLogEmptyMetricLogs covering a fully-empty MetricLogs slice (separate from the empty-timestamp case), plus a short comment on the no-op path in both mysql.go and postgres.go.
  • @vjkumar2756: added a nil check for observationLog at the top of RegisterObservationLog in both files, with a TestRegisterObservationLogNilObservationLog test confirming it returns nil instead of panicking.

Both packages' test suites pass locally with these changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

db-manager : RegisterObservationLog fails with SQL syntax error when observation log has no valid entries (MySQL & Postgres)

3 participants