Add GreptimeDB provider for SQL and PromQL - #1161
houyuwushang wants to merge 3 commits into
Conversation
Support PostgreSQL wire, MySQL wire, HTTP SQL and PromQL using the existing listeners, analyzers and cache engines. Qualify native adapter lookup by protocol and preserve existing providers' defaults. Add the shared seed loader, developer dashboards, compatibility corpus, real-origin integration tests, configuration examples and engine guide. Pin the developer origin to an official image containing the pgwire fix. Fixes trickstercache#1150 Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
|
@houyuwushang thanks i will review this today and provide feedback. initial question: what features require the nightly build vs the stable build? |
|
The nightly pin is for a PostgreSQL compatibility fix. On v1.2.1, Grafana's PostgreSQL datasource sends MySQL, HTTP SQL and PromQL don't depend on that fix. Their initial direct-origin checks worked on v1.2.1, but the completed provider suite was run against the pinned nightly, so I haven't established full stable-version compatibility yet. Once a stable release includes the fix, I'd switch the dev image back and rerun the suite. |
|
@houyuwushang this is excellent, thank you. I've tested the developer environment and it works perfectly, including the data seeder and the user provisioning. There's currently no dashboards for the PromQL, HTTP SQL and MySQL versions of the queries; so i did some some simple command line testing and watched trickster's prom counters increment on the cache lookup status as expected. I have a few findings below - some were identified by the AI Agent and I confirmed in my above testing, and one where I did not provide the right guidance in the todo steps (and my apologies). Security - Authenticated PromQL GET responses may be shared across users. GreptimeDB inherits Prometheus routes that set s-maxage on instant and metadata responses. OPC treats that header as permission to store an authenticated GET under a key without Authorization. Users with different GreptimeDB permissions could therefore receive each other’s cached results. Keep these responses credential-partitioned unless GreptimeDB itself authorizes sharing.
Performance - HTTP SQL delta responses are marshaled twice. Fallback validation marshals the complete result to io.Discard, then the response is marshaled again for the client. Both passes rebuild rows, sort, and JSON-encode, adding CPU and allocations even on cache hits. Validate without full serialization or reuse the serialized result.
Resource Management - HTTP SQL decoding duplicates large results in memory. The decoder materializes the entire JSON response as [][]any while the fetched body is already held, then builds a second dataset and encodes tags per row. This increases peak memory and per-row work for large or high-cardinality results
As part of this, i would also recommend checking out the ClickHouse separation between upstream format and client format, since it's a useful pattern to follow. In the ClickHouse provider, no matter what format the client requests (tsv, csv, json, etc.), clickhouse always requests TSVWithNamesAndTypes from the upstream server, and unmarshals TSV ultimately to a DataSet (it's unmarshaling is not great, so do not copy). If it's possible to unmarshal the upstream response via io.Writer directly to a timeseries.DataSet, that would be even better but may not always be possible given the response's row ordering.
Separate from these defects and performance items, I also found that in the HTTP Acceptance Check on http_sql_test.go:L250 is incorrect. All of the other time series accelerator providers normalizes the incoming time ranges (
|
There was a problem hiding this comment.
see comments above. regarding the io.Writer in the comment above, i will be PRing a drop in package to help with this shortly. @houyuwushang i have a PR in to your branch here at: houyuwushang#1
This branch has not been deployed
2 participants
Description
Fixes #1150. Targets
v2.2.This adds a
greptimedbbackend with PostgreSQL wire, MySQL wire, HTTP SQL and PromQL support. PostgreSQL and MySQL use the existing listeners; the MySQL dialect stays on Vitess. One backend can serve all three listener types, with separate native upstream URLs and cache namespaces.The provider caches supported read queries, including delta caching for the verified time-bucket shapes. Unverified SQL shapes fall back to object caching or passthrough. Ingest and non-TSDB APIs remain proxy-only.
RANGE ... ALIGNdelta caching, TQL caching and gRPC are left for the follow-ups in the checklist.The developer environment includes the shared trips seeder, direct/proxied Grafana datasources and dashboard, configuration examples, a captured SQL compatibility corpus, and a guide for adding another pgwire engine. PromQL route reuse and the pgwire/MySQL hooks keep the existing providers' defaults.
I used the official
nightly-20260923-e91faa9dfimage, pinned by digest, because it contains GreptimeTeam/greptimedb#9295. This is a nightly, not a stable release. Validation covered both pgwire auth modes, MySQL, HTTP SQL, PromQL and two-backend merges, plus the existing MySQL/PostgreSQL paths. The official-image run used the small 190,064-row fixture; the earlier full-fixture lifecycle run used a locally patched origin.Validation includes
go test ./..., affected-package race tests in three independent runs,go vet ./..., diff-scoped lint,govulncheck, the real-origin suites documented inintegration/greptimedb/README.md, and desktop/mobile Grafana checks. Whole-tree lint still reports the eight existing baseline findings; those are outside this change.Type of Change
AI Disclosure