Cherry-pick some commits from the main branch 2 - #1936
Open
leborchuk wants to merge 22 commits into
Open
Conversation
The project name "Cloudberry" was incorrectly capitalized as "CloudBerry" in two README files under contrib/. Correct the capitalization to match the canonical project name. Also, update the legacy brand `Cloudberry Database` to `Apache Cloudberry`.
* ORCA: align CBitSet vec_size for grouping-set bitsets
CreateGroupingSetsForRollup / Cube and the GROUPING_SET_EMPTY case in
GetColumnAttnosForGroupBy were constructing their accumulator/seed
CBitSets via the default ctor (vec_size = 256), then Union'ing in
per-grouping-set bitsets built with vec_size = num_cols. CBitSet::Union
just splices in any missing CBitSetLinks wholesale, so the accumulator
ended up with a link at offset 0 (vec_size 256) plus a stray link at
offset num_cols (vec_size num_cols) covering the high tleSortGroupRef.
CBitSet::Get then computed the offset using the destination's
m_vector_size = 256 and never consulted the stray link, while
CBitSetIter happily walked both -- so Get(k) disagreed with the iterator
for any k >= num_cols.
In CreateDXLProjectNullsForGroupingSets this caused tleSortGroupRefs >=
num_cols to be misclassified as non-grouping columns and NULL'd out,
even in grouping sets that included them. Visible as:
select generate_series(1, a) g, a+b ab
from (values (1,1),(2,2)) t(a,b)
group by rollup(a, ab) order by 1,2;
returning 0 rows instead of 6 -- the rollup(a, ab) branch projected the
a column as NULL, so generate_series(1, NULL) produced no rows.
Fix by passing num_cols when constructing the accumulator and seed
bitsets so all participants in the Union share m_vector_size.
Add the repro to groupingsets.sql.
* ORCA: assert matching CBitSet vec_size and cover cube() in test
Add build-cloudberry-rocky10.yml based on build-cloudberry.yml to validate Apache Cloudberry on Rocky Linux 10. - Use cbdb-build-rocky10-latest and cbdb-test-rocky10-latest images - Pin dnf install to `--releasever=10` - Trigger on push, workflow-file PRs (paths filter), a weekly Monday 02:00 UTC schedule, and manual dispatch, mirroring the rocky8 workflow - Tag workflow, job, and report names with "(Rocky 10)" to distinguish steps from other workflows Assisted-by: Claude Code
Two el10 build breakages, neither PostgreSQL-version specific: - configure-cloudberry.sh printed version info via `ag` (the_silver_searcher), which is not packaged for el10. Use `grep -E`, which is always available. - build-rpm.sh failed in %install because check-rpaths on el10 treats the product's absolute RUNPATH (/usr/local/cloudberry-db/lib) as a fatal invalid rpath. Export QA_RPATHS to demote the standard, invalid and empty rpath findings to warnings. Assisted-by: Claude Code
The RPM verify/copy steps in the build workflows broke on el10: - The OS major version was parsed with `[0-9]`, capturing only "1" from VERSION_ID="10" and looking for an el1 RPM. Use `[0-9]+`. - `rpm -qlp ... | grep -q` raced with pipefail: grep closed the pipe on the first match, rpm died with SIGPIPE, and the pipeline was reported as failed. Drop `-q` and redirect grep so the whole list is consumed. Applied to build-cloudberry.yml, build-cloudberry-rocky8.yml and build-dbg-cloudberry.yml. Assisted-by: Claude Code
gpopt_mock.c uses PG_FUNCTION_ARGS and PG_RETURN_VOID() but only included postgres.h. On GCC 14 (Rocky Linux 10) the undeclared macros became hard errors (-Wimplicit-function-declaration, -Wimplicit-int) rather than warnings. Include fmgr.h, the canonical header for these function-manager macros. Assisted-by: Claude Code
pg_hint_plan's maskout.sh filtered plan output through `egrep`. On Rocky Linux 10 that prints "egrep is obsolescent; using grep -E" to stderr, which leaked into the rowhints test output and broke the regression diff. Use `grep -E` directly. Assisted-by: Claude Code
The singlenode AOCO_Compression test asserted an exact on-disk size (712 bytes / 36.75), which depends on the zlib version. Rocky Linux 10's zlib produced 728 bytes / 35.95 and failed the diff. Ignore the detailed output here. Assisted-by: Claude Code
The sonarqube workflow failed because `@v6` of both SonarSource/sonarqube-scan-action and its install-build-wrapper sub-action are not in the ASF allowed actions list. Pin both actions to commit 713881670b6b3676cda39549040e2d88c70d582e (v8.2.0), the latest approved version from apache/infrastructure-actions/actions.yml that has no expiry constraint.
The function had a few problems: - replace ="a" and replacement="ab" would give an infinite loop leading to OOM. We have a similar case in pg_dump: replace="range", replacement="multirange" - Copying the whole string each time there is a match - Empty replace pattern also leads to infinite loop We can still do better: for example count number of matches and do a single allocation, or even run the replacement inplace. But it's probably not worth it.
ORCA represents names as wide characters; when the database LC_CTYPE cannot decode a multibyte name (e.g. LC_CTYPE='C' with a UTF-8 alias), clib::Vswprintf substitutes the generic "UNKNOWN" string and the DXL-to-PlStmt translator restores the original name from the query tree (commit 6b19c44). That restore only ran for Var target entries, so with the optimizer enabled SELECT '한글' AS "한글"; returned a column named UNKNOWN: constants, aggregates, set operation and set-returning-function outputs never took the repair path. The old repair walked the entire query tree with update_unknown_locale_walker matching on (resorigtbl, resno). That match key is ambiguous: TargetEntries nested inside Aggref arguments or SubLink subqueries can collide with it, either clobbering an already-restored name (empty column header for the first of two aggregates) or restoring a name from the wrong query level. Replace the walker with restore_unknown_locale_resname, which scans only the top-level query targetList for the non-junk entry with the same resno. Only the topmost plan node is translated with a context that carries the original query, and its projection list produces the query output columns in order, so the positional top-level match is exact. A legitimate alias literally named "UNKNOWN" self-matches and the restore is a no-op. Call it for every entry translated by TranslateDXLProjList, for the inlined Append targetlist (UNION ALL), and for the ProjectSet targetlists built by SetupAliasParameter (set-returning functions). Extend gp_locale with alias cases for constants, expressions, aggregates, UNION ALL, set-returning functions, a subquery containing a same-position column, and a legitimate "UNKNOWN" alias.
Add Rocky Linux 10 (rocky10) as a supported target in the package-convenience-binaries workflow, covering both x86_64 and arm64 architectures. Build job: - using apache/incubator-cloudberry:cbdb-build-rocky10-latest Install-test job: - using apache/incubator-cloudberry:cbdb-test-rocky10-latest
Rework the RPM packaging so Cloudberry can be installed to a custom
location and so multiple major versions can coexist, aligning the
behavior with Greenplum's packaging model.
Relocation (rpm --prefix):
- Split the hardcoded install path into base_dir + name components and
set Prefix to the base directory (/usr/local) so RPM's relocate
engine can substitute it correctly.
- Move the convenience symlink out of %files and create it in %post
using RPM_INSTALL_PREFIX, so it follows the actual --prefix (with a
fallback to the default base_dir). %postun removes the symlink only
when it still points to the version being erased.
- Disable build-id links so they are not emitted outside the
relocatable prefix.
Major-version coexistence:
- Embed the major version in the package Name
(apache-cloudberry-db-incubating-<major>), derived from %{version},
so different major versions register as distinct packages and can be
installed side by side (dnf) or under separate prefixes (rpm -i).
- Obsolete the previous unversioned package name on upgrade; this does
not match the versioned names, so majors still coexist.
- In %post, move the generic symlink only when the existing target is
the same major version, leaving other majors untouched. The version
is parsed from the target directory name rather than by sourcing
environment files or executing installed binaries.
- Keep the published file name in the historical format (without the
"-<major>" segment) by renaming artifacts in build-rpm.sh; the file
name does not affect the Name/Version stored in the RPM header.
Packaging correctness and hardening:
- Stop changing installed files to gpadmin ownership in %post; the
package tree stays root-owned.
- Do not expose bundled private shared libraries (libpq.so.5, etc.) as
Provides, and do not require them from the system; they ship in the
package and are resolved via RPATH.
- Validate that version/release macros are supplied, mark
cloudberry-env.sh as %config(noreplace), copy the tree with cp -a to
preserve hidden files and attributes, add coreutils scriptlet
dependencies, add python3 to the el9 runtime requires, and update the
license tag to the SPDX identifier Apache-2.0.
CI workflows:
- Query the installed package by glob (rpm -qa 'apache-...*') instead of
the fixed name, since the registered Name now carries the major.
- Keep locating the published artifact by its historical file name,
which build-rpm.sh restores after the build.
Assisted-by: Claude Code
Assisted-by: DeepSeek
…n file
Replace two separate Ubuntu DEB CI workflow files with one
matrix-driven workflow that runs build + test across Ubuntu
22.04 and 24.04.
Key changes:
- Rewrite: .github/workflows/build-deb-cloudberry.yml
- Remove: build-deb-cloudberry-ubuntu24.04.yml
- PR trigger: both Ubuntu versions now run on every PR (was only
22.04 before; 24.04 only triggered on push to main)
- Remove scheduled cron trigger (push + PR coverage is sufficient)
- Build/deb-install-test jobs use strategy.matrix.ubuntu_version
['22.04', '24.04']; test-deb job receives ubuntu_version via
prepare-test-matrix-deb cross-product expansion
- Container images dynamically resolved via
matrix.ubuntu_version (cbdb-build-ubuntu${version}-latest)
- Job naming: Ubuntu 22.04 keeps original names (no suffix) for
.asf.yaml compatibility; Ubuntu 24.04 appends "(Ubuntu 24.04)"
- Artifact names consistently suffixed with -ubuntu${version}
- Fix: remove broken matrix.name reference in deb-install-test
artifact upload (was referencing a non-existent matrix axis)
See: http://github.com/apache/cloudberry/discussions/1696
Assisted-by: DeepSeek
Replace three separate Rocky Linux CI workflow files with one
matrix-driven workflow that runs build + test across Rocky 8, 9, 10.
Key changes:
- New file: .github/workflows/build-cloudberry-rocky.yml
- Remove: build-cloudberry.yml (Rocky 9), build-cloudberry-rocky8.yml,
build-cloudberry-rocky10.yml
- PR trigger: all three Rocky versions now run on every PR (was only
Rocky 9 before; Rocky 8/10 only triggered on push to main)
- Remove scheduled cron trigger (push + PR coverage is sufficient)
- Test matrix: union of all test suites across versions; every test
now runs on every Rocky version, including ic-recovery, ic-diskquota,
ic-orca-parallel, and gp_relsizes_stats
- Container images and --releasever now resolved dynamically via
matrix.rocky_version
- Artifact names consistently suffixed with -rocky${{ version }}
See: http://github.com/apache/cloudberry/discussions/1696
Assisted-by: Deepseek
Immediate restart forces crash recovery; with hot_standby=off the pidfile briefly reports standby and pg_ctl -w returns early, so gpstart can fail with "Hot standby mode is disabled". Use fast restart like the other diskquota tests; POSTMASTER GUCs only need a normal restart.
GitHub Actions runners now force actions declaring Node.js 20 onto Node.js 24 and emit a deprecation annotation on every job. Node.js 20 support is removed entirely after June 2026, so the workflows need actions whose action.yml declares "using: node24". Version transitions applied across all workflow files: actions/upload-artifact v4 -> v7 (27 call sites) actions/download-artifact v4 -> v8 (11 call sites) actions/checkout v4 -> v7 (9 call sites) actions/setup-java v3 -> v5 (1 call site) Assisted-by: Claude Code
leborchuk
requested review from
reshke and
tuhaihe
and
a lite review from Copilot
August 27, 2026 15:54
There was a problem hiding this comment.
Pull request overview
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Cherry-picks a set of upstream fixes from main into REL_2_STABLE, spanning ORCA/planner correctness fixes, new gpcontrib functionality, packaging improvements, and CI/workflow consolidation.
Changes:
- Fixes ORCA grouping-sets bitset sizing and restores non-ASCII/locale-sensitive column aliases; adds/extends regression coverage for grouping sets and locale.
- Imports and adapts
gp_relsizes_statsintogpcontrib, including SQL objects, background worker implementation, and regression tests. - Updates CI/workflows and packaging scripts (Rocky 10 enablement, workflow matrix consolidation, RPM packaging behavior, and test flake reductions).
Reviewed changes
Copilot reviewed 49 out of 49 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| src/test/unit/mock/gpopt_mock.c | Adds missing include for unit test compilation. |
| src/test/singlenode_regress/sql/AOCO_Compression.sql | Makes compression-ratio output non-brittle by ignoring unstable size results. |
| src/test/singlenode_regress/expected/AOCO_Compression.out | Updates expected output to match ignore markers for unstable size results. |
| src/test/regress/sql/join_hash.sql | Adds planner TODO comment in join hash tests. |
| src/test/regress/sql/groupingsets.sql | Adds grouping sets regressions covering SRFs + derived-expression aliases. |
| src/test/regress/sql/gp_locale.sql | Adds regression coverage for restoring non-ASCII aliases on non-Var projections and set ops. |
| src/test/regress/expected/join_hash_optimizer.out | Updates expected output to include added test comment. |
| src/test/regress/expected/join_hash.out | Updates expected output to include added test comment. |
| src/test/regress/expected/groupingsets_optimizer.out | Updates expected output for new grouping-sets regression queries. |
| src/test/regress/expected/groupingsets.out | Updates expected output for new grouping-sets regression queries. |
| src/test/regress/expected/gp_locale.out | Updates expected output for new locale/alias regression queries. |
| src/common/stringinfo.c | Fixes potential infinite loop in replaceStringInfoString and optimizes replacement logic. |
| src/backend/optimizer/util/restrictinfo.c | Adds tracking for outer-query parameter references to inform motion planning. |
| src/backend/gporca/libgpos/src/common/CBitSet.cpp | Adds assertions to prevent mixing bitsets with mismatched vector sizes. |
| src/backend/gpopt/translate/CTranslatorUtils.cpp | Aligns grouping-set bitset vector sizes to avoid misaligned link offsets. |
| src/backend/gpopt/translate/CTranslatorDXLToPlStmt.cpp | Restores original aliases when ORCA wide-char conversion yields "UNKNOWN"; simplifies previous walker approach. |
| pom.xml | Excludes newly imported gpcontrib content from RAT checks where appropriate. |
| gpcontrib/pg_hint_plan/sql/maskout.sh | Replaces deprecated egrep usage with grep -E. |
| gpcontrib/gp_relsizes_stats/test/sql/grants.sql | Adds regression test for extension privileges and grantability. |
| gpcontrib/gp_relsizes_stats/test/sql/gp_relsizes_stats.sql | Adds functional regression coverage for stats collection and config behavior. |
| gpcontrib/gp_relsizes_stats/test/postgresql.conf.add | Ensures extension is preloaded during tests. |
| gpcontrib/gp_relsizes_stats/test/expected/grants.out | Expected output for new grants regression. |
| gpcontrib/gp_relsizes_stats/test/expected/gp_relsizes_stats.out | Expected output for new functional regression. |
| gpcontrib/gp_relsizes_stats/src/gp_relsizes_stats.c | Implements gp_relsizes_stats background worker + SQL-callable collection functions. |
| gpcontrib/gp_relsizes_stats/sql/gp_relsizes_stats--1.0.sql | Defines extension SQL objects (schema/tables/views/functions/grants). |
| gpcontrib/gp_relsizes_stats/gp_relsizes_stats.control | Adds extension control metadata. |
| gpcontrib/gp_relsizes_stats/README.md | Documents the new extension, configuration, and output tables. |
| gpcontrib/gp_relsizes_stats/Makefile | Adds build/install/regress harness for the new extension. |
| gpcontrib/gp_relsizes_stats/.gitignore | Ignores common build outputs for the new extension. |
| gpcontrib/gp_relsizes_stats/.clang-format | Adds formatting configuration for the new extension. |
| gpcontrib/diskquota/tests/regress/sql/test_max_monitored_databases.sql | Reduces flakiness by silencing gpstop/gpconfig output and adjusting flags. |
| gpcontrib/diskquota/tests/regress/expected/test_max_monitored_databases.out | Updates expected output to match silenced gpstop/gpconfig commands. |
| gpcontrib/Makefile | Adds gp_relsizes_stats to gpcontrib build list. |
| devops/build/packaging/rpm/build-rpm.sh | Relaxes EL10 rpath QA and normalizes published RPM artifact filenames. |
| devops/build/packaging/rpm/apache-cloudberry-db-incubating.spec | Enables relocatable installs via --prefix, versioned package naming, and safer symlink handling. |
| devops/build/automation/cloudberry/scripts/configure-cloudberry.sh | Removes ag dependency by switching to grep -E. |
| contrib/udp2/README.md | Fixes product naming from CloudBerry to Apache Cloudberry. |
| contrib/interconnect/README.md | Fixes product naming casing in documentation text. |
| .github/workflows/sonarqube.yml | Pins SonarQube action by SHA and updates checkout action version. |
| .github/workflows/package-convenience-binaries.yml | Adds Rocky 10 to convenience binaries build/test matrix and updates artifact actions versions. |
| .github/workflows/docker-cbdb-test-containers.yml | Updates checkout action version. |
| .github/workflows/docker-cbdb-build-containers.yml | Updates checkout action version. |
| .github/workflows/coverity.yml | Updates checkout action version. |
| .github/workflows/build-deb-cloudberry.yml | Consolidates Ubuntu 22.04/24.04 into a matrix-driven workflow and updates artifact action versions. |
| .github/workflows/build-deb-cloudberry-ubuntu24.04.yml | Removes redundant Ubuntu 24.04-only workflow after consolidation. |
| .github/workflows/build-dbg-cloudberry.yml | Updates artifact action versions and adapts RPM naming/install verification for versioned name changes. |
| .github/workflows/build-cloudberry.yml | Consolidates Rocky 8/9/10 CI and extends test matrix (incl. gp_relsizes_stats). |
| .github/workflows/apache-rat-audit.yml | Updates checkout/setup-java/upload-artifact action versions. |
Suppressed comments (1)
src/test/regress/sql/join_hash.sql:1
- The newly added test comment contains multiple typos (e.g., 'implemention', 'effitient', 'enhace', 'pushin'). Please correct the spelling to keep the test suite readable and professional, and update the matching expected files accordingly.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Member
|
Cool! Let's find out why |
%__provides_exclude_from drops the auto-generated Provides for every shared object under the install prefix, while %__requires_exclude only removes a hand-maintained list of sonames from the auto-generated Requires. The .so symlinks the package ships (libfoo.so -> libfoo.so.N.M) make rpm emit a Requires on libfoo.so.N, so any bundled library missing from that list turns into an external dependency. REL_2_STABLE builds src/interfaces with SUBDIRS = libpq ecpg gppc, where main builds libpq only, so the package additionally ships libecpg.so.6, libecpg_compat.so.3, libpgtypes.so.3 and libgppc.so.1. None of them were covered, and installing the RPM failed with: nothing provides libecpg_compat.so.3()(64bit) needed by ... nothing provides libgppc.so.1()(64bit) needed by ... libecpg.so.6 and libpgtypes.so.3 did not fail the install, they resolved against system libraries instead of the bundled ones, which is equally wrong. Add the four sonames to %__requires_exclude, and make build-rpm.sh reject any package whose auto-generated Requires names a soname the package itself installs, so the two filters cannot drift apart unnoticed again.
The RPM Name now carries the major version (apache-cloudberry-db-incubating-<major>), so `rpm -ql apache-cloudberry-db-incubating` no longer finds the freshly installed package. In the current-RPM step that made INSTALLED_PG empty and the step failed before it could locate the install tree. Read the Name from the RPM being installed instead, which works for the versioned name and for the historical unversioned one the baseline still uses, and check the result in both steps. This workflow only exists on REL_2_STABLE, so two earlier fixes never reached it: - `rpm -qlp ... | grep -q` races with `set -o pipefail`: grep closes the pipe on the first match, rpm dies with SIGPIPE and the pipeline is reported as failed. Drop -q and redirect grep instead. - The OS major version was parsed with `[0-9]`, which truncates a double-digit VERSION_ID. Use `[0-9]+`.
Member
|
Updated the branch with three changes. This was a force-push: 1. Dropped
|
tuhaihe
approved these changes
Aug 28, 2026
pax-ic-good-opt-off/on fail on Rocky Linux 10: - 712 bytes | 36.75 + 728 bytes | 35.95 mpp17012_compress_test2 is a zlib-compressed AOCO table, so its exact on-disk size depends on the deflate implementation the platform ships. Rocky 8 and Rocky 9 produce 712 bytes, Rocky 10 produces 728. src/test/regress stopped asserting the exact numbers in be38ff5: the query stays inside start_ignore for reference, and the size and the ratio are checked to be within 10% instead. main's copy of the pax suite carries the same treatment for both the zlib and the rle_type table since the PostgreSQL 16 merge (0f4cf8d); this branch's copy predates that. Take both blocks from main verbatim, so the assertions survive and the two branches do not drift further apart. The rle_type table does not depend on zlib and passes today, but it is included to keep the file in sync.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the 2nd part cherry-picking commits from main to REL_2_STABLE branch
First set of commits was merged in #1905
Fixes here:
49ce8e8 - Doc: Fix CloudBerry -> Cloudberry typo in READMEs
befb2bf - ORCA: align CBitSet vec_size for grouping-set bitsets
569828a - Import gp_relsizes_stats into gpcontrib from greenplum
cdf09de - Adapt gp_relsizes_stats for Cloudberry
95a300f - CI: add Rocky Linux 10 build and test CI workflow
f8dc627 - Fix Cloudberry build scripts for Rocky Linux 10
79423b6 - Fix RPM artifact checks for double-digit EL
e938d9b - Fix unit test build under GCC 14
98fda2f - Fix egrep deprecation warning in maskout.sh
6414a9a - Fix singlenode AOCO compression test for zlib
e4805bd - CI: fix SonarQube action to use ASF-approved pinned version
7f5c362 - Fix infinite loop in replaceStringInfoString
813779c - ORCA: restore non-ASCII column aliases for all target entry kinds
d327151 - CI: add Rocky Linux 10 to package convenience binaries matrix
1daae25 - Packaging: Enable RPM package relocation via --prefix
4910d42 - CI: consolidate Ubuntu 22.04/24.04 workflows into single matrix-driven file
0764227 - CI: consolidate Rocky 8/9/10 workflows into single matrix-driven file
50d36d9 - diskquota: avoid flaky gpstop -ari in max_monitored_databases test
51967d3 - CI: bump GitHub Actions to Node.js 24 runtimes
99c62cf - Fix JOIN motion type selection for join quals containing outer refs
Out os scope (not ready yet) - #1911