PXF 2.2 Release plan

[ostinru]

Hi, cloudberry-pxf team!

There were discussion about what should be included into Apache Cloudberry 2.2 release. For cloudberry-pxf following was suggested:

• Add rocky Linux 10 to the list of supported platforms & Java 21 support
• Improve observability

I would like to add following 'best-effort' changes:

• Update hadoop libs to 3.3/3.4
• Declare that 2.x is latest cloudberry-pxf versions that support java 8-11. Then upgrade to java 17 in cloudberry-pxf 3.0
• Deprecate pxf-hbase and remove it in cloudberry-pxf 3.0 update pxf-hbase to hadoop 3.x as well
• Fix portion of tests from pxf_regress is broken need fix it #86

Do you have any plans / ideas on what should be included in cloudberry-pxf-2.2?

[tuhaihe]

Hi, I'm working on adding the PXF docs support to the cloudberry-site in PR #114. Hope can catch up with the coming release.

BTW, any reason for deprecating pxf-hbase and removing it in cloudberry-pxf 3.0?

[ostinru]

BTW, any reason for deprecating pxf-hbase and removing it in cloudberry-pxf 3.0?

Probably, I just don't like it - it amplifies dependency hell and I had never seen real HBase in production. Also, pxf-hbase requires some pxf JARs to be installed on every HBase server before using filter pushdown. Not sure that server administrators are happy to install some JARs to their server fleet.

PS: researched a bit. Huawei Cloud provides HBase as part of CloudTable. Probably, there are some real users.

[tuhaihe]

BTW, any reason for deprecating pxf-hbase and removing it in cloudberry-pxf 3.0?

Probably, I just don't like it - it amplifies dependency hell and I had never seen real HBase in production. Also, pxf-hbase requires some pxf JARs to be installed on every HBase server before using filter pushdown. Not sure that server administrators are happy to install some JARs to their server fleet.

PS: researched a bit. Huawei Cloud provides HBase as part of CloudTable. Probably, there are some real users.

Go it! What about your thoughts? @MisterRaindrop @leborchuk.

[MisterRaindrop]

I’m not sure if PXF-HBase is actually being used in production. I recommend sending an email to follow up for confirmation.

[MisterRaindrop]

Another question: What is the lifecycle support period we provide for PXF?

[ostinru]

Another question: What is the lifecycle support period we provide for PXF?

Same as Cloudberry lifecycle? What is Cloudberry lifecycle?

[tuhaihe]

Another question: What is the lifecycle support period we provide for PXF?

Same as Cloudberry lifecycle? What is Cloudberry lifecycle?

+1 to the idea of following the Cloudberry lifecycle. But we still don't have an official Cloudberry lifecycle in place. I believe this depends on the minor kernel upgrade process for 2.x./3.x -- which also depends on the upstream PostgreSQL lifecycle, IMO.

[tuhaihe]

Hi, I would also like to add a new best-effort point for PXF 2.2: #132

[tuhaihe]

BTW, should we include Rocky Linux 10 in the test matrix?

[MisterRaindrop]

BTW, should we include Rocky Linux 10 in the test matrix?

I think it should be included

[ostinru]

BTW, should we include Rocky Linux 10 in the test matrix?

I think it should be included

Let's add it in two steps:

1. Reduce number of test groups in automation tests (do not run same test twice).
2. Add Rocky Linux 10 for some (or all) tests