PXF 3.0 Release plan

[ostinru]

Hi, cloudberry-pxf team!

I have created this issue to collect requirements for cloudberry-pxf 3.0. Major version opens a window to introduce backward-incompatible changes. In release 2.2 (#113) I suggested postpone following for 3.0:

• Upgrade Spring Boot to 3.x or even 4.x
• Update hadoop libs to 3.3/3.4
• Upgrade to java 17 in cloudberry-pxf 3.0
• Upgrade postgresql driver version

I would like to add following 'best-effort' changes:

• Fix portion of tests from pxf_regress is broken need fix it #86
• upgrade ORC/Parquet libs
• compare iceberg implementations greengagedb/pxf and Feature: Apache Iceberg lake tables as an extension (datalake_fdw skeleton) cloudberry#1842
• Drop Sequence Files support

Do you have any plans / ideas on what should be included in cloudberry-pxf-3.0?

@tuhaihe, what it deadline for PXF 3.0?

[tuhaihe]

1. For PXF 3.0, any chance to include this proposal: Proposal: Cherry-pick pxf-iceberg feature from GreengageDB/pxf to Apache Cloudberry PXF #126?
2. There is no obvious timeline for the 3.0 release now. I think we should wait until all FIXMEs have been resolved in the Cloudberry main branch before starting the 3.0 release discussion. It may still take 1-2 months.

[ostinru]

1. For PXF 3.0, any chance to include this proposal: Proposal: Cherry-pick pxf-iceberg feature from GreengageDB/pxf to Apache Cloudberry PXF #126?

I think that it is good idea to compare greengageDB/pxf and apache/cloudberry#1842 . I am pretty sure that C-lang version will win for two reasons - it is better integrated with cloudberry and there are more C-lang developers in the community.

Will add it to 'best effort'.

[ostinru]

I run through PXF changelogs (only important features):

Broadcom PXF 7.0

1. PXF external clusters (run PXF on dedicated hosts)
2. Remove support with Greenplum Database 5.x, RHEL 7, Hive 1.x and HBase 1.x.
3. Remove read and write support for Sequence Files for s3, abfss, wasbs, and gs profiles.

Broadcom PXF 7.1

1. PXF now automatically deduplicates files received from HDFS-compatible systems to ensure consistent dataset results. This behavior is controlled by the new pxf.hdfs.fragmenter.deduplicate property in the pxf-site.xml per-server configuration file. When set to true (default), PXF removes duplicate files from the list. When set to false, the list of files is returned as is.
2. ✅ Tomcat 9.0.112
3. PXF supports table statistics gathering using the SQL ANALYZE command for the PXF foreign tables created using the pxf_fdw extension.
4. PXF supports TLS/SSL communication between PXF Greenplum extensions and PXF Services.

Broadcom PXF 8.0

1. Java 17 is a minimal supported version
2. The PXF distribution bundles Hadoop 3.4.2 and Hive 3.1.3 client libraries.
3. The Spring Boot dependency is updated to version 3.5.13.
4. The Parquet dependency is updated to version 1.16.0.
5. The ORC dependency is updated to version 1.9.8.
6. The AWS SDK for Java dependency is updated to version 2.29.52.
7. The Tomcat dependency is updated to version 10.1.54.
8. ⚠️ PXF adds support for reading Iceberg tables with the pxf_fdw extension. PXF also supports predicate pushdown with Iceberg tables.
9. (ha-ha!) Removes support for HBase.

Greengage PXF 6.16

1. Added support for reading and writing Iceberg-format data in Hive Metastore and REST catalogs (starting with Greengage DB 6.31.0)

Arenadata PXF 6.x

1. ADBDEV-6581: Add integration with Vault service
2. ADBDEV-4665: Introduce write backpressure logic in pxf-jdbc
3. Implement reload and reloadAll methods for profiles
4. ADBDEV-5065: Add support for JSON and JSONB types for JDBC
5. ADBDEV-5746: Add an option to limit number of logical segments to execute external table query

[tuhaihe]

After confirming with a Sales Engineer from HashData, HBase is often used during the POC process, but it may not actually be used in production environments. FYI.