Skip to content

out_s3: add format=parquet with selectable page compression codecs #11885

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rituparnakhaund wants to merge 3 commits into
base: master
Choose a base branch
from
+396 −75
Open

out_s3: add format=parquet with selectable page compression codecs #11885

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ab78b2b
aws: add page-level compression support to parquet writer
May 30, 2026
59f6103
out_s3: add format=parquet option with page-level compression
May 30, 2026
2f9af92
tests: internal: aws_compress: add parquet page-level compression tests
May 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions include/fluent-bit/aws/flb_aws_compress.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,12 @@
#define FLB_AWS_COMPRESS_ZSTD 4
#define FLB_AWS_COMPRESS_SNAPPY 5

/* Parquet page-level compression codecs */
#define FLB_PARQUET_COMPRESSION_NONE 0
#define FLB_PARQUET_COMPRESSION_SNAPPY 1
#define FLB_PARQUET_COMPRESSION_GZIP 2
#define FLB_PARQUET_COMPRESSION_ZSTD 3
Comment on lines +32 to +35
Copy link
Copy Markdown
Contributor

@ShelbyZ ShelbyZ Jun 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am less sure on introducing parquet/arrow specific compression constants. We are migrating to parquet/arrow being formats that can have compression applied on top.


/*
* Get compression type from compression keyword. The return value is used to identify
* what compression option to utilize.
Expand Down Expand Up @@ -63,4 +69,13 @@ int flb_aws_compression_b64_truncate_compress(int compression_type, size_t max_o
void *in_data, size_t in_len,
void **out_data, size_t *out_len);

/*
* Convert JSON data to Apache Parquet format with the specified
* page-level compression codec (FLB_PARQUET_COMPRESSION_*).
* Returns 0 on success, -1 on failure.
*/
int out_s3_compress_parquet(void *json, size_t size,
void **out_buf, size_t *out_size,
int parquet_compression);
Comment on lines +77 to +79
Copy link
Copy Markdown
Contributor

@ShelbyZ ShelbyZ Jun 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we accept that we can apply compression on top of the existing formats then this method may not be needed long term


#endif
Loading
Loading