Hey all, I would like to partition my data into compressed parquet files. Is it possible to add compression codec (like snappy or gzip ) for parquet files in Pyflink? I couldn't find anything in the documentation.

file_sink = FileSink.for_bulk_format(base_path=output_path, \
                writer_factory=ParquetBulkWriters.for_row_type(
                                                    row_type,
                                                    hadoop_config=Configuration(),
                                                    utc_timestamp=True,
                                                ))  \
                .with_bucket_assigner(BucketAssigner.date_time_bucket_assigner(format_str="yyyy/MM/dd/HH", timezone_id="UTC")) \
                .with_output_file_config( 
                    OutputFileConfig.builder() 
                    .with_part_prefix("prefix") 
                    .with_part_suffix(".parquet") 
                    .build()) \
                .with_rolling_policy(RollingPolicy.on_checkpoint_rolling_policy()) \
                .build()

@piby 180 I have checked that it’s true that there is no documentation on this, however, it should be supported. You can use it as following:

conf = Configuration()
conf.set_string("parquet.compression", "SNAPPY")
ParquetBulkWriters.for_row_type(row_type, hadoop_config=conf, utc_timestamp=True))

Lovely. Thank you