Hey Everyone, We’re using Flink SQL to save data to S3, by using ‘filesystem’ connector. We’re added the partitions in the table as well. The issue we see is there are so many small files being created in S3. We are creating 1 parquet file per day in S3. Sink table query:

CREATE TABLE sink_table_s3 (
  event_id STRING NOT NULL,
  event_type STRING NOT NULL,
  event_name STRING NOT NULL,
  eventId STRING NOT NULL,
  eventName STRING NOT NULL,
  `date` STRING
) PARTITIONED BY (eventId, eventName, `date`) WITH (
  'connector' = 'filesystem', 
  'path' = '<path>', 
  'format' = 'parquet',
  'auto-compaction' = 'true'
);

Insert query:

INSERT INTO sink_table_s3 
SELECT event_id, event_type, event_name, 
DATE_FORMAT(proc_time, 'yyyy-MM-dd') AS `date`, event_id AS eventId, event_name AS eventName
FROM source_table;

I’m adding eventId, eventTime just to make sure those columns are also available in the Parquet file in S3. How can we avoid small files being created?

You can enable compaction to resolve this. See https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/filesystem/#file-compaction

We are creating 1 parquet file per day in S3.

PARTITIONED BY (eventId, eventName,

date

)

So it actually looks like a file per eventId, eventName and date? This seems like a lot of files.

I already enabled compaction, by setting ‘auto-compaction’ to true. Checkpointing is also enabled for every 5 mins.

So then it will take 5 mins before compaction is happening, are checking after that time or in between?

@sap1ens - In our case, One eventId and one eventName can have lot of data and thats why multiple files are being created. But the size of the the files is very less ~ 5 KB.

Partitioning by date doesn’t mean that files will be compacted by date

Okay, then how do I fix this? As it is a bulk format type, does rolling policy help?

It’s pretty hard to “fix” without using a lakehouse format like Iceberg, Hudi or Delta.

Thanks for the info, I’ll check this