Does the Flink s3 sink supports writing to multiple files? My input stream contains data from different customers, I want to partition the data based on

customerId

and then save the data to different paths. Is that something the Flink s3 can do? The example in the documentation is very terse. https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/s3/ In addition, as my input stream is unbounded, is the s3 sink going to write everything into a single s3 object or it will automatically save data to a new object based on some criteria?

1. Yes, you can define your own bucket assigner per record. Sample: https://ideone.com/Yu3Zse

import org.apache.flink.core.io.SimpleVersionedSerializer;
import org.apache.flink.streaming.api.functions.sink.filesystem.BucketAssigner;
import org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.SimpleVersionedStringSerializer;

public class FileBucketAssigner implements BucketAssigner<InputRecordPojo, String> {

  @Override
  public String getBucketId(InputRecordPojo record, Context context) {
	return extractCustomerID(record);
  }

  @Override
  public SimpleVersionedSerializer<String> getSerializer() {
    return SimpleVersionedStringSerializer.INSTANCE;
  }
}

FileSink<InputRecordPojo> sink =
        FileSink.forRowFormat(...)
            .withBucketAssigner(new FileBucketAssigner())
            .build();

2. File sinks have rolling policy, so they won't write all data to a single object, but rather start writing to a new file when certain conditions are met, you can read more about it here.

You can do it, I would recommend reading about Iceberg and use it instead, it has out of the box partitioning and other capabilities such as querying etc..