Hey, I am new to Flink and would like to store messages from a kafka topic to an S3 bucket partitioned by timestamp. I have created Kafka Source Table and S3 Sink Table and am currently facing issues with aws credentials. Flink app is deployed on kubernetes with a service account attached to the pod. But the application is not picking up IRSA role to connect to S3. I have tried flink-s3-fs-hadoop-1.61.1.jar which raised Access Denied Error because it didn’t picked up instance role and I have not provided S3 access keys. I have tried flink-s3-presto-1.61.1.jar which gave me the error UnsupportedOperationException : This s3 file system implementation does not support recoverable writers. Could anyone give me a hint what should I do to make it work? I don’t want to use AWS keys as it is against our company’s policy

The Presto implementation is meant for snapshots (checkpointing and savepointing). You need to use the Hadoop implementation for reading/writing files

Thanks for confirming. Does hadoop implementation work with IRSA roles/ aws iam roles?

It should