Hi Folks
Recently i have been dealing with high cost on my S3 bucket due to huge number of GET calls being made by my flink streaming job, and would be requiring little help here on the same
context - Using Flink Streaming to read events from Kafka and write it in iceberg+s3(upsert mode enabled) managed by hive metastore
possible solution i have been thinking of is to introduce batching of events over 5mins in order to reduce the no calls made to s3(GET+PUT) to write the data, (we are okay with delay in data by 5mins, as these are not really required real time)
with the above approach i have two options
1. to increase the checkpoint interval to 5mins from 30sec default - so that the write happens every 5min
2. or to use flink windowing technique to aggregate over 5min and then write after 5min, keeping checkpoint same as 30sec
please share your suggestions on the same
Thanks !