Hi Folks Recently i have been dealing with high co...
# troubleshooting
r
Hi Folks Recently i have been dealing with high cost on my S3 bucket due to huge number of GET calls being made by my flink streaming job, and would be requiring little help here on the same context - Using Flink Streaming to read events from Kafka and write it in iceberg+s3(upsert mode enabled) managed by hive metastore possible solution i have been thinking of is to introduce batching of events over 5mins in order to reduce the no calls made to s3(GET+PUT) to write the data, (we are okay with delay in data by 5mins, as these are not really required real time) with the above approach i have two options 1. to increase the checkpoint interval to 5mins from 30sec default - so that the write happens every 5min 2. or to use flink windowing technique to aggregate over 5min and then write after 5min, keeping checkpoint same as 30sec please share your suggestions on the same Thanks !
p
Hey there! I'd say the options are not mutually excluding. You could achieve both. Have you tried using
s3a
as your target? I remember reading something related to pricing and s3 with access and Flink
r
Yes, Thanks Pedro It helped i have configured both checkpointing and windowing logic in the pipeline,
🙌 1
Regarding cost, we are already using s3a Apparently we found that the main issue is the same and common originating due to iceberg small files