Hi all, we are trying to ingest large amounts of data (20TB) from S3 using flink filesystem connector to bootstrap a Hudi table. Data is well partitioned in S3 by time, but we faced OOM issue in flink jobs, so we wanted to change the flink job to ingest the data chunk by chuck (partition by partition here) by some kind of looping instead of all at once. Curious what’s the recommended way to do this in Flink. We have been using Table APIs, but open to other APIs. Thanks.

Hi Eric, I used to run Flink jobs which would read 20TB or a lot more from S3 and I never had any issues with OOMs caused by the S3 code. I also often had jobs fail with OOMs, but the cause for that was for streaming jobs always in my code, and for batch mode I had to tune memory settings a bit to get it working. If there is lots of data even small issues can cause OOMs. My advice would be to look at the memory dumps of the failed Task Managers and see if you can do something about that. Hope this helps, Jörn