Hi team, We are managing large state in flink stre...
# troubleshooting
v
Hi team, We are managing large state in flink streaming pipeline. every checkpoint is haveing 3TB data. interval is 30 min. I am seeing a pattern where it is breaking after 2 days. Task manager goes to down with OOMkilled. Stuck where to look into it. can it be memory leak, or not enough resource. ( TM: 25 parralelism, 3 slots, 200GB memory, 16 cpu cores) Totally 13 task manger spinning up.
p
Hey there Vishva! Is the checkpoint incremental? Interesting to see this two days breaking patrern. Do you have savepoints configured? Where are you storing your data? How are you managing flink deployments?
v
• Checkpoint is not incremental • Yes, savepoint is taken every 3 hrs interval. and timeout of 1 hr. • We are using rocksDb statebackend, data is moved to s3. • flink k8s operator manages the deployments.
p
Setting a incremental snapshot may help you. Also using s3a may help you as well
v
Thank you @Pedro Mázala, an incremental snapshot would solve the issue, but job recovery would be challenging. Please validate my understanding.
p
No worries at all! Not challenging per se. But it will surely slow the load a little bit the process. How long does it take you pipelines to go from deployed to running today?
m
@Vishva Mahadevan I'm curious your recovery time with such a large volume of data? I've just started to experiment with Flink and so far like the speed that s3a can recovery the job, but haven't yet experimented with such a large volume. Please let me know. Thanks.
v
@Pedro Mázala we have solved the problem, introducing the fullcleanupsnapshot, previously this config was not present. The recovery time reduced from ~2hrs to 10 min, reduced the nodes too.
🎉 1
@Michael Goodnow with s3a protocal. still the recovery time was high. ( restoring from the savepoint). Constructing the rocksdb state was taking more, the network transfer was not a bottle neck.
👍 1