Hi all, We are facing backpressure in flink sql jo...
# troubleshooting
a
Hi all, We are facing backpressure in flink sql job from sink and the backpressure only comes from a single task. This causes the checkpoint to fail despite enabling unaligned checkpoints and using debloating buffers. We enabled flamegraph and the task spends most of the time doing rocksdb get and put. The sql job, does a left join over two streams with a parallelism of 20. The total data the topics have is 540Gb for one topic and roughly 60Gb in the second topic. We are running 20 taskmanagers with 1 slot each with each taskmanager having 72G mem and 9 cpu. Can you provide any help on how to go about fixing the pipeline? We are using Flink 1.17.2. The issue is similar to this stackoverflow thread, instead of week it starts facing backpressure as soon as the lag comes down to 4-5%.
Does anyone has any idea how to proceed here?
s
Did you evaluate the possibility of Data skew here?
a
There is no data skew, the no of records each subtask gets from the previous operator are same.
s
Could you provide snapshots of the checkpoints for the job? Healthy and unhealthy