Hi all, we have a Flink cluster running in session...
# troubleshootingg
Guruguha Marur Sreenivasa
02/16/2023, 5:29 AMHi all, we have a Flink cluster running in session mode and has about 25 jobs running on the cluster with 80 pods. I have a hunch that Flink doesn't commit offsets correctly at regular intervals causing the consumer lag to shoot up for sometime and rapidly coming down. Is there a setting that I can look at or is there a metric which shows at what times flink consumer committed Kafka offsets?
d
David Anderson
02/16/2023, 5:33 AMIf Flink is checkpointing, then the offsets are committed when the checkpoint is complete.
g
Guruguha Marur Sreenivasa
02/16/2023, 5:34 AMYes, I thought so too - but checkpoints are failing for some reason at random times. Will it cause performance issues if we checkpoint at a higher rate? We're currently checkpointing every 30 seconds.
d
David Anderson
02/16/2023, 5:41 AMDo checkpoints fail because they timeout ?
g
Guruguha Marur Sreenivasa
02/16/2023, 5:41 AMYes
d
David Anderson
02/16/2023, 5:41 AMWhat version of Flink are you using ?
g
Guruguha Marur Sreenivasa
02/16/2023, 5:42 AM1.14.3
d
David Anderson
02/16/2023, 5:43 AMWhen checkpoints fail, does the job immediately fail and restart ?
g
Guruguha Marur Sreenivasa
02/16/2023, 5:44 AMThey restart. The checkpoint eventually is saved and job continues to run.
d
David Anderson
02/16/2023, 5:48 AMYou might try increasing the checkpoint timeout, or increase
execution.checkpointing.tolerable-failed-checkpointsDavid Anderson
02/16/2023, 5:52 AMI’m guessing the consumer lag shoots up every time the job fails, and catches up again after the restart has re-ingested the records since the last checkpoint.
Given you are experiencing checkpoint failures, I wouldn’t suggest increasing the checkpoint frequency.
It could be interesting for you to try the changelog state backend with more frequent checkpoints, but you should probably upgrade to 1.16 before thinking about that.
g
Guruguha Marur Sreenivasa
02/16/2023, 5:57 AMOh ok, we’re currently using HashMap State Backend
d
David Anderson
02/16/2023, 5:58 AMInteresting. How large are the checkpoints?
g
Guruguha Marur Sreenivasa
02/16/2023, 4:56 PMAbout
500KBGuruguha Marur Sreenivasa
02/17/2023, 8:04 PM@David Anderson we're seeing this spiky nature of offsets commits from Flink
It is causing our alerts to falsely trigger and create an incedent.
d
David Anderson
02/17/2023, 8:43 PMAre the spikes correlated with checkpoint failures?
5 Views