When running a Flink job with checkpoints, sometim...
# troubleshootingj
Jotham Wong
08/17/2024, 2:16 PMWhen running a Flink job with checkpoints, sometimes a node just refuses to send the acknowledgement and the checkpoint stalls for a long time. Does anyone know any possible causes?
d
D. Draco O'Brien
08/17/2024, 4:04 PMHave you checked resources usage levels? memory availability etc? network etc? Are you using incremental checkpointing and what is the interval?
D. Draco O'Brien
08/17/2024, 4:06 PM1. Check network, firewalls etc. network issues could easily cause this. Is other traffic getting through?
D. Draco O'Brien
08/17/2024, 4:07 PM2. Resource constraints, CPU/Memory Disk IO
D. Draco O'Brien
08/17/2024, 4:07 PM3. Check health of storage medium used for checkpointing.
D. Draco O'Brien
08/17/2024, 4:08 PM4. check backpressure.
D. Draco O'Brien
08/17/2024, 4:09 PMAll this info comes from various sources logging, prometheus, Flink UI metrics etc. So you should familiarize yourself as much as possible with this.