Hey, I'm facing problem with checkpoints. All checkpoints take up to 3-4s, but sometimes checkpoint takes much more and fails. Any ideas how to debug that?

can you share your confi tab too please?

Increased timeouts to test

state.backend.incremental: 'true'
execution.checkpointing.unaligned: 'true'

Have you tried these settings?

taskmanager.numberOfTaskSlots: "2"
resourcemanager.previous-worker.recovery.timeout: "5 min"
jobmanager.rpc.port: "6123"
taskmanager.rpc.port: "6123"
restart-strategy: "exponential-delay"
restart-strategy.exponential-delay.initial-backoff: "2 min"
restart-strategy.exponential-delay.max-backoff: "5 min"
restart-strategy.exponential-delay.backoff-multiplier: "2.0"
restart-strategy.exponential-delay.reset-backoff-threshold: "10 min"
restart-strategy.exponential-delay.jitter-factor: "0.1"
state.backend: "rocksdb"
state.backend.incremental: "true"
state.backend.rocksdb.checkpoint.transfer.thread.num: "2"
akka.client.timeout: "300s"
execution.checkpointing.interval: 300 s
execution.checkpointing.timeout: 2400 s
akka.ask.timeout: "300s"
heartbeat.timeout: "120000"
s3.path.style.access: "true"
s3.endpoint: "{S3_ENDPOINT}"
state.savepoints.dir: "<s3://checkpoints/flink-app/savepoints/>"
state.checkpoints.dir: "<s3://checkpoints/flink-app/checkpoints/>"

job:
  jarURI: local:///opt/flink/usrlib/app-1.0-SNAPSHOT.jar
  upgradeMode: savepoint
  entryClass: com.bla.bla.Application
  parallelism: 2
  allowNonRestoredState: true

Ok. I'm not super sure how to debug this. But can you open the details of the failed checkpoint? It should tell you at which operator it failed.

there are plenty of them

Typically a failed checkpoint means there was a lot of backpressure and it couldn’t process all records in time. I’d check backpressure and other relevant metrics first.

@sap1ens is there any way to successfully checkpoint even under high pressure?

You can increase the timeout to start 🙂