Hello, My Flink jobs fails with the below exception, let me know if you have any idea. It fails when the throughput is least in the day (night time for us) Flink version: 1.16.1, Kafka In and Kafka Out pipeline org.apache.kafka.clients.producer.internals.Sender org.apache.kafka.clients.producer.internals.Sender213353.951 [kafka-producer-network-thread | producer-2] WARN o.a.k.c.producer.internals.Sender - [Producer clientId=producer-2] Received invalid metadata error in produce request on partition rill_ads_mweb_events-0 due to org.apache.kafka.common.errors.NetworkException: Disconnected from node 4. Going to request metadata update now org.apache.kafka.clients.producer.internals.Sender org.apache.kafka.clients.producer.internals.Sender213426.778 [Source: Kafka Source -> *anonymous_datastream_source$221*[1306] -> Calc[1307] -> Sink: Unnamed (3/4)#1] WARN o.a.flink.runtime.taskmanager.Task - Source: Kafka Source -> *anonymous_datastream_source$221*[1306] -> Calc[1307] -> Sink: Unnamed (3/4)#1 (ca6c10fbe5e342fcee9fd7a49aa6450c_cbc357ccb763df2852fee8c4fc7d55f2_2_1) switched from RUNNING to FAILED with failure cause: java.lang.Exception: Could not perform checkpoint 13711 for operator Source: Kafka Source -> *anonymous_datastream_source$221*[1306] -> Calc[1307] -> Sink: Unnamed (3/4)#1. at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointAsyncInMailbox(StreamTask.java:1166) at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$triggerCheckpointAsync$12(StreamTask.java:1113) at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:367) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:352) at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229) at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:831) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:780) at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935) at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.flink.runtime.checkpoint.CheckpointException: Could not complete snapshot 13711 for operator Source: Kafka Source -> *anonymous_datastream_source$221*[1306] -> Calc[1307] ->Sink: Unnamed (3/4)#1. Failure reason: Checkpoint was declined. at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:269) at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:173) at org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:345) at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.checkpointStreamOperator(RegularOperatorChain.java:228) at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.buildOperatorSnapshotFutures(RegularOperatorChain.java:213) at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.snapshotState(RegularOperatorChain.java:192) at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.takeSnapshotSync(SubtaskCheckpointCoordinatorImpl.java:726) at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:363) at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$13(StreamTask.java:1281) at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) at org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:1269) at org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointAsyncInMailbox(StreamTask.java:1154) ... 14 more Caused by: org.apache.flink.streaming.connectors.kafka.FlinkKafkaException: Failed to send data to Kafka: Disconnected from node 4 at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.checkErroneous(FlinkKafkaProducer.java:1428) at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.flush(FlinkKafkaProducer.java:1116)

I’ve usually seen this when an update happened on the Kafka broker or for another reason brokers became unavailable.