Hello, I'm going fix my Flink job. It's checkpoint...
# troubleshooting
l
Hello, I'm going fix my Flink job. It's checkpoint usually fail at sink step. Sometime it's work, sometime it's failed. I increased heap memory so there's no OOM error. But I just wonder is there any problem with Kafka Sink? It's look like kafka sink receive message too slowly (I consumed data from Kafka Sink topic and diff timestamp) even I just process data within a sliding windows 1 hour. Not sure if these config working
Copy code
properties.setProperty("auto.offset.reset", "latest");
properties.setProperty("enable.auto.commit", "true"); // Commit offset automatically
v
In my opinion, you need to look multiple places to resolve this issue, you have sliding window which create a state, that state has to be written to the disk when checkpoint happens. You check point timeout is defaulted to 10 minutes, you might want to reconsider it execution.checkpointing.timeout. Also take a look what disk system Flink is writing checkpoint to. Does it have enough OPS for this tasks. For application point of view think if you can process window in parallel by keying your stream and may by writing you data in partitions if there is a problem with a Kafka sink. Also you might want to switch on Flame graph to see where actual waits are happening.
👍 1
l
@Valery Maksimenko thank you for your feedback. There’s 2 sink operation: 1 finished 100% with more than 1gb. Other fail at step 4/10 and pending duration 5 minutes, then timeout. I’m using gcs to store checkpoint.