Does anyone have good pointers on how to debug iss...
# randomo
Odin Wang
09/14/2023, 7:53 PMDoes anyone have good pointers on how to debug issues with checkpointing not progressing? I'm using Table API w/ Iceberg source + DataStream API on AWS KDA / Managed Flink.
j
Jeremy Ber
09/14/2023, 7:57 PMWhich piece of checkpointing is delayed?
https://docs.aws.amazon.com/managed-flink/latest/java/troubleshooting-checkpoints.html
o
Odin Wang
09/15/2023, 12:11 AMThanks! The checkpointing was stuck and never made any progress, all the stats in the Flink Web UI dashboard shows 0, e.g.
Odin Wang
09/15/2023, 12:11 AMThe doc is helpful, through the thread dump I was able to figure out that the checkpointing seems to be waiting for lock forever with
Copy code
"Channel state writer <operator name> (73/128)#0" Id=128 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5b5fd993
at java.base@11.0.18/jdk.internal.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5b5fd993
at java.base@11.0.18/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11.0.18/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11.0.18/java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:483)
at java.base@11.0.18/java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:671)
at app//org.apache.flink.runtime.checkpoint.channel.ChannelStateWriteRequestExecutorImpl.loop(ChannelStateWriteRequestExecutorImpl.java:96)
at app//org.apache.flink.runtime.checkpoint.channel.ChannelStateWriteRequestExecutorImpl.run(ChannelStateWriteRequestExecutorImpl.java:75)
at app//org.apache.flink.runtime.checkpoint.channel.ChannelStateWriteRequestExecutorImpl$$Lambda$956/0x0000000800b8a440.run(Unknown Source)
at java.base@11.0.18/java.lang.Thread.run(Thread.java:829)j
Jeremy Ber
09/15/2023, 1:16 PMIs it some kind of networking issue?
o
Odin Wang
11/08/2023, 11:39 PMI'm watching your talk at Flink Forward on this topic and realized that I never replied to this thread.
I was able to get rid of the issue by avoiding batch read (i.e. use streaming read instead) on Iceberg tables w/ the old Iceberg source implementation (not the FLIP27 source). Batch read works fine for the FLIP27 source implementation in Iceberg but we found it to be less reliable. I didn't try to understand what happened under the hood though
j
Jeremy Ber
11/09/2023, 1:05 AMOh I see very interesting! Thanks for following up on it.
2 Views