Hi, we have one operator that have around 60 GB of...
# troubleshooting
g
Hi, we have one operator that have around 60 GB of data. One of the subtasks takes considerably longer than others. E.g. 34 mins vs 2-2.5 mins. in "sync" phase. There is no significant skew. The one that takes 34 mins have 3.05 GB while others have 1.9 GB - 2.98 GB. Any ideas what might be the cause? Thread dump just states the code is in native part:
Copy code
at app//org.rocksdb.Checkpoint.createCheckpoint(Native Method)
	at app//org.rocksdb.Checkpoint.createCheckpoint(Checkpoint.java:50)
	at app//org.apache.flink.state.rocksdb.snapshot.RocksDBSnapshotStrategyBase.takeDBNativeCheckpoint(RocksDBSnapshotStrategyBase.java:174)
Additionally seems IO and Network goes down during the process. Checkpoint starts around 10:51 and ends 11:12 On IO/Network charts (5m moving average) we can see that peak traffic is around beginning of the checkpoint and then it goes down to 0 in case of network and significantly down in case of IO.
No significant CPU/Mem usage