Hi team, I'm using Flink 1.16 and the below error ...
# troubleshooting
x
Hi team, I'm using Flink 1.16 and the below error happens occasionally. I googled and found most similar cases are causing by hardware issues, but we are using cloud storage to save rocksdb files and no issue found after scanning. Does anyone know how this issue happen and how to avoid it?
Copy code
Caused by: org.rocksdb.RocksDBException: block checksum mismatch: stored = 2365559840, computed = 2466618330 in /data/emr/yarn/local/usercache/cpc/appcache/application-xxx/1036861.sst offset 0 size 1661
    at org.rocksdb.Checkpoint.createCheckpoint(Native Method)
    at org.rocksdb.Checkpoint.createCheckpoint(Checkpoint.java:51)
    at org.apache.flink.contrib.streaming.state.snapshot.RocksDBSnapshotStrategyBase.takeDBNativeCheckpoint(RocksDBSnapshotStrategyBase.java:171)
    at org.apache.flink.contrib.streaming.state.snapshot.RocksDBSnapshotStrategyBase.syncPrepareResources(RocksDBSnapshotStrategyBase.java:157)
    at org.apache.flink.contrib.streaming.state.snapshot.RocksDBSnapshotStrategyBase.syncPrepareResources(RocksDBSnapshotStrategyBase.java:77)
    at org.apache.flink.runtime.state.SnapshotStrategyRunner.snapshot(SnapshotStrategyRunner.java:77)
    at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.snapshot(RocksDBKeyedStateBackend.java:593)
    at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:246)
d
ok thats with your RocksDB SST. I think most likely thing is network issues during read/write operations could corrupt data. its also possible you have an incompatible RocksDB and Flink versions. Please check this and also consider checking on RocksDB group as well.
as mentioned below it could also be due to running out of disk space
you might check network logs around the time of these errors.
x
@D. Draco O'Brien Thank you for the replay. But I wonder how come the network issue would cause this exception? IMO, the checksum calculation doesn't require network connection.
d
Your correct that the checksum calc itself does not require network connection however corruption to data during transfer to different nodes or to cloud storage is possible and could lead to discrepancies in data manifesting as checksum error
Let’s also consider if there could be corruption on local storage where flink is running? unexpected system outages, power etc? check instances disk health as well.
File corruption during rocksDB usage can occur. It could be due to HW even in cloud env.
or unexpected interruptions. Basically this is why you want to have full stack logging and tracing available for these periods in time
Check also version of Flink and RocksDB and make sure they are in fact compatible.
It’s possible upgrading or downgrading versions could affect this? Was there any change in the versions? And if so did the errors begin appearing after that or its been occurring with consistent versions.
Also I dont recall your mentioning the RocksDB version? Can you please check what it is?
check flinks release notes for Flink 1.16 for any reference to RocksDB
Refer also to flnks dependency management . Inspect flinks build file pom for specific version of RocksDB that it ships with
x
I'm just using the embedded RocksDB of Flink 1.16, and the version it ships is
6.20.3-ververica-1.0
.
and there's no RocksDB related feature or bug fix according to the release notes of Flink 1.16.
Let me try to reproduce it and have a look at the sst file.