Hey All, we had a rocksdb issue where it looks lik...
# troubleshooting
v
Hey All, we had a rocksdb issue where it looks like there was a thread contention issue. We had to reduce the number of threads using
state.backend.rocksdb.checkpoint.transfer.thread-num: 1
to be able to mitigate this. We are using
1_18
. Are there any known issues related to this ?
Copy code
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f4347307471, pid=1, tid=383
#
# JRE version: OpenJDK Runtime Environment Temurin-17.0.11+9 (17.0.11+9) (build 17.0.11+9)
# Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.11+9 (17.0.11+9, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# C  [librocksdbjni-linux64.so+0x307471]  rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::DBImpl::GetImplOptions&)+0x31
#
# Core dump will be written. Default location: /opt/flink/core.1
#
# An error report file with more information is saved as:
# /opt/flink/hs_err_pid1.log
[thread 384 also had an error]
#
# If you would like to submit a bug report, please visit:
#   <https://github.com/adoptium/adoptium-support/issues>
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

[error occurred during error reporting (), id 0xb, SIGSEGV (0xb) at pc=0x00007f4450534898]
Some more context here is that, we tried these things and what fixed is somewhat surprising: 1. Restart from a previous checkpoint does not work TMs crashloop with
SIGSEGV
error 2. Restart without state works without an issues. But it takes a long time for the backlog to drain (we need to start from earliest Kafka offset as State is important and needs to be rebuilt) 3. Restart from a checkpoint works sometime when # of TMs provisioned is less (ideally just 1) 4. Changing thread-count of rocksdb in the flink configuration like this helped for restore: state.backend.rocksdb.checkpoint.transfer.thread.num: 1 state.backend.rocksdb.thread.num: 1 All this indicates that something was going on with the rocksdb concurrency.
cc: @Pedro Mázala Any ideas on this one ?
p
Hello there, Vikas! I did not encountered myself an issue like this one. Have you checked the open FLIPs?
v
There is this unresolved flip : https://issues.apache.org/jira/browse/FLINK-6761 which potentially could be the issue. But we do not have large values in a single key. So not really sure