Vikas Patil
08/02/2025, 3:49 AMstate.backend.rocksdb.checkpoint.transfer.thread-num: 1 to be able to mitigate this. We are using 1_18. Are there any known issues related to this ?
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f4347307471, pid=1, tid=383
#
# JRE version: OpenJDK Runtime Environment Temurin-17.0.11+9 (17.0.11+9) (build 17.0.11+9)
# Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.11+9 (17.0.11+9, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# C [librocksdbjni-linux64.so+0x307471] rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::DBImpl::GetImplOptions&)+0x31
#
# Core dump will be written. Default location: /opt/flink/core.1
#
# An error report file with more information is saved as:
# /opt/flink/hs_err_pid1.log
[thread 384 also had an error]
#
# If you would like to submit a bug report, please visit:
# <https://github.com/adoptium/adoptium-support/issues>
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
[error occurred during error reporting (), id 0xb, SIGSEGV (0xb) at pc=0x00007f4450534898]Vikas Patil
08/02/2025, 3:49 AMSIGSEGV error
2. Restart without state works without an issues. But it takes a long time for the backlog to drain (we need to start from earliest Kafka offset as State is important and needs to be rebuilt)
3. Restart from a checkpoint works sometime when # of TMs provisioned is less (ideally just 1)
4. Changing thread-count of rocksdb in the flink configuration like this helped for restore:
state.backend.rocksdb.checkpoint.transfer.thread.num: 1
state.backend.rocksdb.thread.num: 1
All this indicates that something was going on with the rocksdb concurrency.Vikas Patil
08/02/2025, 3:56 AMPedro Mázala
08/04/2025, 1:39 PMVikas Patil
08/04/2025, 2:52 PM