Hi all. We've got a Flink job that uses state and ...
# troubleshooting
v
Hi all. We've got a Flink job that uses state and works well ~2-3 days. After that period, suddenly it starts to constantly fail at checkpoint with no chance to recover it. We've found the exception thrown by the process which leads to TM pod killed by K8s:
Copy code
2025-09-04 08:20:53,380 INFO  org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer [] - Kryo serializer scala extensions are not available.
Exception in thread "Thread-57" java.lang.IllegalArgumentException: classLoader cannot be null.
	at com.esotericsoftware.kryo.Kryo.setClassLoader(Kryo.java:975)
	at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.checkKryoInitialized(KryoSerializer.java:553)
	at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:394)
	at org.apache.flink.api.common.typeutils.CompositeSerializer.deserialize(CompositeSerializer.java:156)
	at org.apache.flink.contrib.streaming.state.ttl.RocksDbTtlCompactFiltersManager$ListElementFilter.nextElementLastAccessTimestamp(RocksDbTtlCompactFiltersManager.java:205)
	at org.apache.flink.contrib.streaming.state.ttl.RocksDbTtlCompactFiltersManager$ListElementFilter.nextUnexpiredOffset(RocksDbTtlCompactFiltersManager.java:191)
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000ffff9440bb90, pid=1, tid=391
#
# JRE version: OpenJDK Runtime Environment Temurin-11.0.26+4 (11.0.26+4) (build 11.0.26+4)
# Java VM: OpenJDK 64-Bit Server VM Temurin-11.0.26+4 (11.0.26+4, mixed mode, sharing, tiered, compressed oops, serial gc, linux-aarch64)
# Problematic frame:
# V  [libjvm.so+0x5fbb90]  Exceptions::_throw_oop(Thread*, char const*, int, oopDesc*)+0x180
#
# Core dump will be written. Default location: /opt/flink/core.1
#
# An error report file with more information is saved as:
# /opt/flink/hs_err_pid1.log
#
# If you would like to submit a bug report, please visit:
#   <https://github.com/adoptium/adoptium-support/issues>
#

[error occurred during error reporting (), id 0x5, SIGTRAP (0x5) at pc=0x0000ffff958971ec]
Looks like something that happens during the RocksDb compaction... 😞 The FLink version - 1.18.1