Sharmin Choksey
05/03/2024, 8:39 AM29.0.1 on a k8s cluster deployed using the druid-operator. We have recently switched to cloudnative-pg as the metadatadb and have noticed a significant difference in the manner in which the Kakfa Indexing Tasks terminate during a rolling restart of the middlemanager. The Tasks disappear in reality, but the indexer API continues to report them to be in a RUNNING state. Eventually, they report a failed state with the exception - org.apache.druid.java.util.common.ISE: Previous sequenceNumber [NNN] is no longer available for partition. In our prior cycles of testing upgrades/rollout restarts, the behavior observed was that the live ingestion would halt at the start of the middle manager upgrade or rollout restart, the monotonic count would actually drop, but then at the end of the router restart, the ingestion would resume and lag would finally catch up and query counts converge.
However, with cloudnative-pg as the metadatadb, ingestion never recovers after a rollout restart and ends up with LOST_CONNECTION_STREAM, unable to resume from the last saved offset.Sharmin Choksey
05/03/2024, 8:48 AMrestoreTasksOnRestart: True