Hi We have been using KoP for a while and it has been pretty stable. But recently we often have backlogs stuck and not moving. And then we had to restart some of the brokers to get over the issue. We discovered a few logs from those brokers dumped below.

org.apache.bookkeeper.mledger.ManagedLedgerException$CursorNotFoundException: ManagedCursor not found: <kop-consumer-cursor-persistent://kafka/archiving/useractions.json-partition-2-54292994-105331-8af2e1e053>
2024-04-05T20:56:33,844+0000 [pulsar-ph-kafka-57-2] ERROR io.streamnative.pulsar.handlers.kop.KafkaRequestHandler - Caught error in handler, closing channel
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
2024-04-05T20:56:33,844+0000 [pulsar-ph-kafka-57-2] WARN  io.streamnative.pulsar.handlers.kop.KafkaTopicConsumerManager - [[id: 0x0dc8a7e7, L:/172.31.202.83:9092 ! R:/172.31.102.64:49270]] Error deleting cursor <kop-consumer-cursor-persistent://kafka/archiving/useractions.json-partition-3-54293005-103622-18337c3188> for topic <persistent://kafka/archiving/useractions.json-partition-3> for reason: TopicConsumerManager close but cursor is still outstanding.
org.apache.bookkeeper.mledger.ManagedLedgerException$CursorNotFoundException: ManagedCursor not found: <kop-consumer-cursor-persistent://kafka/archiving/useractions.json-partition-3-54293005-103622-18337c3188>
2024-04-05T20:56:33,844+0000 [pulsar-ph-kafka-57-2] WARN  io.streamnative.pulsar.handlers.kop.KafkaTopicConsumerManager - [[id: 0x0dc8a7e7, L:/172.31.202.83:9092 ! R:/172.31.102.64:49270]] Error deleting cursor <kop-consumer-cursor-persistent://kafka/archiving/useractions.json-partition-2-54292994-105019-d8f0fd1536> for topic <persistent://kafka/archiving/useractions.json-partition-2> for reason: TopicConsumerManager close but cursor is still outstanding.
org.apache.bookkeeper.mledger.ManagedLedgerException$CursorNotFoundException: ManagedCursor not found: <kop-consumer-cursor-persistent://kafka/archiving/useractions.json-partition-2-54292994-105019-d8f0fd1536>
2024-04-05T20:56:33,844+0000 [pulsar-ph-kafka-57-2] WARN  io.streamnative.pulsar.handlers.kop.KafkaTopicConsumerManager - [[id: 0x0dc8a7e7, L:/172.31.202.83:9092 ! R:/172.31.102.64:49270]] Error deleting cursor <kop-consumer-cursor-persistent://kafka/archiving/useractions.json-partition-6-54293135-93039-fd084b5e53> for topic <persistent://kafka/archiving/useractions.json-partition-6> for reason: TopicConsumerManager close but cursor is still outstanding.
org.apache.bookkeeper.mledger.ManagedLedgerException$CursorNotFoundException: ManagedCursor not found: <kop-consumer-cursor-persistent://kafka/archiving/useractions.json-partition-6-54293135-93039-fd084b5e53>
2024-04-05T20:56:33,844+0000 [pulsar-ph-kafka-57-2] WARN  io.streamnative.pulsar.handlers.kop.KafkaTopicConsumerManager - [[id: 0x0dc8a7e7, L:/172.31.202.83:9092 ! R:/172.31.102.64:49270]] Error deleting cursor <kop-consumer-cursor-persistent://kafka/archiving/useractions.json-partition-20-54293101-95875-881b89ac5e> for topic <persistent://kafka/archiving/useractions.json-partition-20> for reason: TopicConsumerManager close but cursor is still outstanding.

Could anyone help point out what might have been the issues?

Hey Yang, this issue may be related to our most recent update. Your team should have received an email, but let me forward you the details.

Hi Ken, our current version of pulsar is 2.10.0.7, so I don't think it's related to the latest version

Hi Yang, sharing some KSN capabilities and latest 3.2 SN release have advanced features available now https://www.youtube.com/playlist?list=PL7-BmxsE3q4XrdXD9EvdCoX4qYe_GQfYK