Hi folks! I’m currently facing an issue with the <...
# general
d
Hi folks! I’m currently facing an issue with the pulsar-io-lakehouse sink connector in my streaming data pipeline on Pulsar, deployed on a Kubernetes cluster. Here’s the scenario: Setup: I have set up a streaming data pipeline on top of Pulsar using the pulsar-io-lakehouse sink connector. I deploy a sink connector using Pulsar Admin REST API. Problem: Whenever a pod for the pulsar-broker goes down for any reason and is subsequently recreated, the sink connector stops working without throwing any exceptions. Observation: As a result, the backlog for the subscription associated with the sink connector starts increasing indefinitely until I manually delete and recreate the sink connector. Expected Behaviour: I expect the sink connector to gracefully handle pod restarts and resume normal operation without manual intervention. Has anyone else encountered a similar issue or have any insights into what might be causing this behaviour? Any suggestions or guidance on how to troubleshoot and resolve this issue would be greatly appreciated. Thanks in advance for your help!
l
please file an issue in https://github.com/streamnative/pulsar-io-lakehouse/issues . It's also worth upgrading your sink connector version to one of the recent versions, https://github.com/streamnative/pulsar-io-lakehouse/releases .
It's also worth ensuring that you are using a supported and recent Pulsar version.