Hi everyone,
I'm facing a critical issue with Apache Pinot realtime ingestion from Pulsar. When my Pinot table crashes or restarts, the consumer subscription gets deleted from Pulsar, causing data loss.
The Problem:
Pinot creates auto-generated subscription names (like reader-494b86923d)
When Pinot crashes, this subscription gets deleted from Pulsar
On restart, Pinot generates a NEW subscription name (reader-383a75812c)
Since the old subscription is gone, Pinot starts from latest instead of resuming where it left off
This causes data loss during the downtime period
What I've tried:
Setting stream.pulsar.consumer.prop.subscriptionName in table config (seems to be ignored)
Configuring Pulsar retention and subscription expiration policies
Using auto.offset.reset=smallest but it doesn't help when subscription is deleted
Questions:
How can I prevent Pinot's consumer subscriptions from being deleted when it crashes?
Does Pinot actually respect the subscriptionName property or does it always auto-generate?
What's the recommended pattern to handle consumer failures without data loss?
Any help would be greatly appreciated! This is causing production data gaps that are hard to recover from.
Thanks!