Hi!
Is Topic Compaction production ready?
My team an I are trying to integrate Pulsar into our system. A main feature we are counting on is topic compaction. I tried using topic compaction in a very simple scenario, just one producer and one consumer, but when the backlog of the consumers subscription is larger then about 200 (very short) messages my consumer just starts consuming many messages in an infinite loop…
When we searched this on the internet we found similar bug issues on GitHub referenced bellow, especially #21074 which sounds quite similar to what we are experiencing, just that in our scenario the scale of the size and number of messages is much smaller.
I am quite surprised this doesn’t work for me since compaction is such an essential feature, is it possible I’m missing something or maybe not using it correctly? If this is a known issue is there any work around that guarantees topic compaction will work successfully? If not, is this bug supposed to be fixed in some release in the future?
My environment is very standard. I experience consumption redelivery loops only when consuming from the topic as compacted.
https://github.com/apache/pulsar/issues/21074
https://github.com/apache/pulsar/issues/21224