Hello ✋ , I deleted a realtime table from my Pinot cluster, but I can still see the consumer group with "empty name" created by pinot on the topic still keeping track of the consumer lags. See image below: Since pinot is using lowlevel consumers, there is actually no real concept of consumer group, and since the consumer group name is "blank" I am not able to delete it. While this may not affect any new realtime table created to consume this topic, is there no way to ensure the consumer is removed from the topic when the realtime table is removed?

@Neha Pawar @Subbu Subramaniam

@Trust Okoroego where do you see the consumer group with empty name ? (I assume you are using Kafka). If you never created an HLC table, then the consumer group part should not be there. Not sure where you are trying to delete it from. Zookeeper? Can you specify the full path name where it exists?

@Subbu Subramaniam I see the consumer group with empty name in kafka. I am using AKHQ to visualize the topics and the related consumers(the screen shot). If I also use kafka-consumer-group.sh tool to check the list of consumers on the topic, I still see it.

Glad that the table looks OK. So, LLC tables also create stream consumers (in your case, kafka consumers) to get metadata about the stream (e.g. get the number of partitions, the offset in each of them, etc.). We get this information every once in while from pinot controllers. I don't think we get metadata from pinot servers at all, although there could be some corner cases where we do. I need to check again. Can you verify a couple of things: (1) Are the segments for the (realtime) table LLC segments, or do you see HLC segments as well? (segment name that follows the pattern

tableName__partitionNumber__sequenceNumer__timestamp

are LLC segmemts). (2) Are the consumers in the consumer group that you see from pinot-controller hosts?

@Subbu Subramaniam This is one of the segments. tb_deliveries__0__0__20211117T1643Z Regarding the second question, I am not sure how to check this. But I am sure that the empty consumer is from Pinot, as I have them in all the tables I am streaming to Pinot. I am running AWS MSK kafka 2.6 and Pinot version 0.8

if all segments follow this naming pattern, you are good. If you are checking kafka on the service side, I am assuming there is some way to check which hosts are in the consumer group. If there is no way, then I am not sure what to do. If there is anything you can do to debug as to when and who created those consumer groups? Is there one group for all tables, or one per table added? @Neha Pawar do you think there is a metadata fetcher leak in the kafka 2.6 plugin? Can you verify?

there is no 2.6 plugin. the plugin should still be kafka-2.0, which hasnt been touched in a long time. i think the 2.6 version here is of thei kafka deployment

@Neha Pawar the 2.6 is the Kafka MSK version.

you should not need this parameter for llc, so try removing it? @Neha Pawar may know better