Hello :hand: , I deleted a realtime table from my ...
# troubleshooting
t
Hello ✋ , I deleted a realtime table from my Pinot cluster, but I can still see the consumer group with "empty name" created by pinot on the topic still keeping track of the consumer lags. See image below: Since pinot is using lowlevel consumers, there is actually no real concept of consumer group, and since the consumer group name is "blank" I am not able to delete it. While this may not affect any new realtime table created to consume this topic, is there no way to ensure the consumer is removed from the topic when the realtime table is removed?
m
@Neha Pawar @Subbu Subramaniam
s
@Trust Okoroego where do you see the consumer group with empty name ? (I assume you are using Kafka). If you never created an HLC table, then the consumer group part should not be there. Not sure where you are trying to delete it from. Zookeeper? Can you specify the full path name where it exists?
t
@Subbu Subramaniam I see the consumer group with empty name in kafka. I am using AKHQ to visualize the topics and the related consumers(the screen shot). If I also use kafka-consumer-group.sh tool to check the list of consumers on the topic, I still see it.
is shows like this: consumer-group1 consumer-group2 consummer-froup3 the space between 2 and 3 is the blanck consumer group
another strange thing I just observed now after creating a new table from a different pinot cluster is that the consumer lags in the topic were cleaned up by the new real-time table I just deployed to another pinot cluster
s
Glad that the table looks OK. So, LLC tables also create stream consumers (in your case, kafka consumers) to get metadata about the stream (e.g. get the number of partitions, the offset in each of them, etc.). We get this information every once in while from pinot controllers. I don't think we get metadata from pinot servers at all, although there could be some corner cases where we do. I need to check again. Can you verify a couple of things: (1) Are the segments for the (realtime) table LLC segments, or do you see HLC segments as well? (segment name that follows the pattern
tableName__partitionNumber__sequenceNumer__timestamp
are LLC segmemts). (2) Are the consumers in the consumer group that you see from pinot-controller hosts?
I am not sure what the blank consumer group is about. Also, please confirm the version of kafka plugin you are using.
t
@Subbu Subramaniam This is one of the segments. tb_deliveries__0__0__20211117T1643Z Regarding the second question, I am not sure how to check this. But I am sure that the empty consumer is from Pinot, as I have them in all the tables I am streaming to Pinot. I am running AWS MSK kafka 2.6 and Pinot version 0.8
s
if all segments follow this naming pattern, you are good. If you are checking kafka on the service side, I am assuming there is some way to check which hosts are in the consumer group. If there is no way, then I am not sure what to do. If there is anything you can do to debug as to when and who created those consumer groups? Is there one group for all tables, or one per table added? @Neha Pawar do you think there is a metadata fetcher leak in the kafka 2.6 plugin? Can you verify?
Other than this, I seem to have hit a wall here.
n
there is no 2.6 plugin. the plugin should still be kafka-2.0, which hasnt been touched in a long time. i think the 2.6 version here is of thei kafka deployment
t
@Neha Pawar the 2.6 is the Kafka MSK version.
@Subbu Subramaniam Pinot creates the empty consumer group. I am sure of this. I had to delete the kafka topic after deleting the pinot real time table. When I recreated the topic, the empty consumer group was gone as expected. Once I register a new real-time table in pinot it comes back.
@Subbu Subramaniam I setup two real-time table consuming the same topic, and I can confirm that only one empty consumer group exist on the Kafka topic for both. Maybe this will not affect the offset of the individual tables if the offset is maintained in pinot for each of the tables, but the concept of one empty consumer group is a bit confusing from a monitoring perspective. Any reason why the implementation does not respect this parameter?
Copy code
"stream.kafka.consumer.prop.group.id":
s
you should not need this parameter for llc, so try removing it? @Neha Pawar may know better