Apache Pinot

*How to reliably confirm Kafka consumers are fully stopped before deleting Kafka topics after a Pinot table deletion in 1.5.1?*
We are performing a clean-slate reset of our Pinot setup by deleting all tables (schema + config) and then deleting the corresponding Kafka topics. However, we observed that some Kafka topics were being *recreated automatically* because the Kafka consumers for those tables had not fully stopped by the time we deleted the topics.
We tried polling `GET /tables/{tableName}/consumingSegmentsInfo` repeatedly until it returned no consuming segments, but this did not reliably prevent the issue.
*Questions:*
1. Is there a definitive API or mechanism to confirm that all Kafka consumers for a table have fully stopped after the table is deleted?
2. Is there a recommended sequence or waiting strategy to ensure consumers are fully torn down before proceeding to delete Kafka topics?
3. Are there any known race conditions in the table deletion flow where consumers can briefly restart or reconnect and recreate a deleted Kafka topic?

If there are no CONSUMING segments then there are no consumers that consume data. I am not sure how you reached this situation, because there is always a consuming segment in the normal, case. If you manually delete a consuming segment, one will be created automatically for you eventually. The best way to verify is to check zookeeper for segments in consuming state.

Best way to not have any consuming segments is to pause consumption on the table. After the pause is completed (it is not instantaneous) you can disable the table. This will also stop the periodic creation of a consumer in the pinot controller.

> Kafka consumers for those tables had not fully stopped by the time we deleted the topics
How did we verified this ..? Any logs on the Controller or Server that says it is consuming ..?


For a second layer of verification, You can stop the entire pinot cluster and see if the topics are re-creating after once again deleting the topics.

If this is the case then there is chance that, the some other process ( a producer  or a different consumer ) is creating them .

The controller also creates consumers to check on the number of partitions in the topic. This happens periodically. Disabling the table will (should) stop all of this activity

If we can update the doc somewhere to reflect this, please let us know, or update it yourself