Hi community, I'm currently using Pinot 1.0.0 with a Kafka real-time table setup, and I have a use case where I would like to trigger an action immediately after a message is consumed by the Pinot real-time table. Is there a way for Pinot to acknowledge the consumption of a Kafka message, allowing me to execute a specific action upon successful ingestion of that message ? Any suggestions or best practices on how to achieve this would be greatly appreciated !

There isn’t a way to do so currently. What’s the use case that needs this?

Here’s the use case: We are ingesting data, and after each ingestion, we need to check if an aggregation of multiple data points over a specific time window exceeds a certain threshold. If this value is greater than the threshold, we trigger a notification. The challenge here is that the aggregation check depends on the new data point being successfully ingested into the table. We need to ensure that the calculation is only performed once the new record is definitely in the table, and not before. Maybe using the table offset could be interesting…

Yes, this could be done via pull (calling offset api in Pinot)

thanks @Mayank

@Hassan Ait Brik What’s your ingestion rate? Usually this kind of problem can be solved in the streaming processing framework like Flink, which you could specify a window for an aggregation function then trigger it based on certain conditions