Hello 👋 Is there any way to know / get notified (e.g. kafka message) when a message is done being ingested and available for query ?

There isn't at message level at the moment. When segment is committed, the Kafka offset is stored in segment ZK metadata.

I see, thanks for the answer @Mayank The idea, is that Pinot consumes some messages which will affect query results (pretty logical ^^) - but some other components relies on theses results and pre-computes other results outside Pinot (typically a Node JS micro service) Therefore the need is to update the pre-materialized view whenever Pinot results will change, i.e. whenever a new message is ingested We're considering listening to the original message (the one consumed by Pinot) as a trigger for recomputation - but then we'd have a race condition between this consumer and Pinot

May be create a column that has some counter/offset that could be used for this synchronization?

Periodically check that the counter hasn't moved, if so, recompute using latest data ?

I was thinking more like get the counter value that you want to synchronize on (perhaps from the apps side), and the add filter on pinot query

where counter <= x)

?

How would this ''get the counter'' be triggered ? Periodically ? Using the same message Pinot will / has (hopefully) already ingested ?

It is a high-water-mark of sorts that your other systems need to maintain that want to synchronize with Pinot (sorry if I am making incorrect assumptions)

So the other system would periodically check if this high water mark is greater than it's latest computation - if so, fetch updated results ? (I've omitted quite a few details, no worries 🙂)