Hi Team, Context: We want to use Apache Pinot for real-time analytics query use cases in our microservices. Since realtime Pinot tables ingest directly from Kafka, ingestion delays/lag can occur. Our requirement is: whenever a document (row) in Pinot is updated, we want to push an event to Kafka with the primary key that changed. This would allow downstream microservices to consume that event, know that a specific record has been updated in Pinot, then trigger real-time analytics queries and perform required downstream actions. Question: Is there any existing feature or recommended workaround in Pinot to detect when a row is updated in a realtime table and trigger an event (e.g., send a Kafka message) so downstream services can be notified?

Why do you think lag/delay can occur in Pinot? What’s the lag/delay tolerance your applications have? Pinot provides an api to get the latest consumed offset.

1. Pinot resources are fixed, is it possible for ingestion lag to increase if the overall throughput rises? 2. I was checking the Pinot Grafana dashboard, and the latency metrics shown there don’t seem to match what we expect. Could you please clarify where I can monitor the actual Kafka-to-Pinot ingestion lag (i.e., how far Pinot consumers are behind the Kafka head)? 3. Also, regarding our earlier discussion — would it make sense to modify Pinot’s consumer logic to publish an event to a Kafka topic after a record is updated in Pinot? Or is there a recommended way to achieve this notification mechanism without changing Pinot’s core ingestion code?

If you have lag, then best course is to identify the cause and fix lag, not to adjust applications using Pinot

@Mayank Even if there’s no lag, we still need to broadcast an event when a row is updated, since the applications are not polling — they will listen for the event and then query the updated data from Pinot.

lag metric :

pinot_server_realtimeIngestionOffsetLag_Value

No metric with this name

what version are you on?

Pinot version is latest 1.4.0. Does

pinot.yml

not expose all metrics? I’m currently using

pinot.yml

, so should I update the

jvmOpts

under the

server

configuration to

server.yml

? For the other components, is it fine to continue using

pinot.yml

or should use relative file?

@Rahul Sharma Pinot doesn't have a way to emit Kafka events to capture offsets. Moreover, I am unsure if that is the right thing to do for Pinot. You might create a lightweight service that polls Pinot for offsets, and emits to Kafka.