Is there a possibility to get cdc (change data capture) stream for pinot tables? Or something like post trigger? We have services which want to get notified when there is new data in the table and polling is not an optimal solution.

this is not supported yet, usually pinot data coming from kafka or s3, maybe kafka or s3 timestamp will be better check criteria?

We are currently listening to kafka topic and S3 events as a workaround. But it might be possible that the file arrived in S3/message arrived in kafka topic but it is not inserted in Pinot due to some error. Or it might get inserted with some delay. So, in this workaround, there are two sources of truth which we do not like. For example, a service listening to a kafka topic receives a message. It assumes that the message is inserted into pinot and tries to run a query on Pinot. The query can fail if Pinot table hasn’t read until the latest offset. It could take 1 second or even one minute for the message to arrive on Pinot. We could also poll Pinot table for new data but it is not optimal or scalable approach. We have a legacy time series database with cdc stream. There are many services which ”react" whenever new data arrives. We are planning to move these legacy usecases to pinot so we have to think about a way to react when new data arrives.

is this just for streaming or you also need this for batch?

would be nice to do it for batch data as well. Duplicate cdc events would be a bummer :/ I mean I know the threshold time for the consuming segment (1h for example) and put a flink consumer on top of it which will filter out all the duplicates. But it will be extra work. Or Watermarking of time column would also be an option. So saving the last timestamp of the time column sent as a watermark in zookeeper

For offline segment trigger, it’s the standard way, we also have this similar flow internally. Helix calls it offlineToOnline transaction, which allows to process a code snippet during segment online

> for realtime tables, it’s a bit hard. One thing I can think of is that there is a field in query response called

minConsumingFreshnessTimeMs

, can you check if that can satisfy your requirements I have to check but it looks like it would be sufficient for us. Is this stored somewhere on zookeeper? We have burstable trading data as realtime feeds. During peak hours the volume can reach upto 100 trades/sec. We now have 60 realtime tables so we are currently brainstorming a general event driven architecture for our streams. During processing of an event, we generally need historical events, so we would rather connect to Pinot where all historical events are stored instead of listening to kafka directly.

when you send any query to pinot, it will return with this value.

We don’t want to poll for every table to get this metric in query response periodically. We want to get notified when this metric changes.

got it, then what about publish this as a per table basis metric?

would be sufficient for us. I don't care if it is updated to zookeeper on per table basis or available as a prometheus metric. Both would work for us.

Then I would say, maybe publish it as JMX metrics. The metrics reporter can be KafkaReporter, which could publish the metrics to Kafka for trigger basis.

Sure, do you want me to create a github issue for it?

Sounds good, check if you can find this metric, which is published from:`IngestionDelayTracker`

ServerGauge.REALTIME_INGESTION_DELAY_MS