Hi, does Pinot offer an implementation of Change Data Capture (CDC) for OLAP queries so that updated query results can be obtained automatically? Or an alternative way to be notified of updated indexes so that we can manually retrigger the query?

Not at the moment. Moreover, with real-time ingestion/indexing unsure if that makes sense.

What are you trying to accomplish?

We’re trying to stream Pinot query results as input to a Flink stream processor. We would like to have a new message in that stream either periodically or when the results of the query differ from the last result (in our understanding, that is equivalent to when the underlying indices change that are relevant for the query). As we understand it, we could implement much of the logic for that ourselves using the Pinot Java client but were just wondering if there are certain things in Pinot that could ease this task. 🙂

Got it. we dont have that but can be a useful contribution

Ok, thank you! We'll first do a basic implementation for a Pinot-source to Flink without touching Pinot internals. We would get back to you when we want to touch Pinot internals to improve our baseline. 🙂

👍