Hi. This isn't a problem. I'm looking for design feedback. I'm using Apache Flink to generate inputs into Pinot's realtime tables. Any experience with doing aggregation in Flink vs Pinot? My current prototype aggregates user events inside of Pinot for streaming but aggregates in Flink for Pinot's offline ingestion. I'm tempted to move the streaming aggregation to Flink too so I can reuse the same code for streaming and batch. Flink's aggregation is nice and very performant. Flink supports sending

delete

and

insert

operations for group by results. Here's a simplified version of my Pinot table to help illustrate my use case: (dimensions) utc_date, seller_id, content_id, (metrics) sum_clicks.

@Dan Hill how are you querying the computed aggregates via Flink ? Are you using some key-value store ?

Hi Chinmay! For the current prototype: • streaming - it just sends the raw events to Pinot for aggregation. • batch - it writes aggregates to S3 and a Pinot ingestion job reads it.

Oh I meant in your new design where you'll use Flink for computing the streaming aggregates - how are you planning to serve it ?

Thanks for the details! TBD. I was planning on still serving using Pinot. It's pretty flexible and easy to use. I wasn't planning on keeping the full raw events in Pinot (there's a lot of extra info that I don't need to query quickly). Does Pinot support exactly once through other streaming routes? I assumed no. My current prototype has similar exactly once issues with Pinot. If I don't rely on Pinot to aggregate for raw events (if delete and insert is supported), I'd assume this is better supported. My Pinot input needs multiple joins. My current plan is to have both a stream and batch version. The stream does an interval join.

As of now, Pinot does not support exactly once. There's work in progress to make upserts work which will accomplish that.