https://pinot.apache.org/ logo
d

Dan Hill

10/01/2020, 1:11 AM
Hi. This isn't a problem. I'm looking for design feedback. I'm using Apache Flink to generate inputs into Pinot's realtime tables. Any experience with doing aggregation in Flink vs Pinot? My current prototype aggregates user events inside of Pinot for streaming but aggregates in Flink for Pinot's offline ingestion. I'm tempted to move the streaming aggregation to Flink too so I can reuse the same code for streaming and batch. Flink's aggregation is nice and very performant. Flink supports sending
delete
and
insert
operations for group by results. Here's a simplified version of my Pinot table to help illustrate my use case: (dimensions) utc_date, seller_id, content_id, (metrics) sum_clicks.
c

Chinmay Soman

10/01/2020, 2:14 AM
@Dan Hill how are you querying the computed aggregates via Flink ? Are you using some key-value store ?
or using queryable state ?
d

Dan Hill

10/01/2020, 2:22 AM
Hi Chinmay! For the current prototype: • streaming - it just sends the raw events to Pinot for aggregation. • batch - it writes aggregates to S3 and a Pinot ingestion job reads it.
c

Chinmay Soman

10/01/2020, 2:23 AM
Oh I meant in your new design where you'll use Flink for computing the streaming aggregates - how are you planning to serve it ?
Is that also sent to Pinot ? or you're using a key-value store for serving ?
In general, you don't need Flink for such pre-aggregation, you could simply rely on Pinot's star-tree index for efficiency and speed
A few things in Flink which make this a bit cumbersome: • Delayed messages : you'll have to play around with
allowed lateness
stuff. This is not a problem in Pinot • Flink job restarts: this may end up re-computing some aggregates which will be re-sent to Pinot. As of now, Pinot does not support exactly once sink for Flink
In addition, you'll also have to handle checkpointing storage for Flink which is unnecessary for such use cases. Also, if your aggregation function / schema changes you'll have to keep redeploying the job which is also slightly inconvenient
@Dan Hill hope this helps
d

Dan Hill

10/01/2020, 3:11 AM
Thanks for the details! TBD. I was planning on still serving using Pinot. It's pretty flexible and easy to use. I wasn't planning on keeping the full raw events in Pinot (there's a lot of extra info that I don't need to query quickly). Does Pinot support exactly once through other streaming routes? I assumed no. My current prototype has similar exactly once issues with Pinot. If I don't rely on Pinot to aggregate for raw events (if delete and insert is supported), I'd assume this is better supported. My Pinot input needs multiple joins. My current plan is to have both a stream and batch version. The stream does an interval join.
c

Chinmay Soman

10/01/2020, 3:45 AM
As of now, Pinot does not support exactly once. There's work in progress to make upserts work which will accomplish that.
For joins/filtering prior to Pinot - Flink is a great candidate ! If you're doing that already then yes - doing pre-aggregation in Flink will be totally fine.