+1 that diagram is a simplified view. There is Kafka in the middle

thanks everyone 🙂

The discussion around the pull based model brings an interesting question. Currently, Pinot tightly couple scrapping (or event sourcing) and indexing. Was there any thought given to separate them to scale independently?

@User @User have done some initial work to separate out ingestion and serving. But I don't think we really thought out separating ingestion and indexing. What would the benefits be? Wouldn't it cause more delays for realtime scenarios?

I think he is talking about a push api. For e.g. can we directly push to Pinot from Flink/Samza without going through Kafka job.

@User @User could you guys see if we have any other thing we need to do for this pull request: https://github.com/apache/incubator-pinot/pull/3869

@User sure

Is there any recommended qps upper limit for the zk cluster managed by Helix? I want to let Pinot servers send notifications to Pinot broker via Helix. If each Pinot server sends 1 notification per second, that means around ~10 qps to Helix for 10 servers. I wonder if such traffic volume will cause message delivery latency issues.

What’s the motivation?

Is this for presto Pinot connector?

This is for pinot upsert, used for syncing the low-water-mark of upsert progress to the broker so it can construct the proper filtering condition when we process the query on broker

we are trying to look into some solutions and see which one will be better

Our original idea was log coordinator pulls from servers and maintains the water marks and brokers/clients poll log coordinator periodically

Does that work you?

we are doing similar, but for our current stage of POC we are going to skip log coordinator and let the broker pull the watermark directly from servers (in a sense they are similar operations)

I think @User is exploring the option of utilizing of helix to pass around this info

Makes sense. Skip Helix for now.

It will quickly add up and number of writes will be a lot.

@User ^

Thanks @User and @User. So Helix seems not designed for such use cases of high write throughputs. I will let broker to directly pull from servers for watermarks info.

👋 Question to the Uber team, I noticed in the Presto Slack channel that Uber using Presto Pinot connector in production and working on pushdown aggregation support. Any thoughts on open sourcing it?

+1 to this 🙂

+1 as well

+1 too

I have had initial conversation about with Girish in Uber. They have plans but not fully ready yet. I will let Girish or Divesh chime in.

👋 welcome @User. Arjun working on user experience and applications on top of Pinot at Slack and he have a lot of things to say about Pinot query engine from the integration perspective 🙂

welcome @User. It will be great to know from your perspective on what we need to build in Pinot

@User some of us cannot make it to the meetup tonight, I am wondering if we have a video feed set up so we can watch online?

also @User ^

Unfortunately we dont have a live feed for the meeting. But we will be recording the event and will be uploading the talks to Youtube channel as soon as video production is ready