do we have this paper somewhere on the public server (github/apache)? right now the only download link for this paper is on ACM and it is not free

Thanks for sharing @User

Hello team Pinot! We've been using Pinot as our backend db for a while now and would like to see some dashboards in Grafana for monitoring purposes. We already have the Prometheus metrics set up and being ingested. I was wondering what are some of good queries for monitoring Pinot cluster health and if there's some documentation about that? Thank you!

@User do you have any documentation that might help here ^^?

@User I think it will be good idea to publish the list of metrics on which we create alerts at LinkedIn

Yes, agree

thats what @User is looking for

I was planning to discuss that internally anways, but reached out to Ananth, in case he had that handy

@User for now, you could refer to the broker and server metrics we have as a starting point:https://github.com/apache/incubator-pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/common/metrics/BrokerMeter.java

Thanks @User

Hi All, I just sent out an email to mailing list with documents for text search

POC -- https://docs.google.com/document/d/1P38NvfNfATiTzd8W_ZBzPyAnP50_lyFk-qm6syqeD2E/edit#heading=h.idgz7yj1sspq

Perf -- https://docs.google.com/spreadsheets/d/1N94uejru-5IfiHuDvqDXXM1V23eqqzLkMgEUxVVxFBY/edit#gid=0

This is fantastic. Can you add the size of index in perf

Sure

Added the index size to perf sheet. Also, the "performance evaluation" section in POC doc has some information on perf experiment setup. I'd suggest reading that as well when looking at perf numbers.

Also, it would be good if people can join apache-pinot-dev google group

It would be easier to share documents that way

@User,Which monitoring tool do you recommend to monitor Pinot, We are running Pinot in Kubernetes cluster. We have NewRelic to monitor other services. Can we use NewRelic to monitor Pinot ?

Sure. As long as they have an agent that can pull JMX metrics from Pinot and push to the NewRelic endpoint

I added the pinot upsert design doc to pinot.apache.org here: https://cwiki.apache.org/confluence/display/PINOT/%5BProposal%5D+Pinot+upsert+design+doc. We still have a lot of things to update/add details so think about this doc as WIP but I am putting it there for more visibilty. Btw I am also thinking about publishing our current working upsert branch to pinot github as a separate branch. Does anyone has some suggestion on how to do it?

Suggestion on creating a branch?

yes, how to create new branch on apache pinot repo? do i need to get someone with permission to do it or maybe I should do it myself

kk, will look into that

Anyone here used elastic search in real-time ?

Updated design doc for segment merge & rollup https://cwiki.apache.org/confluence/display/PINOT/Segment+Merge+and+Rollup