Aggregate stats are available via metrics. If you ...
# general
m
Aggregate stats are available via metrics. If you want individual query level stats, the broker query log is very structured
p
Thanks for your reply. The thing is I want to answer questions like: 1) How many queries are made for table X in a week? 2) Ranking of users as per their query volume 3) In case of error, I want to know who made this query. 4) How many of queries were successful and how many were failed? 5) Anomalies in query response time. Since we are now growing in terms of data volume and number of people using Pinot, these questions are critical to maintain visibility on how the cluster is being used. Since we don’t have a service mesh in our EKS cluster, we are thinking in terms of VPC traffic mirroring to mirror the traffic for analysis but it will contain only requests and not responses. It would be nice if I can put a java class which can simply publish raw request and response body
p
Also, @piby If you feel like there still are issues with how you are getting telemetry, please check the Apache Pinot Github issues to see if there are any related currently-open issues or, if not, consider creating an issue to summarize your views. Just a for-example, there is #15057, which seems at least peripherally related to your desire for better telemetry (not saying "this is it" but just providing an example). Feel free to search about to see if something resonates and comment on it or open your own issue.