@User @User

Hey Guys, creating this channel to brainstorm the idea that you had back then to solve the max qps per minute problem. Neha I was just telling Kishore that the problem is that InGraphs averages metric over a minute and therefore we miss spikes in QPS. Is that right?

I am not sure if I am explaining the problem correctly to Kishore to refresh his memory of what his suggestion was

For reference, this is Neha's PR -- https://github.com/apache/incubator-pinot/pull/3721

yes that's correct, we want to see the max qps in a minute

Kishore, looks like you had suggested to keep a track of in flight queries and emit the max

What I learned from Neha was that you were referring to some Metrics library method that can keep track of max and I didn't see one

such API after looking at the Meter class code

So we need to track total queries per second and emit max at the end of 60 seconds?

Yes, we need to have a QPS metric and emit the max QPS per minute

let me see if the chat is still available in slack

Neha Pawar [10:20 AM] everytime a broker receives a query, in the BaseBrokerRequertHandler, we are setting

BrokerMeter.QUERIES, 1

. now, if we were to use a histogram, we would set something like

BrokerHistogram.QUERY,  MAX, 1

.but this would still end up giving us 1. what i was thinking is we aggregate query count every 10s (or some number of seconds). and emit it to histogram. In that case, histogram would show the max value in that bucket over a minute. but what you're suggesting is that there is a way to make the histogram emit it's size every minute. is that right? Kishore G [10:36 AM] what I was suggesting was to emit the in flight queries count instead of 1 that will give the max queries in flight at any time during that minute we dont have to restrict it to 10 second aggregates Neha Pawar [10:49 AM] where can i get the inflight queries count? Kishore G [11:40 AM] I dont know if we have something for that, but just have a atomic integer that increments when a query is scheduled and decremented when we send the response out with this, we wont even need historgram Neha Pawar [12:13 PM] sorry i still dont understand how that will help. anything emitted as a Metric will be averaged over 1 minute. set as a Gauge will give us the sum over the last minute. if we count requests in flight, and emit them at every increment/decrement as a metric, averaged out values wont help. For example, over a minute if we emit 1,2,3,4,5,4,3, the metric will show as 22/60 = 3.6 (avergaed over 60 points). if we emit it to a Gauge, metric will show 22. Now how do we differentiate between a 22 being shown because of 1,2,3,4,5,4,3 (burst of queries) vs 1,0,1,0,1,0... (22 1's uniformly spread out queries) Kishore G [12:15 PM] you just emit the max right Kishore G [12:27 PM] is my understanding right that you really want to see number 5 in the ingraph? Neha Pawar [12:45 PM] yes, want to see 5 so you're suggesting using the atomic integer to incrememnt /decrement as a replacement to the 10second bucketing i will still have to keep track of max, emit it every minute, and reset it every minute or use histogram Kishore G [1:11 PM] The metrics library should keep track of max We just update the gauge on every broker request Kishore G [1:17 PM] See the top answer here https://stackoverflow.com/questions/30987757/dropwizard-metrics-meters-vs-timers

ok, this is what I found. Reading it now

Yes I had read the article to see who will do our job and it looks like Meter doesn't really keep a track of max

Histogram does right

yes it does... but I think somewhere in the above chat you had mentioned we don't need histogram.... I think that explains why I was kind of going in circles

theoretically we dont but looks like yammer does not support it

okay so to summarize ... we keep an atomic counter (which is a histogram), it is incremented when a query request arrives at broker and decremented when response is sent out, and we update the broker gauge every 1 min with max

does this sound right?

First part is correct.

We can’t update a gauge every minute.

We just update add the current queue size to the histogram every query

When anyone (in graph ) asks for the metric , we return histogram. Max

Okay sounds good. I will get going with the changes. Thanks @User and @User

@User has left the channel

@Neha Pawar has left the channel