https://pinot.apache.org/ logo
Join Slack
Powered by
# qps-metric
  • s

    Sidd

    07/16/2019, 6:07 PM
    @User @User
  • s

    Sidd

    07/16/2019, 6:09 PM
    Hey Guys, creating this channel to brainstorm the idea that you had back then to solve the max qps per minute problem. Neha I was just telling Kishore that the problem is that InGraphs averages metric over a minute and therefore we miss spikes in QPS. Is that right?
  • s

    Sidd

    07/16/2019, 6:10 PM
    I am not sure if I am explaining the problem correctly to Kishore to refresh his memory of what his suggestion was
  • s

    Sidd

    07/16/2019, 6:13 PM
    For reference, this is Neha's PR -- https://github.com/apache/incubator-pinot/pull/3721
  • n

    Neha Pawar

    07/16/2019, 6:15 PM
    yes that's correct, we want to see the max qps in a minute
  • s

    Sidd

    07/16/2019, 6:18 PM
    Kishore, looks like you had suggested to keep a track of in flight queries and emit the max
  • s

    Sidd

    07/16/2019, 6:19 PM
    What I learned from Neha was that you were referring to some Metrics library method that can keep track of max and I didn't see one
  • s

    Sidd

    07/16/2019, 6:19 PM
    such API after looking at the Meter class code
  • k

    Kishore G

    07/16/2019, 6:21 PM
    So we need to track total queries per second and emit max at the end of 60 seconds?
  • s

    Sidd

    07/16/2019, 6:22 PM
    Yes, we need to have a QPS metric and emit the max QPS per minute
  • k

    Kishore G

    07/16/2019, 6:27 PM
    let me see if the chat is still available in slack
  • k

    Kishore G

    07/16/2019, 6:33 PM
    Neha Pawar [10:20 AM] everytime a broker receives a query, in the BaseBrokerRequertHandler, we are setting
    BrokerMeter.QUERIES, 1
    . now, if we were to use a histogram, we would set something like
    BrokerHistogram.QUERY,  MAX, 1
    .but this would still end up giving us 1. what i was thinking is we aggregate query count every 10s (or some number of seconds). and emit it to histogram. In that case, histogram would show the max value in that bucket over a minute. but what you're suggesting is that there is a way to make the histogram emit it's size every minute. is that right? Kishore G [10:36 AM] what I was suggesting was to emit the in flight queries count instead of 1 that will give the max queries in flight at any time during that minute we dont have to restrict it to 10 second aggregates Neha Pawar [10:49 AM] where can i get the inflight queries count? Kishore G [11:40 AM] I dont know if we have something for that, but just have a atomic integer that increments when a query is scheduled and decremented when we send the response out with this, we wont even need historgram Neha Pawar [12:13 PM] sorry i still dont understand how that will help. anything emitted as a Metric will be averaged over 1 minute. set as a Gauge will give us the sum over the last minute. if we count requests in flight, and emit them at every increment/decrement as a metric, averaged out values wont help. For example, over a minute if we emit 1,2,3,4,5,4,3, the metric will show as 22/60 = 3.6 (avergaed over 60 points). if we emit it to a Gauge, metric will show 22. Now how do we differentiate between a 22 being shown because of 1,2,3,4,5,4,3 (burst of queries) vs 1,0,1,0,1,0... (22 1's uniformly spread out queries) Kishore G [12:15 PM] you just emit the max right Kishore G [12:27 PM] is my understanding right that you really want to see number 5 in the ingraph? Neha Pawar [12:45 PM] yes, want to see 5 so you're suggesting using the atomic integer to incrememnt /decrement as a replacement to the 10second bucketing i will still have to keep track of max, emit it every minute, and reset it every minute or use histogram Kishore G [1:11 PM] The metrics library should keep track of max We just update the gauge on every broker request Kishore G [1:17 PM] See the top answer here https://stackoverflow.com/questions/30987757/dropwizard-metrics-meters-vs-timers
  • k

    Kishore G

    07/16/2019, 6:33 PM
    ok, this is what I found. Reading it now
  • s

    Sidd

    07/16/2019, 6:48 PM
    Yes I had read the article to see who will do our job and it looks like Meter doesn't really keep a track of max
  • k

    Kishore G

    07/16/2019, 7:00 PM
    Histogram does right
  • s

    Sidd

    07/16/2019, 7:36 PM
    yes it does... but I think somewhere in the above chat you had mentioned we don't need histogram.... I think that explains why I was kind of going in circles
  • k

    Kishore G

    07/16/2019, 7:37 PM
    theoretically we dont but looks like yammer does not support it
  • s

    Sidd

    07/16/2019, 7:43 PM
    okay so to summarize ... we keep an atomic counter (which is a histogram), it is incremented when a query request arrives at broker and decremented when response is sent out, and we update the broker gauge every 1 min with max
  • s

    Sidd

    07/16/2019, 7:44 PM
    does this sound right?
  • k

    Kishore G

    07/16/2019, 7:47 PM
    First part is correct.
  • k

    Kishore G

    07/16/2019, 7:48 PM
    We can’t update a gauge every minute.
  • k

    Kishore G

    07/16/2019, 7:48 PM
    We just update add the current queue size to the histogram every query
  • k

    Kishore G

    07/16/2019, 7:49 PM
    When anyone (in graph ) asks for the metric , we return histogram. Max
  • s

    Sidd

    07/16/2019, 7:51 PM
    Okay sounds good. I will get going with the changes. Thanks @User and @User
  • k

    Kishore G

    05/27/2020, 5:20 PM
    @User has left the channel
  • n

    Neha Pawar

    05/04/2022, 8:45 PM
    @Neha Pawar has left the channel