kk, got it

Hi Team, I was trying to understand a few patterns. What should be expected pattern for the 99% latency If QPS increases ?

Given a cluster with N broker and M servers, 99th percentile latency usually follows a hockey stick pattern with respect to QPS. We just did these experiments internally for some use cases. The latency very slowly increases with QPS and then at a particular inflection point the latency shoots up exponentially.

something like this

image.png

this is just a sample tos how how it might look

And also assumes that query patterns are fixed

Once the servers are maxed out (CPU wise), the latency shoots up like how Ravi mentioned; most of it is due to queueing delays; you will most likely see requests hitting the broker configured timeout (30s by default) and plateau there

Thanks..I am a student currently and we had implemented a new segment assignment strategy and published a paper on the same last year. I am trying to run the experiments and get a pattern of 99% latency on increasing QPS, the latency is decreasing.

Trying to figure out why.. some leads would be appreciated

How are you measuring the latency? Also, could you provide details about your experiment: eg: cluster setup (how many servers/brokers), data size, query pattern, how are you firing queries, qps, latency observed etc?

1 Broker, 1 Server, 30K records, QPS (100,200,300), latency observed was 32 ms maximum

How long do you let it run for each qps?

5 minutes

And you are observing latency at 300qps < latency at 200|100qps?

Yes

What's the actual latency numbers?

With such small data size, may be you need to increase qps quite a bit

Also, make sure that each query is getting correct response.

How are you sending the queries? Do you have query runner that is firing queries in parallel (multiple threads)?

Sorry the record size is 1198388

1.19M records? What's the data size in MB?

Yes we are executing queries parallelly, we have a query generator

What's the actual latency numbers?

Typically, if latency is going down over time, it is because of jvm optimizations kicking in, and/or data fully in memory. Also possible that queries are not being run (eg connection refused)

I'd recommend making sure that the latency numbers are real (ie queries actually executing), and if so, then keep increasing qps until latency starts degrading

Yea..Give me a minute.. I will give the latency numbers

Can you point us to the paper

Sure