Hi team, we are observing a pattern of latency inc...
# troubleshootingc
Carl
11/11/2021, 4:56 PMHi team, we are observing a pattern of latency increase daily in Pinot query. E.g. p95 increase from <100ms to 400ms, and this increase last for less then a hour each day. Is there some system metrics we could look at to identify t root cause for this?
m
Mayank
11/11/2021, 4:59 PM
Check for data push, qps increase, segments flush in RT to begin with
c
Carl
11/11/2021, 5:03 PMIt’s a offline table. We push data to multiple tables at different time each day. All these table seems to follow the same pattern of latency increase at similar time.
Carl
11/11/2021, 5:27 PMThere does seem to be increases of Qps at some hours of each day. But the brokers CPU has been always less than 10%. Should we start considering increase broker amount?
m
Mayank
11/11/2021, 5:55 PM
Just to confirm, you are saying that data push does not align with latency increase, but read qps increase does? If so, what's the server side CPU usage?
c
Carl
11/11/2021, 5:59 PMYes, the Qps is more align with the latency increases. Server CPU also increases almost the same time, but still less than 10%
m
Mayank
11/11/2021, 6:02 PM
How's the IO?
c
Carl
11/11/2021, 6:12 PMYou mean server disk usage? Or doc scanned?
m
Mayank
11/11/2021, 6:14 PM
Disk reads
Mayank
11/11/2021, 6:14 PM
Also, is the server CPU at peak time more than normal?
c
Carl
11/11/2021, 6:21 PMServer CPU peak time is 10% normally it’s 1%
Carl
11/11/2021, 6:28 PMWe do not have disk IO metrics, but I can see scanned docs increases 10 times more daily at the peak time than normal of a day.
Carl
11/11/2021, 7:24 PMJust checked the fs read total is also aligned with the latency increases.
a
Ashwin
11/11/2021, 10:22 PMHow have you configured your partitions? Could those be contributing to increased latency?
m
Mayank
11/12/2021, 1:20 AM
How much data per server and how what’s the memory available?
2 Views