hey all, I observed this weird behavior, with same...
# generals
Shounak Kulkarni
04/29/2020, 6:58 AMhey all, I observed this weird behavior, with same setup the latency and server cpu usage is varying at different time:
case 1 - PT done after few minutes of table creation
1000 rps -> mean latency 185ms
cpu usage by server -> 2.5 corescase 2 - PT done after 8-10 hrs of table creation
1000 rps -> mean latency 20 ms
cpu usage by server -> 850 millicoresAny idea what may be the reason?
k
Kishore G
04/29/2020, 7:00 AM
Same queries?
s
Shounak Kulkarni
04/29/2020, 7:00 AMyes
k
Kishore G
04/29/2020, 7:01 AM
only thing I can think of is JVM has warmed up
Kishore G
04/29/2020, 7:03 AM
and most data is in system page cache
s
Shounak Kulkarni
04/29/2020, 7:03 AMactually before the first case i ran 500 rps for 15 mins to warm up
Shounak Kulkarni
04/29/2020, 7:03 AMand there was no activity between the first and second case
k
Kishore G
04/29/2020, 7:04 AM
whats the query
s
Shounak Kulkarni
04/29/2020, 7:05 AM"select MAX(timeSinceEpoch), activityType FROM usermap where timeSinceEpoch >= 1587180605 and userId = '%d' group by activityType"
"select MAX(timeSinceEpoch), modelYearCode FROM usermap where timeSinceEpoch >= 1587180605 and userId = '%d' group by modelYearCode top 10"
"select MAX(timeSinceEpoch), cCode FROM usermap where timeSinceEpoch >= 1587180605 and userId = '%d' and activityType = 'BMO' group by cCode top 10"
k
Kishore G
04/29/2020, 7:10 AM
yeah, it might just be the warm up
Kishore G
04/29/2020, 7:10 AM
restart the servers and rerun the benchmark
s
Shounak Kulkarni
04/29/2020, 7:11 AMok
Shounak Kulkarni
04/29/2020, 7:26 AM1000 rps -> 17ms
server cpu usage-> 850 millicores
s
Sidd
04/29/2020, 7:27 AMIs it now in the same ballpark across multiple runs?
s
Shounak Kulkarni
04/29/2020, 7:28 AMyes same setup
s
Sidd
04/29/2020, 7:34 AMI recommend using PerfBenchmarkRunner. Profile each run using YourKit or linux perf tool. It will generate the call-graph and you can see what is contributing to time
s
Shounak Kulkarni
04/29/2020, 7:41 AMok will look into that tool. thanks!
s
Sidd
04/29/2020, 7:57 AMIt is part of pinot.. you can attach a profiler to it
Copy code
sh pinot-tools.sh PerfBenchmarkRunners
Shounak Kulkarni
04/30/2020, 12:05 PMSo I tried to recreate the above two cases. The only difference is that as the default time threshold for segment creation is 6 hrs, the consuming segment moves to completion and hence there is no data in consuming segment. Not sure why the consuming segment is causing the issue
s
srisudha
05/01/2020, 1:16 AMWhat Shounak has observed seems strange because as per him whenever there is a consuming segment in memory and different queries are fired the query latency is very high. Is this how it is supposed to work ? Or are there any configuration to counter this?
k
Kishore G
05/01/2020, 1:40 AM
Can we start a channel for this?
Kishore G
05/01/2020, 1:41 AM
@Shounak Kulkarni @srisudha
s
srisudha
05/01/2020, 2:39 AMSure doing it now
s
Sidd
05/01/2020, 2:42 AMHow many consuming vs offline segments are there?
Sidd
05/01/2020, 2:43 AMDid we check from the broker logs that when the query latencies are high, it is the realtime servers that dominate the response time
Sidd
05/01/2020, 2:43 AM?
Sidd
05/01/2020, 2:45 AMThe broker log will print our per query the number of offline and consuming segments used to serve the query and elapsed time per offline and realtime server
k
Kishore G
05/01/2020, 2:46 AM
can we please create a channel, thread feature in slack is very confusing