Hi Team, I am running load test on our test pinot ...
# troubleshootinga
Aditya
02/01/2022, 4:45 PMHi Team, I am running load test on our test pinot cluster, experimenting with getting high qps on same hardware (3 servers * 4 cores, 16 GB ram)
Data size is 100M rows, distributed over a year
The filter column is from_user_id (string), with sorted index
Copy code
Query : select sum(amount) as amt from transactions where from_user_id = "some id"r
Richard Startin
02/01/2022, 4:59 PMdoes user_id need to be a string?
Richard Startin
02/01/2022, 4:59 PMi.e. does it contain non numeric characters?
a
Aditya
02/01/2022, 5:01 PMyes, its uuid
r
Richard Startin
02/01/2022, 5:08 PMok that's a shames, numbers allow much faster filtering
Richard Startin
02/01/2022, 5:09 PMit would be good if you could provide a profile so we know where the bottleneck is, then we can suggest around that
Richard Startin
02/01/2022, 5:09 PM Copy code
cmd <pid> JFR.start duration=60s settings=profile filename=user_lookup_load_test.jfrRichard Startin
02/01/2022, 5:10 PMan optimisation was made for sum recently which might be relevant to you
a
Aditya
02/01/2022, 5:43 PMshall I run this during load test or running with just one query is sufficient
using docker images so i'll run this inside the container
Copy code
jcmd 1 JFR.start duration=60s settings=profile filename=user_lookup_load_test.jfrm
Mayank
02/01/2022, 7:56 PM
Partitioning gains for throughput are a lot more when you have larger number of segments that can be pruned out (or even servers that won’t be queried).
a
Aditya
02/02/2022, 6:10 AMi think a partitioned replica group config like this will increase throughput for current hardware setup, if I keep only 3 partitions one for each server
Copy code
"replicaGroupPartitionConfig": {
"replicaGroupBased": true,
"numInstances": 0,
"numReplicaGroups": 1,
"numInstancesPerReplicaGroup": 3,
"numPartitions": 3,
"numInstancesPerPartition": 1
}