This message was deleted.
# troubleshooting
s
This message was deleted.
m
our requirement is respoonse time ie lsess then a minute with 2000 concurrent queries.
The error we are seeing is query time out becasue of mergebuffers and query time out.
For what it's worth, I'm aware of one use case where the unwanted response time was caused by a datatype problem, i.e., the data was ingested as an int instead of a string.
g
my top three generic performance tips: 1) ensure your setup is configured well for your server. the best way to tell if this is done well is to look at your system metrics. when you're doing a load test, the server should be running at 100% CPU (meaning it's fully utilized). if you're looking for subsecond queries, you want to be minimizing disk i/o, so ensure disk i/o is also low. you also want time spent doing GC to be relatively low-- a few seconds per minute at most. (you can see this by adding
+XX:+PrintGC
to your jvm.config or by using jvm tools.) if any of these are not looking ideal, you may need to adjust settings. 2) ensure you don't have too many small segments: overheads per-segment can be high in this case. you can use the web console or sys.segments table to see the average number of rows per segment. Generally we target a few million rows per segment. Lower is OK if you have a smaller amount of data. But you don't want tons of segments that are, like, 10000 rows each 3) you can use a flame graph (https://support.imply.io/hc/en-us/articles/360033747953-Profiling-Druid-queries-using-flame-graphs) to see what specific code is taking up your processing time. very useful tool!
When doing a lot of concurrent queries (like you are) it's super important to minimize the CPU usage of each query. A small change in CPU use can make a big difference Some things that help here: • Make sure you are doing filtering efficiently: ensure your time filters are planned to
intervals
, and if you have a specific column you're often filtering on, consider applying secondary partitioning (aka clustering) • Avoid unnecessary query time expressions • Use the flame graph technique (see prior comment) to find where time is being spent