This message was deleted.

really depends on a lot of factors, but assuming you have done all optimizations required for segment sizing/indexing/partitioning/compaction and historical performance config tuning, then id recommend looking at several metrics • CPU Usage on the historicals (Check if its consistently reaching 90% or more for long durations due to multiple concurrent expensive queries being executed) • Jetty threadpool is low on threads https://druid.apache.org/docs/latest/operations/metrics/#jetty

This is just a quick look for the last week of all historical CPU Usage versus CPU Limits and Requests

This cluster is almost idle, can you also pull up the same metrics on the broker?

last 15 min for broker

did you run into any issues with handling the 120K queries? IE blocked stalled requests? slow query performance? etc.... this likely just points to a configuration optimization issue where we may need to bump the number of processing threads or http threads to handle the incoming requests on the broker/historical/router

yeah sure, tbh I have never experienced any issues handling that amount of queries. That being said, the 120k queries are for a large time frame, so the actual request per second querying druid is much less than 1rps. • broker config:

- name: DRUID_MAXDIRECTMEMORYSIZE
              value: 9000m
            - name: DRUID_XMS
              value: 6500m
            - name: DRUID_XMX
              value: 6500m
            - name: druid_announcer_type
              value: http
            - name: druid_broker_http_maxQueuedBytes
              value: "50000000"
            - name: druid_broker_http_numConnections
              value: "20"
            - name: druid_node_type
              value: broker
            - name: druid_processing_buffer_sizeBytes
              value: "500000000"
            - name: druid_processing_numMergeBuffers
              value: "12"
            - name: druid_processing_numThreads
              value: "1"
            - name: druid_query_scheduler_numThreads
              value: "39"
            - name: druid_request_logging_setContextMDC
              value: "true"
            - name: druid_request_logging_setMDC
              value: "true"
            - name: druid_request_logging_type
              value: slf4j
            - name: druid_server_http_numThreads
              value: "40"
            - name: druid_serverview_type
              value: http
            - name: druid_service
              value: "druid/broker"
            - name: druid_sql_avatica_enable
              value: "true"
            - name: druid_sql_enable
              value: "true"
            - name: druid_sql_http_enable
              value: "true"
            - name: druid_sql_planner_metadataSegmentCacheEnable
              value: "true"

• historical:

- name: DRUID_MAXDIRECTMEMORYSIZE
              value: "160g"
            - name: DRUID_XMS
              value: "24g"
            - name: DRUID_XMX
              value: "24g"
            - name: druid_cache_sizeInBytes
              value: "2000000000"
            - name: druid_cache_type
              value: "caffeine"
            - name: druid_historical_cache_populateCache
              value: "true"
            - name: druid_historical_cache_useCache
              value: "true"
            - name: druid_node_type
              value: "historical"
            - name: druid_processing_buffer_sizeBytes
              value: "700000000"
            - name: druid_processing_numMergeBuffers
              value: "8"
            - name: druid_processing_numThreads
              value: "31"
            - name: druid_segmentCache_locations
              value: '[{"path":"/opt/druid/var/druid/segment-cache","maxSize":"7330900000000"}]'
            - name: druid_server_http_numThreads
              value: "100"
            - name: druid_server_maxSize
              value: "7330900000000"

• router

- name: DRUID_MAXDIRECTMEMORYSIZE
              value: 128m
            - name: DRUID_XMS
              value: 1500m
            - name: DRUID_XMX
              value: 1500m
            - name: druid_node_type
              value: router
            - name: druid_router_coordinatorServiceName
              value: druid/coordinator
            - name: druid_router_defaultBrokerServiceName
              value: druid/broker
            - name: druid_router_defaultRule
              value: _default_tier
            - name: druid_router_http_numConnections
              value: "20"
            - name: druid_router_http_numMaxThreads
              value: "25"
            - name: druid_router_http_readTimeout
              value: PT8M
            - name: druid_router_http_requestBuffersize
              value: "16384"
            - name: druid_router_managementProxy_enabled
              value: "true"
            - name: druid_router_tierToBrokerMap
              value: '{"_default_tier":"druid/broker"}'
            - name: druid_server_http_numThreads
              value: "25"