https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
s
Context: We are using Druid 0.21.0 version with 8 data nodes, 2 master and 3 query
Broker runtime.properties
Copy code
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   <http://www.apache.org/licenses/LICENSE-2.0>
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
#

druid.service=druid/broker
druid.plaintextPort=8082

# HTTP server settings
# Modify this number base on Number of Data instances 350 * 2 instances < 800 in historical
druid.server.http.numThreads=350
druid.server.http.enableRequestLimit = true
druid.server.http.defaultQueryTimeout = 60000
druid.server.http.queueSize = 30000

# HTTP client settings
# Modify this number base on Number of Data instances
druid.broker.http.numConnections=350
druid.broker.http.maxQueuedBytes=15MiB

# Processing threads and buffers
druid.processing.buffer.sizeBytes=1GiB
druid.processing.numMergeBuffers=8
druid.processing.numThreads=4
druid.processing.tmpDir=var/druid/processing

# Query cache disabled -- push down caching and merging instead
druid.broker.cache.useCache=false
druid.broker.cache.populateCache=false
k
ā€¢ Check for segment counts. Have the number of segments increased. ā€¢ If you have metrics dumped somewhere, check jetty metrics and broker metrics to see if you can get more clues. Metrics documentation: https://druid.apache.org/docs/latest/operations/metrics.html#broker ā€¢ Try to isolate the query which is causing the issue. Generally subQueries can cause this. You can also take flame graph of one broker for like 5 minutes to see where the bottle neck is as well. https://support.imply.io/hc/en-us/articles/360033747953-Profiling-Druid-queries-using-flame-graphs
s
great points, thanks a lot @Karan Kumar! very helpful - we will try it out and update! šŸ™‡
b
Excellent advice. Although I would have though segment counts would affect coordinator and historical performance more than broker, but I could be wrong. Always good to compact and keep counts down where possible.
s
yeah, segments counts would impact historical performance and query would just merge the responses.. so lesser segment count would be better right?
b
yes ā€¦. faster queries due to less segments to scan
k
broker also keeps the segmentId, rowSignature -> historical in memory, so that also eats up broker heap.
s
understood.. so more the segments, more the broker heap usage?
b
yes ā€¦.as broker needs to keep track of segments in all historicals
šŸ™ 1
s
Sorry one more question: We are thinking to scale our data nodes from 8 to 12.. does that mean we need to scale the query nodes too vertically? The reason is each query node needs to connect to all data nodes right, so need more processing power? So, when data nodes are horizontally scaled, scale query nodes vertically is correct understanding?
b
do check this out : https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html#number-of-brokers
So, when data nodes are horizontally scaled, scale query nodes vertically is correct understanding?
Yes, we would look at the 
numThreads
and 
mergeBuffers
for groupBy queries etc
šŸ‘ 1
k
As brandon pointed out, you really donot need to scale the query servers for adding 4 more historicals
2 Views
Open in Slack
PreviousNext
Powered by