https://linen.dev logo
Join Slack
Powered by
Hi there: We have been trying to improve druid pe...
# general
b
Hi there: We have been trying to improve druid performance. Historical: Resources: - 5 Nodes: 10CPU x 36G MeM x 1TB of Disk (each) Disk usage: ~ 450GB each Overall config:
Copy code
nodeConfigMountPath: "/opt/druid/conf/druid/cluster/data/historical"
replicas: 5
runtime.properties: |
 druid.service=druid/historical
 # druid.plaintextPort=8083

 # HTTP server threads
 druid.server.http.numThreads=50

 # Processing threads and buffers
 druid.processing.buffer.sizeBytes=1024MiB
 druid.processing.numMergeBuffers=2
 druid.processing.numThreads=7
 druid.processing.tmpDir=var/druid/processing

 # Segment storage
 druid.segmentCache.locations=[{"path":"var/druid/segment-cache","maxSize":"800g"}]

 # Query cache
 druid.historical.cache.useCache=true
 druid.historical.cache.populateCache=true
 druid.cache.type=caffeine
 druid.cache.sizeInBytes=3000MiB

 druid.monitoring.monitors=["org.apache.druid.server.metrics.EventReceiverFirehoseMonitor","org.apache.druid.client.cache.CacheMonitor","org.apache.druid.java.util.metrics.JvmMonitor","org.apache.druid.java.util.metrics.JvmCpuMonitor","org.apache.druid.java.util.metrics.CpuAcctDeltaMonitor","org.apache.druid.java.util.metrics.JvmThreadsMonitor","org.apache.druid.server.metrics.QueryCountStatsMonitor","org.apache.druid.server.metrics.SubqueryCountStatsMonitor","org.apache.druid.server.metrics.HistoricalMetricsMonitor","org.apache.druid.server.metrics.SegmentStatsMonitor","org.apache.druid.server.metrics.ServiceStatusMonitor"]
extra.jvm.options: |-
 -Xms8g
 -Xmx8g
 -XX:MaxDirectMemorySize=32g
resources:
 limits:
  cpu: 10
  memory: 33Gi
 requests:
  cpu: 9
  memory: 28Gi
The main Datasource we query its investing data from Kafka: • Datasource name: event • Availability: Fully available (5,904 segments) • Historical load/drop queues: No segments to load/drop • Total data size: 550.32 GB • Running tasks: index_kafka: 20 • Segment rows minimum / average / maximum: 0.052 M / 2.849 M / 4.217 M • Total rows: 16,776,938,822 • Avg. row size (bytes): 32 • Replicated size: 1.10 TB • Compaction: Fully compacted (except the last PT1H of data, 423 segments skipped) • % Compacted bytes / segments / intervals: 100.00% / 100.00% / 100.00% • Left to be compacted: 0.00 B in my ingestion granularitySpec, I have this config: "granularitySpec": { "type": "uniform", "segmentGranularity": "DAY", "queryGranularity": "HOUR", "rollup": true, "intervals": [] }, We are running some aggregations
And our queries seem to be intermittent, We see it works fast and also getting timeouts we are not sure our configs are correct or is there something to do with ingestion or query, model adjustments
We are really struggling with it, if someone has any tip, advice,
Im running a group by with in clause
Copy code
SELECT orgId, signature, severity, "log.action" as logAction, "source.name" as sourceName, message, SUM("count") as total
FROM "event"
WHERE signature IS NOT NULL AND severity IS NOT NULL AND "log.action" is not null AND "source.name" is not null AND message is not NULL AND
"__time" between  '2024-09-21' and '2024-11-17 23:59:59' and
orgId in ('sdfsdf','dasdasd,'.... )
GROUP BY orgId, signature, severity, "log.action", "source.name", message
ORDER BY 6 DESC
LIMIT 25
OFFSET 0
use most CPU and starving resources
j
For a 10 vCPU Historical. that runs a lot of Group By queries, you could up processing.numThreads to 9 as the rule of thumb. Also check clarity metric 
mergeBuffer/pendingRequests
... if you are seeing any then you can try increasing processing.numMergeBuffers to 5-10 as well.
If that works on the Historical level, then suggest also inrease numMergeBuffers on Broker as well.
Open in Slack
PreviousNext
Powered by