https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
s
There was another thread that was reporting something similar. In their case they had just upgraded to 25.0 which defaults to HTTP based communication between processes instead of ZK. This causes more work on the http threads. The solution was to increase http threads to accommodate this work. Just throwing it out there to see if this might help.
g
note also that a common cause for 137 errors is overcommitting the memory on the server in Druid 25 (the latest release) you can start druid via the 
bin/start-druid
script and it will automatically determine reasonable configs given the available resources on the server
s
@Gian Merlino We faced this issue due to the znode size reached in the zookeper by default it is 1MB we increased it to 12MB using the max jute buffer and then it got fixed
@Gian Merlino can you please suggest the best documentation or approach for setting up the druid monitoring. Because we are settuped druid exporter right now but it was emitting huge number of metrics and it was bombarding our prometheus and grafana
@Gian Merlino if you can provide me any design document for implementing the monitoring it will be really helpful
g
Personally I like using Druid itself to monitor Druid. Druid itself can handle the number of metrics that Druid generates 🙂
The idea would be use the Kafka emitter, and then set up Druid to read that data back from Kafka. This is more or less what we do at Imply. (We have some extra stuff in the pipeline but it's basically that idea.)
If you want to keep using prometheus/grafana then I believe the other emitters have some ability to control what metrics and dimensions get emitted. You could remove some of those to reduce the load
s
okay
@Gian Merlino This is my common configuration file... can you please suggest me any corrections i need to make to get the metric count less.
Copy code
# Extensions
    druid.extensions.directory=/opt/druid/extensions
    druid.extensions.hadoopDependenciesDir=/opt/druid/hadoop-dependencies
    druid.extensions.loadList=["druid-kafka-indexing-service","druid-s3-extensions","postgresql-metadata-storage","druid-datasketches","druid-lookups-cached-global","druid-parquet-extensions","druid-avro-extensions","kafka-emitter"]

    # Zookeeper
    druid.zk.paths.base=/druid
    druid.zk.service.host=${ZOOKEEPER_SERVER}

    # Logging
    druid.emitter=composing
    druid.emitter.logging.logLevel=debug
    druid.startup.logging.logProperties=true

    #http
    # druid.emitter=http
    druid.emitter.http.recipientBaseUrl=<http://druid-http-exporter.cloudops.svc.cluster.local:8080/druid>

    # For PostgreSQL (make sure to additionally include the Postgres extension):
    druid.metadata.storage.type=postgresql
    druid.metadata.storage.connector.connectURI=jdbc:postgresql://${POSTGRES_URL}:${POSTGRES_PORT}/${POSTGRES_DB}
    druid.metadata.storage.connector.user=${POSTGRES_USER}
    druid.metadata.storage.connector.password=${POSTGRES_PASSWORD}

    # Service discovery
    druid.selectors.indexing.serviceName=druid/overlord
    druid.selectors.coordinator.serviceName=druid/coordinator

    # Indexing service logs
    druid.indexer.logs.type=s3
    druid.indexer.logs.s3Bucket=${BUCKET_INDEX}
    druid.indexer.logs.s3Prefix=indexing-logs

    # Monitoring
    druid.monitoring.monitors=["org.apache.druid.java.util.metrics.JvmMonitor"]

    # Deep Storage
    druid.storage.type=s3
    druid.storage.bucket=${BUCKET_STORAGE}
    druid.storage.baseKey=segments
    druid.storage.useS3aSchema=True

    # Security
    druid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password"]

    # SQL
    druid.sql.enable=true
    #Enable Metrics
    druid.monitoring.emissionPeriod=PT15S
we are using druid exporter to get the metrics from druid and push it to prometheus
g
I don't see your full emitter config there; you're using 
composing
emitter so there should be some 
druid.emitter.composing.emitters
and then emitter-specific config. is it possible something is missing?
s
Your druid.monitoring.emissionPeriod is set to every 15 seconds. The default is one minute. That would reduce it to 25% for time based metrics.
s
@Sergio Ferragut can we put more than 1 minute as well is it ?
s
Yes. You can determine the cadence for many of the metrics. Metrics that are query specific will be emitted when the query is processed, so those will not be affected by the 
druid.monitoring.emissionPeriod
.
s
@Sergio Ferragut Even after increasing the emissionperiod to 60s still it is emitting of same size
I was thinking even though the druid has metrics it will only emit every one minute... so here after 1 minute druid is emitting all the metrics at once
does metrics count depends on segment count as well ?
s
it varies by metric, some metrics emit more instances based on individual queries and segments involved. So yes, take a look here: https://druid.apache.org/docs/latest/operations/metrics.html#historical The dimensions associated to each metric define their granularity, and yes in many cases 
segment
is a dimension, so the metric will be emitted for each segment involved.
3 Views
Open in Slack
PreviousNext
Powered by