Hi, Can someone please share the process to activate metrics using prometheus emitter. What I have done so far: 1. Added prometheus emitter in download process in Dockerfile so that extension package should be there. 2. In Configuration file (config Map of K8), I have added prometheus emitter in extension load list. Also added

druid.emitter.prometheus.port=4321

druid.emitter=prometheus

druid.emitter.prometheus.strategy=exporter

3. Now When I try to curl on <<pod.address>>:4321/ I don’t see anything. I get 500: Bad gateway. Please support

Hi, please check the logs to identify any issues during the activation of the monitors (druid.monitoring.monitors). Keep only the minimum recommended monitors for the specific service and restart. As far as I remember, it might be necessary to obtain additional access for certain monitors in Kubernetes (K8s).

Below are the logs. Can you suggest any solution.

[ansible@ip-10-20-27-128 ~]$ kubectl logs druid-router-2 | grep prometheus
Defaulted container "druid-router" out of: druid-router, vault-agent, router-init (init), vault-agent-init (init)
{"instant":{"epochSecond":1722259881,"nanoOfSecond":736179000},"thread":"main","level":"INFO","loggerName":"org.apache.druid.guice.ExtensionsLoader","message":"Loading extension [prometheus-emitter], jars: prometheus-emitter-30.0.0.jar, simpleclient-0.16.0.jar, simpleclient_common-0.16.0.jar, simpleclient_httpserver-0.16.0.jar, simpleclient_pushgateway-0.16.0.jar, simpleclient_tracer_common-0.16.0.jar, simpleclient_tracer_otel-0.16.0.jar, simpleclient_tracer_otel_agent-0.16.0.jar","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","contextMap":{},"threadId":1,"threadPriority":5}
{"instant":{"epochSecond":1722259882,"nanoOfSecond":323246000},"thread":"main","level":"INFO","loggerName":"org.apache.druid.guice.ExtensionsLoader","message":"Loading extension [prometheus-emitter], jars: prometheus-emitter-30.0.0.jar, simpleclient-0.16.0.jar, simpleclient_common-0.16.0.jar, simpleclient_httpserver-0.16.0.jar, simpleclient_pushgateway-0.16.0.jar, simpleclient_tracer_common-0.16.0.jar, simpleclient_tracer_otel-0.16.0.jar, simpleclient_tracer_otel_agent-0.16.0.jar","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","contextMap":{},"threadId":1,"threadPriority":5}
{"instant":{"epochSecond":1722259884,"nanoOfSecond":407638000},"thread":"main","level":"INFO","loggerName":"org.apache.druid.guice.JsonConfigurator","message":"Skipping druid.emitter.prometheus.strategy property: one of it's prefixes is also used as a property key. Prefix: druid","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","contextMap":{},"threadId":1,"threadPriority":5}
{"instant":{"epochSecond":1722259884,"nanoOfSecond":408143000},"thread":"main","level":"INFO","loggerName":"org.apache.druid.guice.JsonConfigurator","message":"Skipping druid.emitter.prometheus.port property: one of it's prefixes is also used as a property key. Prefix: druid","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","contextMap":{},"threadId":1,"threadPriority":5}
{"instant":{"epochSecond":1722259884,"nanoOfSecond":513481000},"thread":"main","level":"INFO","loggerName":"org.apache.druid.emitter.prometheus.Metrics","message":"Using default metric configuration","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","contextMap":{},"threadId":1,"threadPriority":5}
{"instant":{"epochSecond":1722259884,"nanoOfSecond":601119000},"thread":"main","level":"INFO","loggerName":"org.apache.druid.server.emitter.EmitterModule","message":"Using emitter [org.apache.druid.emitter.prometheus.PrometheusEmitter@60c96eb4] for metrics and alerts, with dimensions [{version=30.0.0}].","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","contextMap":{},"threadId":1,"threadPriority":5}