This message was deleted Apache Druid #general

Join Slack

This message was deleted.

# general

Slackbot

03/30/2023, 6:42 AM

This message was deleted.

Kyle Hoondert

03/30/2023, 6:59 AM

JVM metrics should be emitted for individual peon processes, they can be identified by their host:port

Sachidananda

03/30/2023, 7:00 AM

we have prometheus based emitter where 9102 port is used Do we need to open these ports as well for scraping these metrics

Kyle Hoondert

03/30/2023, 7:02 AM

with Prometheus - you will likely need to set up a push-gateway to collect peon metrics

Kyle Hoondert

03/30/2023, 7:02 AM

https://druid.apache.org/docs/latest/development/extensions-contrib/prometheus.html#override-properties-for-peon-tasks

Sachidananda

03/30/2023, 7:04 AM

thanks a lot

Nick M

03/30/2023, 1:00 PM

The push gateway for peon metrics doesn't scale that well in my experience, and there are issues with push metrics being used in this manner (a push gateway never deletes metrics and we've found old metrics still being emitted long after the task has stopped). Maybe it was the scale of peons we had running that caused the push gateway to stop responding though so you might be OK 🤞

🙌 1

Sachidananda

06/12/2023, 10:45 AM

Hi Druid Experts I have promethues metrics enabled with

exporter

strategy for peon metrics I have set the following in my helm chart

Copy code

druid_indexer_fork_property_druid_emitter_prometheus_strategy: "pushgateway"
    druid_indexer_fork_property_druid_emitter_prometheus_pushGatewayAddress: "<http://pushgateway-address>"

how to check for the peon metrics are emitted or not when i submit a task initial log i am seeing

druid_indexer_fork_property_druid_emitter_prometheus_pushGatewayAddress

getting set however this is not getting set in logs

druid_indexer_fork_property_druid_emitter_prometheus_strategy

kindly help. Thanks in advance.

Sergio Ferragut

06/13/2023, 12:10 AM

Are you setting

druid_indexer_fork_property_druid_emitter_prometheus_strategy

pushgateway

Sachidananda

06/13/2023, 4:19 AM

yes i am setting

druid_indexer_fork_property_druid_emitter_prometheus_strategy

in helm chart but in runtime.properties file generated during k8s start up its not present.

druid_indexer_fork_property_druid_emitter_prometheus_pushGatewayAddress

this is present in runtime.properties file

Sergio Ferragut

06/13/2023, 5:03 PM

ok... so that seems like it is missing something in the helm chart templates. I'll take a look. Can you share the middle manager config in your values.yaml?

Sergio Ferragut

06/13/2023, 7:46 PM

The docker image uses the

druid.sh <Service Name>

as the entrypoint for the image, It reads the

env

at pod start up and finds all the

druid_*

environment variables, converts

inro

and writes them into a tmp conf folder that it uses to initiate the service. I tested it with a docker compose with the variable set in the environment file (the equivalent of the

config:

section of the helm chart) and I did see the property in the config file. The location of the config file depends on the startup environment as well. It depends on whether

DRUID_SINGLE_NODE_CONF

is defined in the environment. In my case the file was located at:

/tmp/conf/druid/single-server/micro-quickstart/middleManager

Dinesh pundkar

06/21/2023, 4:55 AM

@Sergio Ferragut Apologies for responding late and Thanks for checking on this !!!

Dinesh pundkar

06/21/2023, 4:55 AM

We will verify one more time from our side and then get back to you

Open in Slack

Previous Next