This message was deleted Apache Druid #troubleshooting

Join Slack

This message was deleted.

# troubleshooting

Slackbot

12/28/2022, 4:21 AM

This message was deleted.

Brandon Tan

12/28/2022, 4:23 AM

query/segments/count

Copy code

This metric is not enabled by default. See the QueryMetrics Interface for reference regarding enabling this metric.

Sachidananda

12/28/2022, 4:24 AM

query/segments/count

is different i am talking about

segment/count

Brandon Tan

12/28/2022, 4:37 AM

Have you tried these 2 monitors:

Copy code

org.apache.druid.server.metrics.HistoricalMetricsMonitor

org.apache.druid.server.metrics.SegmentStatsMonitor

Sachidananda

12/28/2022, 4:42 AM

Copy code

org.apache.druid.server.metrics.HistoricalMetricsMonitor

this i have added in historical nodes and in druid code i can see HistoricalMetricsMonitor.java:104 - its emitting

segment/count

but it is not shown in grafana and others like

segment/used, segment/usedPercent

are in same file, these are getting emitted and i can see in grafana

Brandon Tan

12/28/2022, 4:43 AM

so you are good now ?

Sachidananda

12/28/2022, 4:45 AM

no no i mean all others are emitted except

segment/count

Brandon Tan

12/28/2022, 4:46 AM

Can you try to add this :

Copy code

org.apache.druid.server.metrics.SegmentStatsMonitor

Sachidananda

12/28/2022, 4:47 AM

sure let me try and update

Sachidananda

12/28/2022, 5:10 AM

it seems we are on 0.22.1 and this monitor SegmentStatsMonitor is not present in this version however i can see as per code this file HistoricalMetricsMonitor.java:104 - its emitting

segment/count

but not sure why its not showing up in grafana

Brandon Tan

12/28/2022, 5:25 AM

Try to stand up 24.0.2

Suneet Saldanha

12/29/2022, 3:35 PM

Which emitter are you using? Some emitters include filter lists that limit which metrics and dimensions are emitted by default. This is the list for the prometheus emitter for example https://github.com/apache/druid/blob/master/extensions-contrib/prometheus-emitter/src/main/resources/defaultMetrics.json

Sachidananda

12/30/2022, 4:25 AM

Hi @Suneet Saldanha Thanks a lot for your response I am using prometheus emitter. I can see

"coordinator_segment/count"

"historical_segment/count"

in default list but i don't see these appearing in grafana Can you tell me what to search for finding these 2 metrics?

Suneet Saldanha

01/05/2023, 3:26 AM

hi @Sachidananda, happy new year!

Suneet Saldanha

01/05/2023, 3:27 AM

Sorry for the delayed response. I'm not very familiar with the prometheus extension, but reading the code it looks like maybe it's not correctly filtering this metric on the emitter side

Suneet Saldanha

01/05/2023, 3:28 AM

I'd recommend just changing the list to include

segment/count

instead of trying to specify different dimensions based on the service that is reporting the

segment/count

Suneet Saldanha

01/05/2023, 3:28 AM

ie change the last 2 lines to

Copy code

"segment/count" : { "dimensions" : ["dataSource"], "type" : "gauge", "help": "Number of served segments."},

and see if that works

Suneet Saldanha

01/05/2023, 3:33 AM

this was a contrib extension, so it's possible there may be a bug with how it;s trying to detect service specific metrics. I noticed the documentation for the prometheus emitter mentions that the service and the metric should be separated by a

but instead the defaultDimensions use

See the last 2 lines in https://github.com/apache/druid/blob/master/docs/development/extensions-contrib/prometheus.md#metric-mapping

Sachidananda

01/06/2023, 5:05 AM

Thanks @Suneet Saldanha for the detailed response will do some trail and error

Open in Slack

Previous Next