This message was deleted Apache Druid #troubleshooting

Join Slack

This message was deleted.

# troubleshooting

Slackbot

04/02/2023, 2:07 PM

This message was deleted.

Park Hoon

04/02/2023, 2:09 PM

Here is MM configuration for emitter and monitors.

Copy code

druid_indexer_fork_property_druid_emitter: prometheus
    druid_indexer_fork_property_druid_emitter_prometheus_strategy: pushgateway
    druid_indexer_fork_property_druid_emitter_prometheus_pushGatewayAddress: ...
    druid_indexer_fork_property_druid_monitoring_monitors:
      '
        [
          "org.apache.druid.java.util.metrics.JvmMonitor", 
          "org.apache.druid.java.util.metrics.JvmCpuMonitor"
        ]
      '

Amatya Avadhanula

04/03/2023, 11:58 AM

Hi, the streaming lag metrics are emitted by the Overlord

Sanjay

04/03/2023, 2:56 PM

you may include those monitors as part of

common.runtime.properties

Park Hoon

04/03/2023, 4:02 PM

Hi @Amatya Avadhanula, @Srinivas Narava Thanks for the answer. I am running the overload inside coordinator. Which monitor should I set in the coordinator configuration?

Park Hoon

04/04/2023, 12:06 AM

Currently, I did set these monitors for coordinator.

Copy code

druid_monitoring_monitors:
      '
        [
          "org.apache.druid.java.util.metrics.JvmMonitor", 
          "org.apache.druid.java.util.metrics.JvmCpuMonitor",
          "org.apache.druid.server.metrics.TaskCountStatsMonitor"
        ]
      '

Amatya Avadhanula

04/04/2023, 6:20 AM

Do you see

task/action/run/time

as one of the metrics from the overlord? If not, it's possible that your overlord is not emitting metrics

Park Hoon

04/04/2023, 7:00 AM

@Amatya Avadhanula I can’t see the task_action_run_time metric from coordinator prometheus endpoint (:9090) I will attach the whole metrics.

Park Hoon

04/04/2023, 7:00 AM

Should I run overlord separately in case of kubernetes environment?

Amatya Avadhanula

04/04/2023, 7:06 AM

It'd be great if you could try that. It appears that the metrics here probably belong only to the coordinator and not the overlord

Park Hoon

04/04/2023, 7:08 AM

Okay, will try it.

Park Hoon

04/04/2023, 7:52 AM

I executed overlord service separately but still not getting the

task/action/run/time

metrics from overlord.

Park Hoon

04/04/2023, 7:52 AM

I attached overlord metrics.

overlord-metrics

Park Hoon

04/04/2023, 7:56 AM

Seems kinesis indexing service doesn’t report lag

Copy code

{
  "0": {
    "index_kinesis_track_kinesis_r0_60d0414dfa8c05d_nhpoiddd": {
      "movingAverages": {
        "buildSegments": {
          "5m": {
            "processed": 0.33287108369807955,
            "unparseable": 0,
            "thrownAway": 0,
            "processedWithError": 0
          },
          "15m": {
            "processed": 0.6930406200864409,
            "unparseable": 0,
            "thrownAway": 0,
            "processedWithError": 0
          },
          "1m": {
            "processed": 0.00408677143846408,
            "unparseable": 0,
            "thrownAway": 0,
            "processedWithError": 0
          }
        }
      },
      "totals": {
        "buildSegments": {
          "processed": 5,
          "processedWithError": 0,
          "thrownAway": 0,
          "unparseable": 0
        }
      }
    }
  }
}

3 Views

Open in Slack

Previous Next