I am setting up prometheus to monitor pinot some metrices ar Apache Pinot #troubleshooting

I am setting up prometheus to monitor pinot, some ...

Saumya Upadhyay

04/26/2022, 9:31 AM

I am setting up prometheus to monitor pinot, some metrices are not coming up, guessing it might be issue with setting up scrape configs , please tell where we need to add this config in which file :

Copy code

controller:
  ...
  service:
    annotations:
      "<http://prometheus.io/scrape|prometheus.io/scrape>": "true"
      "<http://prometheus.io/port|prometheus.io/port>": "8008"
  ...
  podAnnotations:
    "<http://prometheus.io/scrape|prometheus.io/scrape>": "true"
    "<http://prometheus.io/port|prometheus.io/port>": "8008"

Richard Startin

04/26/2022, 9:38 AM

which metrics are not showing up?

Richard Startin

04/26/2022, 9:39 AM

the way we configure this in managed deployments is to pass the name of the rules file to the prometheus agent

Saumya Upadhyay

04/26/2022, 9:39 AM

These are some which we need to see if segments are going offline, and really wanted to know if there some OOM or resource related issues.

Richard Startin

04/26/2022, 9:42 AM

yes, clearly, prometheus isn't configured properly

Saumya Upadhyay

04/26/2022, 9:45 AM

We followed this document (https://docs.pinot.apache.org/operators/tutorials/monitor-pinot-using-prometheus-and-grafana), only part where we missed is adding scrape metrices, can you please tell where we should add this scrape annotations configs.

Richard Startin

04/26/2022, 9:47 AM

we set

Copy code

-javaagent:${PINOT_HOME}/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent.jar=${PORT}:${PINOT_HOME}/etc/jmx_prometheus_javaagent/configs/{server|broker|controller|minion|pinot}.yml

Richard Startin

04/26/2022, 9:48 AM

it looks to me like you haven't added the javaagent

Saumya Upadhyay

04/26/2022, 10:16 AM

We are running it in EKS, and we are giving this javaagent in controller statefullsets :

Copy code

env:
            - name: JAVA_OPTS
              value: >-
                -Xms256M -Xmx1G -XX:+UseG1GC -XX:MaxGCPauseMillis=200
                -Xlog:gc*:file=/opt/pinot/gc-pinot-controller.log
                -Dlog4j2.configurationFile=/opt/pinot/conf/log4j2.xml
                -Dplugins.dir=/opt/pinot/plugins
                -javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent.jar=8008:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml

Saumya Upadhyay

04/26/2022, 12:00 PM

Thanks @User for your quick response, we got to manage to resolve this in EKS pod so we need to add that javaagent for every pod and also annotations. But not sure why pinot server taking so much memory. @

Richard Startin

04/26/2022, 1:15 PM

how much memory are you expecting it to take?

Open in Slack

Previous Next