Apache Pinot

Screenshot 2022-10-03 at 8.03.16 PM.png

Hi , could anyone please help me to understand why pinot is not emitting metrics. I am following this doc <https://docs.pinot.apache.org/operators/tutorials/monitor-pinot-using-prometheus-and-grafana>

metrics.txt

this the &lt;ip&gt;:8008/metrics which i downloaded from browser. Most of the values are 0 in the attached file. currently only 2 metrics are visible i.e. controller leader and realtime table count. Thanks in advance.

Hi <@U043HAVTLM7>, charts in Grafana will not show up until they are emitted. So this might explain why you may only be seeing a few metrics initially.

So if you ingest some data/run some queries, you'll probably see more activity in the Grafana dashboard

Screenshot 2022-10-03 at 10.43.06 PM.png

I have ingested i table already with 100k records.

Hi <@UKH768C4D>, grafana is more like this now.

<@U043HAVTLM7>, did you check if the jmx metrics are available? `You can port forward port 8008 to local and access metrics though: <http://localhost:8008/metrics>`

If you see the metrics in jmx and not prometheus, it could be related to the prometheus config definition or the grafana template

<@UKH768C4D>, yes I can see the metrics there, but most of them are 0. moreover, I see metrics only of Type Controller are visible. Please refer to the metrics.txt file I have attached above.

<@U043HAVTLM7> did you try port forwarding on the other Pinot components? E.g if you port forward on the broker, you should see the broker metrics

do you mean this <@UKH768C4D> (for server)=&gt; jvmOpts: "-javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent-0.12.0.jar=8008:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml -Xms4G -Xmx12G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:file=/opt/pinot/gc-pinot-server.log -Dlog4j2.configurationFile=/opt/pinot/conf/log4j2.xml -Dplugins.dir=/opt/pinot/plugins -Dhelixmanager.waitForConnectedTimeout=120000 -Dzk.session.timeout=120000"
(for broker)=&gt; jvmOpts: "-javaagent:/opt/pinot/etc/jmx_prometheus_javaagent/jmx_prometheus_javaagent-0.12.0.jar=8008:/opt/pinot/etc/jmx_prometheus_javaagent/configs/pinot.yml  -Xms256M -Xmx1G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:file=/opt/pinot/gc-pinot-broker.log"

<@U043HAVTLM7> if you port forward to your broker, do you see any of the broker metrics being emitted?
```kubectl port-forward broker-pod-name 8008:8008```

yes, I can see the broker metrics, thanks a lot <@UKH768C4D>

but, is there a way I can get all the metrics on &lt;pinot-external-ip&gt;:8008 ?

Most of the metrics are visible now except for the cpu used and jvm used.

Sorry i missed this <@U043HAVTLM7>. Each pinot component is emitting its own set of metrics on the configured port. But it prometheus' job to scape these metrics and aggregate it. So you need to talk to prometheus for getting the consolidated metrics.