Hi Prometheus grafana and pinot is deployed in two different

Question

Hi Prometheus grafana and pinot is deployed in two different namespace Can we integrate that two and monitor We are not getting pinot log metrics in Prometheus < Xiang Fu> < Mayank>

Daniel Lavoie · Answer

That should work out of the box

Daniel Lavoie · Answer

What is your scrape configuration in Prometheus

Daniel Lavoie · Answer

Are the pinot endpoints registered

Shailesh Jha · Answer

pinot is not there is scrape configuration

Daniel Lavoie · Answer

Is your prometheus configured to listen to k8s annotations to discover scraping configs

Shailesh Jha · Answer

I have added service annotations true 8008

Shailesh Jha · Answer

according to docs

Daniel Lavoie · Answer

That is for Pinot

Daniel Lavoie · Answer

But is Prometheus aware is needs to listens to k8s annotations

Daniel Lavoie · Answer

Also

Daniel Lavoie · Answer

`service annotations` is not the spot for your annotations

Shailesh Jha · Answer

But is Prometheus aware is needs to listens to k8s annotations Can you helm me with this

Daniel Lavoie · Answer

I can t helm you slightly smiling face

Daniel Lavoie · Answer

Only help rolling on the floor laughing

Shailesh Jha · Answer

yeah smiley

Daniel Lavoie · Answer

Are the annotations present on your pinot pods

Shailesh Jha · Answer

podAnnotations true 8008

Daniel Lavoie · Answer

Please show the output of `kubectl describe pod o yaml lt pod name gt `

Daniel Lavoie · Answer

labels are fine

Daniel Lavoie · Answer

Can you share your prometheus configuration file

Shailesh Jha · Answer

let me get that

Shailesh Jha · Answer

Is describe pod works

Shailesh Jha · Answer

prometheus is deployed by someone else will take time to get files

Daniel Lavoie · Answer

That prometheus needs to be configured to look up for the pod annotations

Daniel Lavoie · Answer

The configuration is not part of the pod definition

Daniel Lavoie · Answer

your prometheus SRE should know how to enable that

Shailesh Jha · Answer

let me check

Shailesh Jha · Answer

Thanks Daniel

Shailesh Jha · Answer

after adding JVMopts config for jmx prometheus javaagent in values yaml `pinot controller ClusterIP 115 12 45 123 lt none gt 9000 TCP 27h` Pinot controller should have 8008 port also < Daniel Lavoie>

Shailesh Jha · Answer

In the documentation This point is not clear `You can port forward port 8008 to local and access metrics though `

Daniel Lavoie · Answer

In this tutorial pinot is running in Kubernetes so `kubectl port forward ` to confirm the endpoint is available

Shailesh Jha · Answer

yes we are running in kubernetes only Does this commend looks correct to you `kubectl port forward service pinot controller 8008 9000`

Daniel Lavoie · Answer

No

Shailesh Jha · Answer

ok Then

Shailesh Jha · Answer

can you send me the CMD

Daniel Lavoie · Answer

`kubectl port forward lt pod name gt 8008 8008`

Shailesh Jha · Answer

< Mayank>

Shailesh Jha · Answer

`kubectl port forward service pinot controller 8008 8008` `error Service pinot controller does not have a service port 8008` < Daniel Lavoie>

Daniel Lavoie · Answer

What is your helm value for `prometheusExporter enabled`

Shailesh Jha · Answer

where this could be found

Daniel Lavoie · Answer

Nevermind it s not in the oss helm chart

Shailesh Jha · Answer

ok

Daniel Lavoie · Answer

Yeah as is the helm chart needs to be customized to expose the prometheus exporter port

Shailesh Jha · Answer

can you help me with adding `prometheusExporter enabled`

Daniel Lavoie · Answer

It more than that the whole helm chart needs to be customized

Shailesh Jha · Answer

ooh

Shailesh Jha · Answer

Thanks for this article < Daniel Lavoie> This helped me a lot to debug