https://pinot.apache.org/ logo
s

Shailesh Jha

05/24/2021, 2:13 PM
Hi, Prometheus/grafana and pinot is deployed in two different namespace. Can we integrate that two and monitor?? We are not getting pinot log metrics in Prometheus. @Xiang Fu @Mayank
d

Daniel Lavoie

05/24/2021, 2:14 PM
That should work out of the box.
What is your scrape configuration in Prometheus?
Are the pinot endpoints registered?
s

Shailesh Jha

05/24/2021, 2:17 PM
pinot is not there is scrape configuration.
d

Daniel Lavoie

05/24/2021, 2:18 PM
Is your prometheus configured to listen to k8s annotations to discover scraping configs?
s

Shailesh Jha

05/24/2021, 2:19 PM
I have added service:     annotations:        "prometheus.io/scrape": "true"       "prometheus.io/port": "8008"
according to docs
d

Daniel Lavoie

05/24/2021, 2:19 PM
That is for Pinot
But is Prometheus aware is needs to listens to k8s annotations?
Also
service.annotations
is not the spot for your annotations
s

Shailesh Jha

05/24/2021, 2:21 PM
But is Prometheus aware is needs to listens to k8s annotations? Can you helm me with this?
d

Daniel Lavoie

05/24/2021, 2:22 PM
I can’t helm you 🙂
Only help 🤣
s

Shailesh Jha

05/24/2021, 2:22 PM
yeah😃
d

Daniel Lavoie

05/24/2021, 2:22 PM
Are the annotations present on your pinot pods?
s

Shailesh Jha

05/24/2021, 2:23 PM
podAnnotations:      "prometheus.io/scrape": "true"     "prometheus.io/port": "8008"
d

Daniel Lavoie

05/24/2021, 2:23 PM
Please show the output of
kubectl describe pod -o yaml <pod-name>
s

Shailesh Jha

05/24/2021, 2:26 PM
d

Daniel Lavoie

05/24/2021, 2:28 PM
labels are fine
Can you share your prometheus configuration file?
s

Shailesh Jha

05/24/2021, 2:31 PM
let me get that
Is describe pod works??
prometheus is deployed by someone else. will take time to get files.
d

Daniel Lavoie

05/24/2021, 2:46 PM
That prometheus needs to be configured to look up for the pod annotations
The configuration is not part of the pod definition.
your prometheus SRE should know how to enable that
s

Shailesh Jha

05/24/2021, 2:54 PM
let me check
Thanks Daniel
after adding JVMopts config for jmx_prometheus_javaagent in values.yaml
pinot-controller            ClusterIP      115.12.45.123   <none>          9000/TCP                     27h
Pinot controller should have 8008 port also?? @Daniel Lavoie
In the documentation https://docs.pinot.apache.org/operators/tutorials/monitor-pinot-using-prometheus-and-grafana This point is not clear
You can port forward port 8008 to local and access metrics though: <http://localhost:8008/metrics>
d

Daniel Lavoie

05/25/2021, 12:17 PM
In this tutorial, pinot is running in Kubernetes, so
kubectl port-forward
to confirm the endpoint is available
s

Shailesh Jha

05/25/2021, 12:23 PM
yes we are running in kubernetes only Does this commend looks correct to you??
kubectl port-forward service/pinot-controller 8008:9000
d

Daniel Lavoie

05/25/2021, 12:35 PM
No
s

Shailesh Jha

05/25/2021, 12:36 PM
ok Then
can you send me the CMD?
d

Daniel Lavoie

05/25/2021, 1:15 PM
kubectl port-forward <pod-name> 8008:8008
s

Shailesh Jha

05/25/2021, 1:16 PM
@Mayank
kubectl port-forward service/pinot-controller 8008:8008
error: Service pinot-controller does not have a service port 8008
@Daniel Lavoie
d

Daniel Lavoie

05/25/2021, 1:19 PM
What is your helm value for
prometheusExporter.enabled
?
s

Shailesh Jha

05/25/2021, 1:20 PM
where this could be found?
d

Daniel Lavoie

05/25/2021, 1:20 PM
Nevermind, it’s not in the oss helm chart.
s

Shailesh Jha

05/25/2021, 1:23 PM
ok
d

Daniel Lavoie

05/25/2021, 1:23 PM
Yeah, as is, the helm chart needs to be customized to expose the prometheus exporter port
s

Shailesh Jha

05/25/2021, 1:24 PM
can you help me with adding
prometheusExporter.enabled
d

Daniel Lavoie

05/25/2021, 1:24 PM
It more than that, the whole helm chart needs to be customized
s

Shailesh Jha

05/25/2021, 1:24 PM
ooh
Thanks for this article @Daniel Lavoie https://medium.com/apache-pinot-developer-blog/monitoring-apache-pinot-99034050c1a5 This helped me a lot to debug.