https://pinot.apache.org/ logo
Join Slack
Powered by
Hi, Prometheus/grafana and pinot is deployed in tw...
# troubleshooting
s
Hi, Prometheus/grafana and pinot is deployed in two different namespace. Can we integrate that two and monitor?? We are not getting pinot log metrics in Prometheus. @Xiang Fu @Mayank
d
That should work out of the box.
What is your scrape configuration in Prometheus?
Are the pinot endpoints registered?
s
pinot is not there is scrape configuration.
d
Is your prometheus configured to listen to k8s annotations to discover scraping configs?
s
I have added service:     annotations:        "prometheus.io/scrape": "true"       "prometheus.io/port": "8008"
according to docs
d
That is for Pinot
But is Prometheus aware is needs to listens to k8s annotations?
Also
service.annotations
is not the spot for your annotations
s
But is Prometheus aware is needs to listens to k8s annotations? Can you helm me with this?
d
I can’t helm you 🙂
Only help 🤣
s
yeah😃
d
Are the annotations present on your pinot pods?
s
podAnnotations:      "prometheus.io/scrape": "true"     "prometheus.io/port": "8008"
d
Please show the output of 
kubectl describe pod -o yaml <pod-name>
s
desc-cont-0.txt
desc-cont-0.txt
d
labels are fine
Can you share your prometheus configuration file?
s
let me get that
Is describe pod works??
prometheus is deployed by someone else. will take time to get files.
d
That prometheus needs to be configured to look up for the pod annotations
The configuration is not part of the pod definition.
your prometheus SRE should know how to enable that
s
let me check
Thanks Daniel
after adding JVMopts config for jmx_prometheus_javaagent in values.yaml 
pinot-controller            ClusterIP      115.12.45.123   <none>          9000/TCP                     27h
Pinot controller should have 8008 port also?? @Daniel Lavoie
In the documentation https://docs.pinot.apache.org/operators/tutorials/monitor-pinot-using-prometheus-and-grafana This point is not clear 
You can port forward port 8008 to local and access metrics though: <http://localhost:8008/metrics>
d
In this tutorial, pinot is running in Kubernetes, so 
kubectl port-forward
to confirm the endpoint is available
s
yes we are running in kubernetes only Does this commend looks correct to you?? 
kubectl port-forward service/pinot-controller 8008:9000
d
No
s
ok Then
can you send me the CMD?
d
kubectl port-forward <pod-name> 8008:8008
s
@Mayank
kubectl port-forward service/pinot-controller 8008:8008
error: Service pinot-controller does not have a service port 8008
@Daniel Lavoie
d
What is your helm value for 
prometheusExporter.enabled
?
s
where this could be found?
d
Nevermind, it’s not in the oss helm chart.
s
ok
d
Yeah, as is, the helm chart needs to be customized to expose the prometheus exporter port
s
can you help me with adding 
prometheusExporter.enabled
d
It more than that, the whole helm chart needs to be customized
s
ooh
Thanks for this article @Daniel Lavoie https://medium.com/apache-pinot-developer-blog/monitoring-apache-pinot-99034050c1a5 This helped me a lot to debug.
Open in Slack
PreviousNext
Powered by