Title
n
Nyasha Chigwamba
05/25/2023, 3:06 PMHi, I am trying out Robusta to monitor my vcluster workloads. Are there any guidelines for using Robusta with vcluster(s)
a
arik alon
05/25/2023, 3:12 PMDid you install Robusta inside the vcluster?
n
Nyasha Chigwamba
05/25/2023, 3:14 PMThank you for the reply @arik alon. Initially, I had not, then I couldn't see my vcluster apps. I ended up installing in vcluster.
However, my preference is to avoid installing it in the vcluster
I have now mapped the prometheus-server service from the host cluster to the vcluster. However, I can't see the CPU and Memory graphs of pods and Robusta shows the error: "Cannot display graphs because Prometheus was not found.
Tell Robusta where to find it, or enable our builtin Prometheus."
a
arik alon
05/25/2023, 3:18 PMSo, just to verify, Prometheus and Robusta are installed on the host cluster or vcluster?
n
Nyasha Chigwamba
05/25/2023, 3:19 PMPrometheus install on host (the server service is mapped into the vcluster)
Robusta (now installed on both host and vcluster)
a
arik alon
05/25/2023, 3:20 PMThis is the Robusta bundled Prometheus stack or your own Prometheus?
n
Nyasha Chigwamba
05/25/2023, 3:20 PMMy own
a
arik alon
05/25/2023, 3:20 PMgot it
Can you please share the list of services on the vcluster?
I’d like to see the name and port of the prometheus service
n
Nyasha Chigwamba
05/25/2023, 3:24 PMScreenshot_2023-05-25_at_11_22_42_AM.png
globalConfig:
prometheus_url: "<http://prometheus-server.monitoring.svc.cluster.local>"
alertmanager_url: "<http://prometheus-alertmanager.monitoring.svc.cluster.local:9093>"a
arik alon
05/25/2023, 3:27 PMLooks like you configured it properly
n
Nyasha Chigwamba
05/25/2023, 3:27 PMI can also run curl successfully from robusta runner:
root@robusta-runner-57ccd6cdbb-bc7fj:/app# curl -v <http://prometheus-server.monitoring.svc.cluster.local>
* Trying 10.0.115.126:80...
* Connected to prometheus-server.monitoring.svc.cluster.local (10.0.115.126) port 80 (#0)
> GET / HTTP/1.1
> Host: prometheus-server.monitoring.svc.cluster.local
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Content-Type: text/html; charset=utf-8
< Location: /graph
< Date: Thu, 25 May 2023 15:27:16 GMT
< Content-Length: 29
<
<a href="/graph">Found</a>.
* Connection #0 to host prometheus-server.monitoring.svc.cluster.local left intact
root@robusta-runner-57ccd6cdbb-bc7fj:/app#a
arik alon
05/25/2023, 3:28 PMcan you share the logs from the robusta-runner on the vcluster?
the list of services, and the configuration are both on the vcluster, right?
n
Nyasha Chigwamba
05/25/2023, 3:29 PMcorrect
a
arik alon
05/25/2023, 3:31 PMcan you please share the logs from the robusta-runner inside the vclluster?
n
Nyasha Chigwamba
05/25/2023, 3:33 PMrobusta-runner.log
a
arik alon
05/25/2023, 3:35 PMLooks like there’s a valid connection to Prometheus
from the UI, are you checking cpu and memory from this cluster:
dev-uksouth-primary.dev-referenceWhere do you see the error?
n
Nyasha Chigwamba
05/25/2023, 3:35 PMRobusta UI
Screenshot_2023-05-25_at_11_36_34_AM.png
a
arik alon
05/25/2023, 3:42 PMI think I understand the issue
Robusta is querying Prometheus by pod name and namespace
The pod namespace is the namespace in the vcluster, but I think Prometheus is running on the host, and there’s it’s a different namespace
The error message here is missleading
If we could relabel the
namespaceBut I think the Robusta on the host should be working properly, for the host pods
Is it?
n
Nyasha Chigwamba
05/25/2023, 4:03 PMYou are correct, host pods work fine.
Let me look into relabeling
a
arik alon
05/25/2023, 4:04 PM👍
n
Nyasha Chigwamba
05/25/2023, 4:04 PMThank you
@arik alon I looked into relabeling. However, it was getting a bit tricky. The pod metrics are coming from cadvisor node metrics and being pushed to the metrics server. I could not quickly figure out how to relabel at the time of scraping. I just installed robusta in the vcluster with prometheus enable. This is obvious not cost-effective for me but I can now at least see the CPU graphs for my vcluster workloads
a
arik alon
05/31/2023, 2:01 PMgot it. Thanks for the update