https://livekit.io logo
Join Slack
Powered by
This message was deleted.
# helpdesk
s
This message was deleted.
m
Hey Jonathan, sometimes things slip through the cracks given how busy the team is. It’s absolutely nothing personal. I don’t want you to feel like you’re not asking appropriate questions, or something like that. Maybe @able-gigabyte-21598 or @great-umbrella-58130 can help with this one?
a
If you want to keep a minimum of 2 available, you can use 
sum(livekit_egress_available) > 1
instead of an average
r
Thanks @dc and @dsa. I’ve seen that comment in the docs. I tried to set this up with my metrics query to Prometheus but ends up returning no values. I also noticed I can set up a custom metric as part of the helm chart. I tried putting that statement there. That also did not work. Where should this be put?
g
Is the metric getting scraped by Prometheus and returning a 0 value, or is it not being scraped and returning no result?
r
With the Boolean expression I get no value can be picked up from the hpa. But doing a kubectl —raw I can see the metric is there but has no values
To get it to work I had a query like this sum(livekit_egress_available) by (<<.Group>>)
But adding the > 2 returns no values
g
Got it, so the raw metric has values, but not when you’re comparing it
r
Yes
g
That makes sense if you have no pods available
r
But I have 2 pods available in my cluster
g
looking into how this works now
r
Thanks 🙏
g
do you know what version of kubernetes you’re running?
I also want to check, are you deploying egress with the helm chart?
r
I’m using the helm chart you guys provided
K8s version 1.21
g
so I think what you’re seeing is correct. the metric value you see in the HPA is the result of dividing the query by the number of pods. so 1 shows that there is 1 available for each of your 2 pods.
r
Ok cool. So now if the number goes down how do I use that to scale?
Knowing that the scale algorithm is current/target=scaling
g
right… the number should be going up. if the number is under the target, it’ll scale down. if its above the target, it’ll scale up
r
I did as an example if I have 2 pods the current will be 1 and if I have a target of 1 to scale pods I need would need livekit_egress_available to go to 2. From the formula 2/1= 1 so scale up but actuall it goes to 500m/1 if one is in use so no scaling is initiated. It seemed to me I need the inverse or livekit_egress_inuse at least to work how I understand it
Hence my confusion
g
I’m honestly just as confused
I’d suggest switching over to the CPU based scaler instead until I can get some clarity on how best to use the custom scaler
r
Ok thanks I hope at least it’s clear what I’m talking about now
Thanks for your help so far 🙏
g
yes, what you said makes sense! thanks for bringing this up!
r
Thank you 😊
a
if you’re using the average I think you can use 
1-livekit_egress_available
so it starts at 0 and goes up
r
In the Prometheus query or hpa object?
a
this will still be percentage based, if you want a constant number you’d need something like 
{total_instances} - sum(livekit_egress_available)
I’m not too familiar with k8s/hpa, so I’m not sure what the difference is
r
There are 2 parts. The query from prometheus that determines the current part of the formula and the target number is given in the hpa object (yaml config file for pod autoscaling)
a
seems like that would be the prometheus query then
r
Yes let me see if that works. PromQL is not my forte either
No dice 😞. Seems Prometheus doesn’t allow me to do that. Back to no values given for the metric
@able-gigabyte-21598: Seems I just can’t get the query to work with the regular prometheus-adaptor and altering values in the helm chart
I used keda having just discovered it and it takes the prometheus query directly
Then finally I had success.
Note that I still think it should be livekit_egress_inuse as I ended up doing a query like this to get things to scale as expected
(sum(kube_pod_labels{pod=~“^.*livekit-egress.*“}) - sum(livekit_egress_available))/sum(kube_pod_labels{pod=~“^.*livekit-egress.*“})
target value being 0.2
Sorry for all the noise seems my prometheus-adaptor foo is weak
Once again thanks for the help
g
Thanks for the updates! Sounds like it works with KEDA and not prometheus-adapter? And you’re now using a query to subtract available from total number of ingress pods.
r
Yes the value needs to be inverted to work how I would expect in k8s
g
Awesome. Makes sense! I wonder why Prometheus-adapter didn’t work for you
r
Not sure. I left an issue on the github hopefully I’ll get a response. But at least for now I have a solution
g
Maybe it doesn’t support full promql, but that’s just a guess
Glad you figured it out! We’ll have to update our docs
r
I think an example of the yaml or helm values used would help alot. The example gives the logic you would need but not how to implement in k8s objects and PromQL
👍 1
🙏 1
Hey @great-umbrella-58130 @able-gigabyte-21598: Another update I found out the correct syntax for the prometheus-adaptor should look something like this - seriesQuery: ‘livekit_egress_available{namespace!=“”,pod!=“”}’ resources: overrides: namespace: {resource: “namespace”} pod: {resource: “pod”} service: {resource: “service”} name: matches: “^(.*)_egress_available” as: “${1}_recorders_inuse” metricsQuery: ‘(sum(kube_pod_labels{pod=~“^.*livekit-egress.*“}) by (<<.GroupBy>>) - sum(livekit_egress_available) by (<<.GroupBy>>))/sum(kube_pod_labels{pod=~“^.*livekit-egress.*“}) by (<<.GroupBy>>)’
🙏 1
3 Views
Open in Slack
PreviousNext
Powered by