https://livekit.io logo
Join Slack
Powered by
How to configure hpa setting for livekit on basis ...
# ask-ai
w
How to configure hpa setting for livekit on basis of both memory and cpu threshhold?
t
Thanks for reaching out, Test. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
To configure Horizontal Pod Autoscaling (HPA) for LiveKit based on both memory and CPU thresholds, follow these steps: 1. Enable Autoscaling: Set 
autoscaling.enabled
to 
true
in your 
values.yaml
file. 2. Set CPU and Memory Thresholds: Configure 
targetCPUUtilizationPercentage
and 
targetMemoryUtilizationPercentage
to your desired thresholds. 3. Define Resource Requests and Limits: Ensure resource requests and limits are set appropriately for your pods. For example:
Copy code
resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    cpu: 1000m
    memory: 1Gi
4. Optional - Use Custom Metrics: For more precise scaling, you can use the 
livekit_ingress_available
Prometheus metric. This requires installing the Prometheus adapter and configuring a custom metric. Here is an example configuration:
Copy code
autoscaling:
  enabled: true
  minReplicas: 1
  maxReplicas: 5
  targetCPUUtilizationPercentage: 60
  targetMemoryUtilizationPercentage: 60
Best Practices: • Start with conservative settings (e.g., 60%) and adjust based on workload requirements. • Monitor performance using Prometheus and Grafana to fine-tune your scaling strategy. • Set appropriate resource limits to avoid overcommitment and maintain pod stability. For more details, refer to the LiveKit documentation on autoscaling and custom metrics. Sources: Self-hosting the Ingress Service | LiveKit Docs | Kubernetes & Helm Charts | livekit/livekit_composite | DeepWiki
w
load threshold in WorkerOptions?
t
Hey there! Just wanted to explain a bit about the 
loadThreshold
in 
WorkerOptions
for LiveKit. It's a number between 0 and 1 that helps decide when a worker should stop taking on new jobs. By default, in production, this is set to 
0.75
. So, when a worker’s load goes over this threshold, it’ll be marked as unavailable and won’t accept any more jobs for the time being. Usually, this load is measured by looking at the overall CPU usage. Hope that clears things up! Sources: Deployment and scaling | LiveKit Docs | Deploying to production | LiveKit Docs
w
can we mark unavailable on the basis of memory consumption as well?
t
Hey there! Just a quick heads-up: LiveKit workers usually decide if they’re available based on CPU usage. But if you want, you can switch things up to mark workers as unavailable when their memory usage gets too high. You’d do this by adding a custom 
loadFunc
in 
WorkerOptions
. Here’s how you can set that up: 1. First, create a custom load function that checks memory usage and returns a value between 0 and 1. For example:
Copy code
const customMemoryLoadFunc = async () => {
  const memoryUsage = process.memoryUsage();
  return memoryUsage.heapUsed / memoryUsage.heapTotal;
};
1. Next, update your 
WorkerOptions
to use this function and pick a 
loadThreshold
that fits your memory limits. Like this:
Copy code
const options = {
  loadFunc: customMemoryLoadFunc,
  loadThreshold: 0.8 // Feel free to tweak this based on your needs
};
And that’s it! Now your workers will automatically become unavailable if their memory usage goes over your chosen threshold. Hope this helps!
w
how to create a loadThreshhold function with combination of both cpu and memory?
t
Hey there! If you want to create a 
loadThreshold
function in LiveKit that takes into account both CPU and memory usage, you can easily do this by implementing a custom 
load_fnc
in 
WorkerOptions
. Here’s a simple example to get you started:
Copy code
import psutil

async def combined_load_func():
    # Grab CPU usage
    cpu_usage = psutil.cpu_percent(interval=1)
    # Grab memory usage
    mem_usage = psutil.virtual_memory().percent
    # Combine both, giving CPU a bit more weight than memory
    load = (cpu_usage * 0.7) + (mem_usage * 0.3)
    # Make sure load doesn’t go over 1.0
    return min(load, 1.0)

# Now, set them up in WorkerOptions
opts = WorkerOptions(
    entrypoint_fnc=your_entrypoint,
    load_fnc=combined_load_func,
    load_threshold=0.8,
)
Here are some key things to keep in mind: • Combining Metrics: This example mixes CPU and memory usage, with CPU weighted at 70% and memory at 30%. Feel free to tweak these numbers based on what fits your workload best. • Capping Load: Using 
min
makes sure the load value never exceeds 1.0, keeping things safe. • Setting Threshold: The 
load_threshold
is set at 0.8, so your worker will pause taking new jobs once the combined load goes over 80%. With this approach, your worker can smartly balance both CPU and memory usage, helping things run smoother overall. If you have any questions or want to chat more about it, just let me know!
w
if load_threshhold reaches can we trigger a new worker so that it can accept the new calls?
t
When a worker hits its 
loadThreshold
, it stops taking on new jobs and is marked as unavailable. But just so you know, LiveKit doesn’t automatically spin up new workers in this situation. To keep things running smoothly, you might want to set up an external auto-scaling solution—like Kubernetes Horizontal Pod Autoscaler (HPA) or your cloud provider’s auto-scaling groups. These tools can watch your worker loads and launch new instances when needed, making sure new calls get routed to the right place. If you'd like, I’m here to help you get started with auto-scaling in your setup—just let me know! Sources: Worker options | LiveKit Docs | Deploying to production | LiveKit Docs | WorkerOptions | LiveKit Agents | Deployment and scaling | LiveKit Docs
w
import psutil async def combined_load_func(): # Grab CPU usage cpu_usage = psutil.cpu_percent(interval=1) # Grab memory usage mem_usage = psutil.virtual_memory().percent # Combine both, giving CPU a bit more weight than memory load = (cpu_usage * 0.7) + (mem_usage * 0.3) # Make sure load doesn’t go over 1.0 return min(load, 1.0) # Now, set them up in WorkerOptions opts = WorkerOptions( entrypoint_fnc=your_entrypoint, load_fnc=combined_load_func, load_threshold=0.8, ) hpa settings in k8s for above configurations so that when worker in marked unavailble k8s also spwas a new pod
t
Hey there! To get HPA up and running in Kubernetes for your LiveKit workers—so new pods spin up whenever a worker goes offline—here’s a simple step-by-step guide: 1. Set Up HPA Using CPU Utilization: You can start with Kubernetes’ built-in CPU metrics. For instance, in your 
values.yaml
file, try this: `autoscaling: enabled: true minReplicas: 1 maxReplicas: 5 targetCPUUtilizationPercentage: 50`This means pods will scale when CPU hits 50%. 2. Use Custom Metrics (Optional): If you want to get a bit more precise, you can use the 
livekit_ingress_available
metric from Prometheus. Here’s how that might look:
Copy code
autoscaling:
  enabled: true
  minReplicas: 1
  maxReplicas: 5
  custom:
    metricName: livekit_ingress_available
    targetAverageValue: 50
3. Adjust Scaling Parameters: Make sure to tweak 
minReplicas
and 
maxReplicas
to fit your workload. Also, consider adjusting cooldown periods to better handle steady load increases or slow task draining. If you want a hand setting up Prometheus or need help fine-tuning anything, just give me a shout—I’m here to help! Sources: Deploying to production | LiveKit Docs | gclb-sample.yaml | Deployment and scaling | LiveKit Docs | Self-hosting the Ingress Service | LiveKit Docs
w
at what time intervals the load_fnc is called?
t
Hey there! Just a quick heads-up: the 
load_fnc
function in LiveKit runs every 0.5 seconds, thanks to the 
UPDATE_LOAD_INTERVAL
constant. This way, it keeps things updated regularly to help manage job distribution smoothly. Sources: livekit-agents/livekit/agents/voice/generation.py
w
def combined_load_func(): # Grab CPU usage (non-blocking, uses previous interval measurement) cpu_usage = psutil.cpu_percent() # Grab memory usage mem_usage = psutil.virtual_memory().percent # Give equal weight to both CPU and memory # Return the higher of the two usage percentages as a decimal (0.0 to 1.0) max_usage = max(cpu_usage, mem_usage) load = max_usage / 100.0 # Convert percentage to decimal # Make sure load doesn’t go over 1.0 return min(load, 1.0) i have used this and added a log in this function but i am not getting that log
t
Hey there! If the log inside your custom 
load_fnc
function isn’t showing up, no worries—here are a few things that might be causing it and how you can fix them: 1. Logger Usage: Double-check that you’re using the right logger. LiveKit has a logger you can import from 
livekit.agents.log
. Using this logger helps make sure your logs get captured properly. 2. Log Level Configuration: Take a peek at your logging level settings. If it’s set to 
WARNING
or higher, then 
INFO
or 
DEBUG
messages won’t show up. You can tweak this by using 
logging.basicConfig()
or adjusting the logger’s level directly. 3. Custom 
load_fnc
Not Being Called: Make sure your 
load_fnc
is correctly assigned in the 
WorkerOptions
and that your worker is actually running as expected. Here’s a quick example of a 
load_fnc
with logging set up just right:
Copy code
import logging
from livekit.agents import WorkerOptions

# Get the default logger
logger = logging.getLogger("livekit")

def custom_load_fnc(worker):
    load = len(worker.active_jobs) / 10  # Example load calculation
    <http://logger.info|logger.info>(f"Custom load function reporting load: {load}")
    return min(load, 1.0)

# Configure the logger if needed
logging.basicConfig(
    level=<http://logging.INFO|logging.INFO>,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)

opts = WorkerOptions(
    load_fnc=custom_load_fnc,
    load_threshold=0.9,
)
If you want, feel free to share more about your logging setup or environment—I’m here to help! Sources: Worker options | LiveKit Docs | livekit-agents/livekit/agents/voice/generation.py
Open in Slack
PreviousNext
Powered by