local deployment, we use oke, and many pods, what should be the pod size and how many agent the pod can get. and what should be the

load_threshold

and

load_fnc

Thanks for reaching out, Mostafa. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

i want to set the max agents per pod to 7, how can i do that

Thanks — I’m responding to your request about limiting each pod to a maximum of 7 agents. Below are a clear explanation, an example you can drop into your AgentServer, and a few practical tips for Kubernetes/OKE. How it works (brief) • LiveKit decides whether an AgentServer will accept a new job by calling

load_fnc

which must return a value in [0,1]. If that value is >=

load_threshold

, the server will stop accepting new jobs. • To limit to 7 agents, make

load_fnc

return active_jobs / 7 (clamped to 1.0) and set

load_threshold

slightly below 1.0 so the server stops accepting new jobs when it reaches 7. Example (Python) 1. Create the server with a threshold of 0.9. 2. Use a

compute_load

function that returns min(active_jobs / 7, 1.0).

from livekit.agents import AgentServer

# set threshold slightly below 1.0 so 7 jobs cause the server to stop accepting more
server = AgentServer(load_threshold=0.9)

def compute_load(agent_server: AgentServer) -> float:
    # agent_server.active_jobs is the list of currently running jobs
    return min(len(agent_server.active_jobs) / 7.0, 1.0)

server.load_fnc = compute_load

Practical tips for OKE/Kubernetes • One AgentServer per pod: run a single AgentServer instance in each pod and use horizontal scaling (multiple pods) rather than packing multiple AgentServers in one pod. • Resource requests/limits: match pod CPU/RAM to expected load (a good starting baseline is 4 vCPU / 8Gi per pod for typical voice AI workloads). Adjust up if your jobs use heavier models or additional processing. • Autoscaling: configure your HPA to scale out before pods hit the 7-job limit. For example, scale-up when average jobs per pod reaches 5–6, or when CPU utilization is ~50%. • Observability: monitor active job counts, CPU, and memory. Tune the divisor (7) and

load_threshold

as you observe real traffic. • Graceful termination: set

terminationGracePeriodSeconds

long enough (e.g., 10+ minutes) so in-flight jobs aren’t killed during rollouts. If you want, I can: • Produce a Kubernetes pod manifest example with resource requests/limits and an HPA configured to scale based on a custom metric (active jobs) or CPU. • Recommend precise HPA thresholds if you share expected job duration, typical CPU usage per job, and whether you’re using GPU nodes. Sources: Server options | LiveKit Documentation | Self-hosted deployments | LiveKit Documentation

what do you mean by that, One AgentServer per pod: run a single AgentServer instance in each pod and use horizontal scaling (multiple pods) rather than packing multiple AgentServers in one pod. we have agent with name x deployed local and it contains 24 pods

Thanks — I’m responding to your question about what “one AgentServer per pod” means and how that applies to your agent named "x" with 24 pods. What “one AgentServer per pod” means • A pod is Kubernetes’ unit of deployment. It can run one or more containers/processes. • “One AgentServer per pod” means each pod runs a single instance of the LiveKit

AgentServer

process. That AgentServer manages its own jobs and reporting of load. • The alternative (not recommended) is to run multiple AgentServer processes inside a single pod. That groups multiple independent servers into one Kubernetes scheduling/termination unit. Why we recommend one AgentServer per pod • Predictable scaling: Kubernetes scales pods. If each pod is exactly one AgentServer, adding pods increases capacity in a 1:1 way. • Resource isolation: Resource requests/limits, pod scheduling, and node placement apply to the whole pod. When one AgentServer = one pod, you know how much CPU/RAM each server will get. • Simpler autoscaling and metrics: HPA or custom metrics can measure per-pod CPU or active-job count and scale pods accordingly. If a pod contains multiple AgentServers, mapping metrics to capacity becomes harder. • Easier lifecycle management: Pod termination, rolling updates, and graceful draining are simpler when there’s a single server to drain per pod. This reduces risk of killing multiple servers at once. • Troubleshooting: Logs, crashes, resource spikes, and restarts are easier to correlate to a single AgentServer instance. How this applies to your setup (agent "x" with 24 pods) • If each of those 24 pods runs one AgentServer instance, that matches the recommended pattern: you effectively have 24 AgentServers and scale capacity by adding/removing pods. • If those 24 pods each run multiple AgentServer processes, you should consider moving to one AgentServer per pod because: ◦ You’ll get clearer capacity planning (e.g., “24 pods × max 7 agents per pod = 168 concurrent agents”). ◦ Autoscaling can be based on per-pod metrics rather than complex intra-pod counts. Quick checks you can run 1. Inspect the pod spec for agent "x": ◦ Look at the container command or entrypoint. Does it start one AgentServer or multiple processes? ◦ Count processes in a running pod (e.g.,

kubectl exec -it <pod> -- ps aux

). 1. Confirm resource requests/limits: ◦ If a pod runs multiple AgentServers but requests resources sized for a single server, you may be overloading the pod. 1. Check your metrics: ◦ Are you scaling HPA based on pod CPU or a custom metric? Does that metric reflect the number of AgentServers inside the pod? If your current 24 pods already each run one AgentServer, you’re aligned with the recommendation. If not, tell me: • Whether each pod currently runs one process or multiple processes, • What your desired max agents per pod is (you previously said 7), • If you want an example pod manifest or an HPA example to scale to the target capacity. I can then give a concrete plan: • how many pods you need for a given target concurrency, • a sample Kubernetes manifest that runs one AgentServer per pod with proper requests/limits, • an HPA recipe (CPU-based or custom metric based on active job count) to scale smoothly.

is this code ok ? MAX_CONCURRENT_AGENTS = 7 def *compute_load*(options: WorkerOptions) -> float: return *min*(*len*(options.active_jobs) / MAX_CONCURRENT_AGENTS, 1.0) cli.run_app( WorkerOptions( entrypoint_fnc=partial(entrypoint, avatar_dispatcher_url=args.avatar_url), prewarm_fnc=prewarm, worker_type=WorkerType.ROOM, agent_name=AGENT_NAME, load_fnc=compute_load, load_threshold=0.9, ) )

Thanks — I’m responding to your ask to confirm whether that code correctly limits each pod to 7 concurrent agents. Short answer: yes, your approach is correct, and the snippet looks right with one small note about naming/typing consistency. Below I confirm what your code does, point out the minor improvements, and give a ready-to-run example plus a couple of operational tips. What your code does •

compute_load

returns a normalized load in [0,1] by dividing current active jobs by

MAX_CONCURRENT_AGENTS

(7). • With

load_threshold=0.9

, once a pod reaches 7 active jobs the returned load will be 1.0, which is above 0.9, so the server will stop accepting new jobs. This enforces the 7-agents-per-pod limit as intended. Minor improvements and a corrected example • Use consistent names and types used by your runtime (you used

WorkerOptions

and

options.active_jobs

— that matches your environment). Ensure the function signature matches the expected callback type (some SDKs expect the server instance or options type; your snippet looks consistent with

options: WorkerOptions

). • Add simple guards to avoid division by zero and keep the code clear. Example (cleaned and ready to drop in):

MAX_CONCURRENT_AGENTS = 7

def compute_load(options: WorkerOptions) -> float:
    # Ensure MAX_CONCURRENT_AGENTS is positive
    if MAX_CONCURRENT_AGENTS <= 0:
        return 1.0
    return min(len(options.active_jobs) / float(MAX_CONCURRENT_AGENTS), 1.0)

cli.run_app(
    WorkerOptions(
        entrypoint_fnc=partial(entrypoint, avatar_dispatcher_url=args.avatar_url),
        prewarm_fnc=prewarm,
        worker_type=WorkerType.ROOM,
        agent_name=AGENT_NAME,
        load_fnc=compute_load,
        load_threshold=0.9,
    )
)

Operational tips • Test under load: run a load test to confirm that 7 active jobs produces the expected CPU/memory profile and that pods stop accepting new jobs once full. • Autoscaling: configure your HPA to scale pods before they reach the 7-job limit (e.g., scale up when average jobs per pod is 5–6 or when CPU ~50%) so you don’t run out of capacity. • Observability: emit/collect the active job count as a metric so you can create alerts/ dashboards and tune MAX_CONCURRENT_AGENTS if needed. • Graceful shutdown: keep a long enough

terminationGracePeriodSeconds

so pods can finish in-flight jobs when being drained. If you want, I can: • Provide an example Kubernetes Deployment manifest that runs one AgentServer per pod with resource requests/limits, • Show a HorizontalPodAutoscaler example using a custom metric (active jobs) or CPU, • Help you pick HPA thresholds if you share typical job duration and per-job CPU usage. Would you like the manifest + HPA examples? If so, tell me whether you prefer CPU-based or active-job-count autoscaling.

how can make a new deployment without affecting the current users?

I’m responding to your question about deploying a new version without affecting current users. Below are practical, proven ways to do zero‑downtime deployments on OKE (Kubernetes) for LiveKit AgentServer pods, plus concrete settings and an example flow you can apply immediately. High-level strategies (pick one based on risk tolerance) • Rolling update (recommended default): Gradually replace pods with new version one (or a few) at a time so most pods remain serving traffic. • Canary release: Deploy a small percentage of pods with the new version, observe behavior, then increase rollout if healthy. • Blue/Green deployment: Deploy a full parallel environment (green), switch traffic to it when ready, then decommission the old (blue). Best for large changes but costlier. Key Kubernetes features and settings to enable safe rollouts 1. Readiness probe ◦ Make sure your pod has a readinessProbe that only becomes true when the AgentServer is fully ready to accept new jobs (after warmup, model loading, etc.). Kubernetes only sends traffic to pods marked ready. 1. Liveness probe ◦ Use a livenessProbe to detect and restart unhealthy processes quickly. 1. terminationGracePeriodSeconds ◦ Set this high enough to allow in‑flight jobs to finish gracefully (10+ minutes for long voice/AI sessions is a common starting point). 1. PreStop hook (optional) ◦ Use a PreStop hook to tell the agent to stop accepting new work and finish in‑flight jobs before SIGTERM is sent. 1. Pod Disruption/Drain handling ◦ Ensure your AgentServer respects the SIGTERM lifecycle (stop new jobs, wait for existing jobs to complete or persist state). 1. Horizontal Pod Autoscaler (HPA) ◦ Ensure HPA scaling acts smoothly during rollouts (avoid simultaneous scale-down that removes capacity needed during upgrade). 1. Load/Traffic split (for canary/blue-green) ◦ Use service selectors, labels, or an ingress/controller that supports traffic splitting to route a percentage of traffic to canary pods. Concrete rolling update recipe (safe, cost-efficient) 1. Add readiness & liveness probes to your Deployment: ◦ readinessProbe: HTTP or TCP endpoint that returns success only when agent is warmed and ready. ◦ livenessProbe: short interval check to restart stuck processes. 1. Ensure graceful shutdown: ◦ Set

terminationGracePeriodSeconds: 600

(10 minutes) or higher if jobs are longer. ◦ Implement a PreStop hook or SIGTERM handler in your app that: ▪︎ stops accepting new jobs ▪︎ waits for in-flight jobs to finish (or checkpoint) ▪︎ then exits 1. Configure Deployment rollout behavior: ◦ set strategy: ▪︎ type: RollingUpdate ▪︎ rollingUpdate: ▪︎ maxUnavailable: 10% (or 1) ▪︎ maxSurge: 1 (or 25%) ◦ This ensures Kubernetes only replaces a small number of pods at a time. 1. Trigger the rollout: ◦ kubectl apply -f deployment.yaml ◦ Monitor: kubectl rollout status deployment/<name> ◦ Watch metrics (active job counts, CPU, error rates) while pods update. 1. If you see problems, rollback: ◦ kubectl rollout undo deployment/<name> Canary flow (more cautious) 1. Create a canary Deployment with a small replica count (e.g., 1–2 pods) or label (version: v2). 2. Route a small portion of traffic to canary (service selector or ingress split). 3. Monitor for errors, latency, resource usage for a period. 4. Gradually increase canary pods or traffic percentage until full rollout. Blue/Green flow (full isolation) 1. Stand up green environment (Deployment + Service) using new images/config. 2. Warm pods until readiness is true and metrics are normal. 3. Switch a Service to point to green (update selector or update ingress). 4. Keep blue running for a rollback plan, then decommission. Recommended default config values to start with • readinessProbe: HTTP GET /healthz or a custom endpoint; initialDelaySeconds: 10–30; periodSeconds: 5 • livenessProbe: HTTP GET /healthz; failureThreshold: 3; periodSeconds: 15 • terminationGracePeriodSeconds: 600 (10 minutes) • Deployment strategy: ◦ maxUnavailable: 10% (or 1) ◦ maxSurge: 25% (or 1) • HPA: scale up when CPU ~50% or when average active jobs per pod >= 5 (if you expose active jobs as a metric) Example rollout commands to monitor • Start rollout: kubectl apply -f deployment.yaml • Watch rollout: kubectl rollout status deployment/<agent-deployment> • Inspect pods: kubectl get pods -l app=<agent> -o wide • See pod termination events: kubectl describe pod <pod-name> • Rollback if needed: kubectl rollout undo deployment/<agent-deployment> Operational checklist before you deploy • Confirm readiness probe only becomes true when the agent is ready (warm models loaded). • Confirm SIGTERM handling lets existing jobs finish or checkpoint within terminationGracePeriodSeconds. • Ensure metrics/alerts are in place (active jobs, CPU, error rates). • If using HPA, temporarily raise min replicas during rollout if you expect transient increased load. • Perform a small canary before full rollout for high-risk changes. If you want, I can: • Provide a sample Kubernetes Deployment manifest with readiness/liveness probes, PreStop hook, terminationGracePeriodSeconds set, and RollingUpdate strategy. • Provide an example HPA manifest (CPU-based or custom active-job metric) and how to expose active job count using a simple metrics exporter. • Walk through an example canary manifest and service-splitting for traffic percentage. Which would you like next — a Deployment + HPA manifest, or a canary example? Also tell me whether your AgentServer exposes a health endpoint currently and how long typical jobs run so I can tune terminationGracePeriodSeconds.