hi

Thanks for reaching out, Aaron. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

logger.error(f"Error during shutdown cleanup: {e}") ctx.shutdown() asyncio.create_task(shutdown_cleanup()) await session.start( agent=agent, room=ctx.room, ) if name == "__main__": cli.run_app( WorkerOptions( entrypoint_fnc=entrypoint, prewarm_fnc=prewarm, agent_name="ml_agent", ) ) tell how to increase the cpu usage here and where so that workers are avialable always

Thanks — you want to increase CPU usage so workers stay available. I’ll respond to the snippet you shared and outline the places to change configuration and runtime behavior. What part of the flow do you want to affect most — (A) make each agent process use more CPU (so it can handle heavier work), or (B) keep more worker processes available (so capacity is higher)? Pick A or B and I’ll give exact code/config changes. Quick guidance while you decide: • To make a single agent use more CPU: run the agent on a machine with more vCPUs (recommended 4 cores / 8GB baseline) and avoid burstable instance types (use m5/c5/c6i on AWS). Prewarm models in your

prewarm

function so sessions start quickly. • To increase total capacity (more always-available workers): run multiple agent processes (horizontal scaling) and autoscale based on a load function. Implement a

load_fnc

that returns current load (0–1) from

len(agent_server.active_jobs) / N

and set

load_threshold

lower than your autoscale trigger. • Also ensure graceful shutdown/drain settings (increase

drain_timeout

) so workers don’t exit while sessions are active. Tell me A or B (or both), and which deployment you use (local/docker, Kubernetes, or cloud provider). I’ll give the exact code/config snippets next. Sources: Server options | LiveKit Documentation | Self-hosted deployments | LiveKit Documentation | Troubleshooting Latency and Timeout Errors with Turn Detection on AWS | Voice Agent Architecture: STT, LLM, and TTS Pipelines Explained | LiveKit | Build a multilingual voice agent that automatically switches languages | Li..., with interactive elements

i want to make the capacity higher

Thanks — since you want higher capacity (more workers always available), here’s a concise plan and actionable steps you can apply whether you run locally, in Docker, or on Kubernetes/cloud. 1) Strategy overview • Scale horizontally: run more agent processes/workers (multiple OS processes or pods) so more sessions can be served concurrently. • Use a

load_fnc

to measure per-agent load and stop assigning new sessions when an agent is busy. • Autoscale (cloud/K8s): scale the number of worker instances based on CPU or custom load metrics so capacity increases automatically before saturation. • Prewarm models and increase drain timeout so new jobs start fast and workers don’t exit mid-session. 2) Immediate code/config changes • Implement a load function that returns 0–1 based on active jobs. Example pattern:

def compute_load(agent_server) -> float:
    # allow N concurrent jobs per worker; adjust N to tune capacity per worker
    N = 10
    return min(len(agent_server.active_jobs) / N, 1.0)

Pass it to your AgentServer / worker startup and set a

load_threshold

(e.g., 0.7). • Prewarm resources in your

prewarm

function so workers are ready instantly:

async def prewarm(proc):
    proc.userdata["models"] = load_models()  # load LLM/ASR/VAD here

• Increase

drain_timeout

so workers finish sessions instead of shutting immediately:

AgentServer(..., drain_timeout=2400)  # e.g., 40 minutes

3) Deployment / scaling recommendations • Run multiple worker processes: ◦ Local/Docker: run several container instances of the agent. ◦ Kubernetes: run a Deployment with multiple replicas. • Autoscaling triggers: ◦ Use CPU utilization (target e.g., 50%) or a custom metric derived from

len(active_jobs)/N

. ◦ Set autoscale to add capacity before

load_threshold

is reached (e.g., scale when load ~0.5 if threshold = 0.7). • Avoid burstable instances (t3/t4). Use dedicated instance families (m5/c5/c6i) and size with ~4 vCPUs + 8GB baseline per agent as a starting point. 4) How to size and tune • Start with N (concurrent jobs per worker) conservative (5–10) depending on model CPU usage. • Monitor metrics: active sessions per worker, CPU, memory, latency. Increase replicas or N as you observe headroom. • Consider mixing instance sizes or dedicated worker pools for heavy vs light jobs. 5) Health checks & graceful rollout • Enable the built-in health endpoint and ensure your orchestrator checks it. • During deployments, drain old pods (respect drain_timeout) so active sessions aren’t dropped. If you tell me: • where you deploy (Docker, Kubernetes, or cloud provider) and • a target concurrency (how many concurrent sessions you want total), I’ll provide exact example manifests / CLI commands and a recommended N and autoscaling policy. , with interactive elements

no i want to increase the load threshold, so give met he code to increase it the workeroptions code

You can set

load_threshold

on

WorkerOptions

when starting your worker. Example — adjust the value (0.0–1.0) to your desired threshold:

from livekit import WorkerOptions

def entrypoint(...):
    ...

def prewarm(...):
    ...

if __name__ == "__main__":
    cli.run_app(
        WorkerOptions(
            entrypoint_fnc=entrypoint,
            prewarm_fnc=prewarm,
            agent_name="ml_agent",
            load_threshold=0.9,  # increase this to accept more load (0.0-1.0)
        )
    )

If you also use a custom

load_fnc

, ensure its returned value is scaled to 0–1 so

load_threshold

compares correctly. Want an example

load_fnc

too? , with interactive elements