we are deploying with cloud scale plan trying to get 250 outbound callers concurrently from 8am-8pm however we run into alot of critical errors, job continuously times out and we are only at 3 idle processes per worker, in our toml file we start at 10 workers scale to 15, how do we optimize our toml and idle processes to prevent these unresponsive job errors

Thanks for reaching out, Bobvi. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

1. Run a staged load test: ◦ Ramp to 100 → 200 → 250 concurrent callers while watching idle process utilization, queue depth, and job latency. Increase idle processes or max workers if timeouts reappear. how do i run. a stage loaded tests, i dont have any auto scaling thing ssetup, our sip and callers are in the same spto

Acknowledged — you said you need to run a staged load test (100 → 200 → 250 concurrent callers) but don’t have autoscaling and want concrete steps on how to run the test and monitor results. Below is a concise, actionable plan you can run now. 1) Prep • Pick a test runner machine (or 2) separate from your agents so clients don’t consume agent CPU. • Install LiveKit CLI:

npm install -g livekit-cli

• Gather connection info: LiveKit URL, API key/secret, and a test room name. 2) Configure a realistic caller simulation • Each virtual caller should: ◦ Join a room. ◦ Publish an audio track (looped or prerecorded). ◦ Subscribe to agent audio. • If you need voice scripts or VAD behavior, use short audio files and spacing to mimic real calls. 3) Staged test commands (run sequentially) • Stage 1 — 100 callers for 10 minutes:

lk load-test --url <wss://YOUR_DOMAIN> --api-key KEY --api-secret SECRET \
    --room loadtest --audio-publishers 100 --subscribers 100 --participant-count 100 --duration 10m

• Stage 2 — 200 callers for 10 minutes (wait 5 min after stage 1 stabilizes):

... --audio-publishers 200 --subscribers 200 --participant-count 200 --duration 10m

• Stage 3 — 250 callers for 30 minutes (wait after stage 2 stabilizes):

... --audio-publishers 250 --subscribers 250 --participant-count 250 --duration 30m

4) What to monitor during each stage • Idle process utilization (how many warm processes remain). • Job queue depth and job start latency. • Agent CPU, memory, and thread utilization. • Error logs (timeouts, “session did not start” messages). • Network metrics and packet loss (since callers/SIP are in same region). 5) If you see timeouts/failures • First, increase

num_idle_processes

(try 5 → 10 → 15) and re-run the stage that failed. • If idle processes still deplete quickly, increase worker capacity or use more powerful machines (16+ cores recommended). • Rerun the failed stage after each change to confirm improvement. 6) No-autoscale considerations • Because you have no autoscaling, provision headroom before peak (higher min workers and more idle processes). • Use the staged test to find the minimal

num_idle_processes

and worker count that sustain 250 callers without timeouts. 7) Quick diagnostics I can help with • Share your current

livekit.toml

(worker min/max) and your agent config snippet showing

num_idle_processes

. • Tell me whether test traffic is bursty (many starts at once) or steady. Would you like me to propose exact toml and idle-process values to paste once you share your current config? Sources: Benchmarking | LiveKit docs | Agent server options | LiveKit docs | Troubleshooting Latency and Timeout Errors with Turn Detection on AWS

how do i setup autoscaling for livekit cloud [project] subdomain = "buckeye-dashboard-p02a9yob" [agent] id = "CA_ubuZhpVxLJvL" in the toml

You're asking how to set up autoscaling for your LiveKit Cloud project/agent (the toml snippet you showed). Autoscaling on LiveKit Cloud is automatic — you do not add autoscaling fields to

livekit.toml

. Your current file is already correct:

[project]
subdomain = "buckeye-dashboard-p02a9yob"

[agent]
id = "CA_ubuZhpVxLJvL"

What to do next • Deploy the agent:

lk agent deploy

— LiveKit Cloud will scale instances up/down based on demand. • Monitor scaling and health in the Cloud dashboard: https://cloud.livekit.io → Projects → Agents. • Tail agent logs to investigate session failures:

lk agent logs --id CA_ubuZhpVxLJvL

. If you see insufficient scaling or timeouts, tell me: • Recent error messages from logs (copy/paste a snippet), and • Your current agent

num_idle_processes

and worker capacity (if you customized them). I’ll recommend config values or next debugging steps based on those details. Sources: Deploying to production | LiveKit Docs | Deploy and scale agents on LiveKit Cloud | Deploy and scale agents on LiveKit Cloud | Agent deployment CLI reference | LiveKit docs | Deploying to custom environments | LiveKit docs

how much idle processes should we have per worker to flood over

For handling heavy bursts (like 250 concurrent outbound callers) start with 5–10 idle processes per worker, then tune upward if needed. Why: • Each idle process lets a worker accept a job immediately without cold-start delay. With only 3 idle processes you saw backpressure and timeouts. • 5–10 gives reasonable warm capacity while keeping memory/CPU reasonable; if jobs are light you can start at 5, if jobs preload heavy models or do STT/TTS use 10+. How to tune quickly 1. Set

num_idle_processes

= 5 and run a staged load (100 → 200 → 250). 2. If idle processes hit 0 quickly or you see timeouts, raise to 10 and retest. 3. If still failing, increase worker headroom (more workers or larger machines) rather than only increasing idle processes. Monitoring to guide changes • Idle process count over time (are they exhausted?) • Job queue depth and job start latency • CPU/memory per agent process • Error/timeouts rate If you share your current

num_idle_processes

, worker count, and a short log snippet showing timeouts, I’ll suggest exact values to try next.

can we add definitive scaling polcies to our toml ? and give me the exact commands to setup a load test for our cloud setup

Short answers first: • You cannot add definitive autoscaling policies to

livekit.toml

for LiveKit Cloud — autoscaling is automatic and managed by LiveKit Cloud. • Below are exact commands and a step-by-step staged load-test plan you can run against your Cloud project. Autoscaling note • LiveKit Cloud handles scaling;

livekit.toml

only needs your project subdomain and agent id:

[project]
subdomain = "buckeye-dashboard-p02a9yob"

[agent]
id = "CA_ubuZhpVxLJvL"

Staged load test — exact commands and plan Prerequisites: • Install LiveKit CLI:

npm install -g livekit-cli

• Have these values ready: ◦ LiveKit WSS URL (e.g. wss://cloud.livekit.io or your custom domain) ◦ API key and API secret with permission to create test participants ◦ A test room name (e.g. loadtest-room) Run sequential stages (wait a few minutes between stages to let metrics stabilize): 1) Stage 1 — 100 callers, 10 minutes

lk load-test \
  --url <wss://YOUR_LIVEKIT_DOMAIN> \
  --api-key YOUR_API_KEY \
  --api-secret YOUR_API_SECRET \
  --room loadtest-room \
  --audio-publishers 100 \
  --subscribers 100 \
  --participant-count 100 \
  --duration 10m

2) Stage 2 — 200 callers, 10 minutes

lk load-test \
  --url <wss://YOUR_LIVEKIT_DOMAIN> \
  --api-key YOUR_API_KEY \
  --api-secret YOUR_API_SECRET \
  --room loadtest-room \
  --audio-publishers 200 \
  --subscribers 200 \
  --participant-count 200 \
  --duration 10m

3) Stage 3 — 250 callers, 30 minutes

lk load-test \
  --url <wss://YOUR_LIVEKIT_DOMAIN> \
  --api-key YOUR_API_KEY \
  --api-secret YOUR_API_SECRET \
  --room loadtest-room \
  --audio-publishers 250 \
  --subscribers 250 \
  --participant-count 250 \
  --duration 30m

If your test clients need to send prerecorded audio (recommended for reproducible tests), create a folder of short WAV/OGG files and pass the appropriate flags or use a small client script that uses the LiveKit SDK to publish them. If

lk load-test

in your CLI version lacks an audio-file flag, use a simple Node/Python script to create participants and publish audio. What to monitor during the test • Idle process utilization (are

num_idle_processes

exhausted?) • Job queue depth and job start latency • Agent CPU and memory per process • Error logs for timeouts (copy any timeout log lines) • Network packet loss/latency (your callers/SIP are co-located which helps) Tuning guidance if you hit timeouts • Increase

num_idle_processes

to 5→10→15 and repeat the failing stage. • If idle processes still drop to zero quickly, increase worker capacity or use larger machines. • Re-run only the stage that failed after each change. If you want, paste: • Your current

livekit.toml

• Current

num_idle_processes

and worker settings (if set in agent code) • A short log snippet showing the timeout error I’ll give exact values to paste into your agent config and a short retest plan.

i cant do this ? # Production scaling [agent.scaling] min_instances = 10 max_instances = 20

You cannot add manual scaling policies (like

[agent.scaling] min_instances/max_instances

) to

livekit.toml

for LiveKit Cloud. LiveKit Cloud manages autoscaling automatically; the TOML only needs your project subdomain and agent id:

[project]
subdomain = "buckeye-dashboard-p02a9yob"

[agent]
id = "CA_ubuZhpVxLJvL"

What you can do now • Deploy your agent with

lk agent deploy

and Cloud will scale instances based on load. • Monitor scaling and health in the Cloud dashboard: https://cloud.livekit.io → Projects → Agents. • Tail logs to investigate failures:

lk agent logs --id CA_ubuZhpVxLJvL

. If you want explicit control over scaling you must self-host agents (e.g., Kubernetes) and implement autoscaling there; I can walk you through that setup and provide example Kubernetes HPA/YAML if you want. Would you like: • Help troubleshooting your current timeouts (paste logs and your

num_idle_processes

), or • A guide to self-hosted autoscaling with exact YAML examples?

so for cloud we need to do 5 idle processes per owrker ?

Yes — start with 5 idle processes per worker as a baseline for LiveKit Cloud. Why: each idle process accepts a job immediately without cold-start delay. With only 3 you saw backpressure; 5 gives better warm capacity while keeping CPU/memory reasonable. If jobs preload heavy models (STT/TTS) or you still see timeouts, increase to 10–15. How to set it (agent code example)

server = AgentServer(
  setup_fnc=prewarm,
  num_idle_processes=5
)

Tuning steps 1. Deploy with

num_idle_processes=5

. 2. Run a staged load test (100 → 200 → 250) and watch idle-process utilization, job queue depth, and timeouts. 3. If idle processes reach 0 quickly or timeouts occur, raise to 10 and retest; if still failing, increase worker capacity or use larger machines. If you share your current

num_idle_processes

, worker count, or a timeout log snippet, I’ll recommend exact values to try next.

give me a breakdown on how to do this 2) Configure a realistic caller simulation • Each virtual caller should: ◦ Join a room. ◦ Publish an audio track (looped or prerecorded). ◦ Subscribe to agent audio. • If you need voice scripts or VAD behavior, use short audio files and spacing to mimic real calls. 3) Staged test commands (run sequentially) • Stage 1 — 100 callers for 10 minutes:

lk load-test --url <wss://YOUR_DOMAIN> --api-key KEY --api-secret SECRET \
    --room loadtest --audio-publishers 100 --subscribers 100 --participant-count 100 --duration 10m

• Stage 2 — 200 callers for 10 minutes (wait 5 min after stage 1 stabilizes):

... --audio-publishers 200 --subscribers 200 --participant-count 200 --duration 10m

• Stage 3 — 250 callers for 30 minutes (wait after stage 2 stabilizes):

... --audio-publishers 250 --subscribers 250 --participant-count 250 --duration 30m

4) What to monitor during each stage • Idle process utilization (how many warm processes remain). • Job queue depth and job start latency. • Agent CPU, memory, and thread utilization. • Error logs (timeouts, “session did not start” messages). • Network metrics and packet loss (since callers/SIP are in same region) the virtual caller setup

Below is a concise, step-by-step breakdown to build realistic virtual callers for your staged load test, plus exact commands and a simple custom-bot option if you need prerecorded audio or VAD-like behavior. 1) Use the LiveKit CLI (quick setup) • Install:

npm install -g livekit-cli

• Required values: WSS URL (e.g., wss://YOUR_DOMAIN.livekit.cloud), API key, API secret, test room name. 2) Basic virtual caller behavior (what each caller must do) • Join the test room as a participant. • Publish an audio track (either synthetic/silent audio from the CLI or a prerecorded file). • Subscribe to agent audio (so agents’ replies are received). • Optionally toggle publishing on/off to mimic VAD (only speak in bursts). 3) Run the built-in load tester (sequential stages) • Stage 1 — 100 callers, 10m:

lk load-test \
    --url <wss://YOUR_DOMAIN.livekit.cloud> \
    --api-key YOUR_API_KEY \
    --api-secret YOUR_API_SECRET \
    --room loadtest \
    --audio-publishers 100 \
    --subscribers 100 \
    --participant-count 100 \
    --duration 10m

• Stage 2 — 200 callers, 10m (wait ~5m after Stage 1 stabilizes):

lk load-test ... --audio-publishers 200 --subscribers 200 --participant-count 200 --duration 10m

• Stage 3 — 250 callers, 30m (wait after Stage 2 stabilizes):

lk load-test ... --audio-publishers 250 --subscribers 250 --participant-count 250 --duration 30m

4) If you need actual speech (prerecorded audio) or VAD behavior Option A — Simple: use the CLI if it supports an audio file flag (some versions do). If not, use Option B. Option B — Lightweight custom bot (Node example outline) • Create N instances (or run multiple processes) that: a. Connect with the LiveKit SDK using the test API key/room token. b. Publish an audio track created from a small looped WAV file. c. Sleep/send silence for a configurable gap to simulate talk turns. d. Subscribe to agent tracks and optionally log when audio is received. • Run the script with different process counts to reach 100/200/250 participants. Minimal Node pseudocode steps: • Create a local PeerConnection using the LiveKit SDK. • Create a MediaStreamTrack from an audio file or generated frames. • Publish track, then periodically stop/start the track to simulate VAD. 5) Realistic traffic patterns to emulate • Only ~10–30% of callers actively speaking at any one time — others are silent subscribers. • Speech bursts: 2–10s talk, 2–8s silence. • Mix short calls (30–60s) and long calls (5–30m) to stress different resources. 6) What to monitor during each stage • Idle process utilization (are warm processes exhausted?) • Job queue depth and job start latency • Agent CPU, memory, thread counts • Error logs (timeouts, “session did not start”) • Network metrics and packet loss 7) Quick tuning loop • If idle processes hit 0 or timeouts appear: increase

num_idle_processes

(5→10→15), rerun failing stage. • If still failing: increase worker count or use larger instance types and rerun. 8) If you want, I can: • Provide a small Node script example to publish a prerecorded WAV and toggle silence (suitable to spawn many processes). • Suggest exact

num_idle_processes

and worker counts once you paste your current agent config and a timeout log snippet. Which would you like next — the Node script example or help tuning specific values from your config/logs? Sources: Benchmarking | LiveKit docs | Stream music over WebRTC using React and WebAudio | Agent server options | LiveKit docs | examples/voice_agents | livekit/livekit_composite | DeepWiki | Agents Framework | livekit/livekit_composite | DeepWiki

this is an example of the errors we get executionId: 'a2fc13a5-ca18-59d3-870d-a2ca5c7a82e7', errorType: 'unknown', errorMessage: 'Error connecting to LiveKit WebSocket', recoverable: undefined, fullError: '{\n "body": null,\n "retryable": true,\n "name": "APIConnectionError"\n}', timestamp: '2025-12-16T230029.927Z' } {"level":50,"time":1765926029928,"pid":183,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-5fcd6bc769-7dnwf","type":"stt_error","timestamp":1765926029927,"label":"inference.STT","error":{"body":null,"retryable":true,"name":"APIConnectionError"},"recoverable":false,"msg":"AgentSession is closing due to unrecoverable error"} [Process] Unhandled Promise rejection: { reason: APIConnectionError: failed to recognize speech after 4 attempts at SpeechStream.mainTask (file:///app/node_modules/.pnpm/@livekit+agents@1.0.25_@livekit+rtc-node@0.13.22_zod@3.25.76/node_modules/@livekit/agents/dist/stt/stt.js8219) at process.processTicksAndRejections (nodeinternal/process/task queues105:5) { body: null, retryable: false }, promise: Promise { <rejected> APIConnectionError: failed to recognize speech after 4 attempts at SpeechStream.mainTask (file:///app/node_modules/.pnpm/@livekit+agents@1.0.25_@livekit+rtc-node@0.13.22_zod@3.25.76/node_modules/@livekit/agents/dist/stt/stt.js8219) at process.processTicksAndRejections (nodeinternal/process/task queues105:5) { body: null, retryable: false }, [Symbol(async_id_symbol)]: 4904, [Symbol(trigger_async_id_symbol)]: 4878, [Symbol(kResourceStore)]: JobContext { shutdownCallbacks: [Array], _primaryAgentSession: [AgentSession], _sessionDirectory: '/tmp/livekit-agents/job-AJ_GCTSqjLYeiMd', connected: true, onParticipantConnected: [Function: bound onParticipantConnected] } } } {"level":40,"time":1765921006046,"pid":136,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-66fbc5cdb6-ghg22","msg":"job is unresponsive"} {"level":40,"time":1765921007791,"pid":136,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-66fbc5cdb6-ghg22","msg":"job is unresponsive"}

Thanks — I reviewed the error logs you pasted. Quick summary of what’s happening and exactly what to check/fix next. What the logs show • Agents fail to connect to LiveKit WebSocket:

APIConnectionError: Error connecting to LiveKit WebSocket

. • STT calls fail after retries:

failed to recognize speech after 4 attempts

(STT returns APIConnectionError). • After those failures, sessions close and jobs become unresponsive:

AgentSession is closing due to unrecoverable error

/

job is unresponsive

. Likely root causes • Network/connectivity problems from agent workers to LiveKit and/or STT endpoints (blocked ports, DNS, or transient network outages). • STT/inference concurrency or plan limits causing retries to exhaust (STT may be rate-limited or not allowed at your current plan). • Resource exhaustion or process hangs on agent workers under load (CPU, memory, or too few warm processes), causing timeouts and unresponsiveness. Immediate checks (do these now) 1. Verify plan limits ◦ Confirm your Cloud plan supports the concurrent sessions you’re testing. 250 concurrent sessions requires a Scale plan — lower plans will hit quota limits and cause connection/retry failures. 1. Confirm connectivity from agent pods ◦ From a running agent pod/container, run: ▪︎ DNS resolve/livekit URL:

nslookup <your-livekit-host>

or

dig

▪︎ WebSocket/connect test (curl won’t test wss, but check TCP):

curl -v https://<your-livekit-host>/health

(if available) and run the LiveKit connection test utility from that host. ◦ If behind a firewall or NAT, ensure outbound TLS (443) is allowed and WebSocket/TURN endpoints aren’t blocked. 1. Check STT/inference quotas & endpoints ◦ Ensure your STT provider credentials are valid and the plan supports your concurrency. Free/lower tiers often have very low concurrency limits causing the repeated

failed to recognize speech after 4 attempts

. 1. Tail agent logs for connection details ◦ Run:

lk agent logs --id CA_ubuZhpVxLJvL

and search for repeated connection errors, DNS failures, or timeouts. Copy a few consecutive error entries if you want me to inspect them. Configuration fixes to apply quickly 1. Increase

num_idle_processes

to reduce cold starts while you stabilize: ◦ Start with

num_idle_processes = 5

. If STT preloads heavy models, raise to 10–15 and retest. 1. Ensure agents run on sufficiently sized instances: ◦ Use CPU-optimized machines (16+ cores recommended) if many STT jobs run concurrently. 1. If STT is a bottleneck, reduce per-call STT concurrency or switch to a higher-throughput STT model/provider. Testing steps to validate fixes 1. Run a small connectivity check from agent pod (DNS + TLS). 2. Run a small staged load test: 10 → 50 → 100 participants, watch for the same errors. 3. If connectivity and small loads are fine, ramp further to 250 following the staged plan. If you want help right now • Paste the output of a short agent log excerpt showing the WebSocket/STT errors (3–6 lines around one failure). • Tell me your Cloud plan (Build/Ship/Scale) and current

num_idle_processes

value. I’ll parse the logs and recommend the exact

num_idle_processes

, worker sizing, and next test stage. Sources: Agent server options | LiveKit docs | Deploy and scale agents on LiveKit Cloud | Quotas and limits | LiveKit docs | LiveKit Pricing | Diagnosing Connection Errors with Connection Test Utility

what causes {"level":40,"time":1765921007791,"pid":136,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-66fbc5cdb6-ghg22","msg":"job is unresponsive"} LiveKit Assistant

"job is unresponsive" means the agent process handling a job stopped responding to its internal health checks or callbacks and the job dispatcher marked it as hung. Common causes and how to diagnose/fix: Causes • Resource exhaustion on the worker (CPU, memory, thread pool) causing event loop stalls or slow I/O. • Network failures: loss of connectivity to LiveKit signaling, STT/TTS, or external APIs causes long blocking retries. • Exhausted idle/warm processes so the job hits a cold start and the process never becomes responsive in time. • Blocking or long-running synchronous work on the main event loop (e.g., heavy model loads, synchronous I/O). • External API rate limits or timeouts (STT provider) that cause repeated retries and eventual job hang. • Process crashes or GC pauses that make the process unresponsive briefly but long enough for the job watchdog to mark it hung. Immediate diagnostic steps 1. Tail logs around the unresponsive message (±30s) and capture preceding errors (WebSocket, STT, OOM, stack traces). 2. Check CPU / memory / thread metrics for the pod/process at the timestamp. Note sustained high CPU or OOM kills. 3. Verify network reachability from that worker to LiveKit and your STT/TTS endpoints (DNS, TLS, port 443). 4. Inspect agent-side retries: look for repeated APIConnectionError or long retry loops before the unresponsive log. 5. Confirm

num_idle_processes

and worker count at the time — were idle processes exhausted? 6. If Node: look for long event-loop blocks or unhandled promise rejections in logs; enable async-profiler or 0.01s event-loop sampling during tests. Short-term mitigations • Increase

num_idle_processes

(start at 5) to reduce cold starts. • Ensure agents run on larger instances (more CPU) and avoid burstable instance types. • Add timeouts and backoff on external API calls so retries don’t block indefinitely. • Offload heavy model loads to prewarm hooks so job handlers remain non-blocking. • Stagger job starts in load tests (small ramp) to avoid simultaneous spikes. Longer-term fixes • If STT/inference is a bottleneck, upgrade provider plan or use a higher-throughput model. • Add observability (metrics for job latency, queue depth, per-process CPU/memory) and alert on idle-process exhaustion. • If you need fixed scaling policy/control, consider self-hosting agents with orchestrator-based autoscaling (K8s HPA) so you can set min/max instance counts. If you want, paste: • 30–60s of logs before the "job is unresponsive" line, and • Your current

num_idle_processes

, worker count, and machine size. I’ll parse them and give exact next steps (specific config values and a short retest plan).

this is my agent ts for livekit cloud 2805 agent: fileURLToPath(import.meta.url), 2806 agentName: 'voice-agent', // Generic agent that executes workflow steps from metadata 2807 2808 - // Production scaling: 5 idle processes per agent to prevent worker overload 2809 - numIdleProcesses: 3, // Reduced from 12→5→3 to fix runner initialization timeouts 2808 + // Production scaling: 5 idle processes per agent for warm capacity without cold-start delay 2809 + numIdleProcesses: 5, // Baseline for LiveKit Cloud - increase to 10-15 if timeouts persist 2810 loadThreshold: 0.75, // Reject new jobs at 75% load (was 0.95) - leaves headroom 2811 jobMemoryLimitMB: 1200, // Per-job memory limit (VAD needs ~400MB only) does this work fine

Thanks — I reviewed your agent config snippet. Quick answers and recommendations: 1)

numIdleProcesses: 5

— good baseline • Use 5 idle processes per agent as you set. It provides warm capacity without excessive memory overhead. If you still see timeouts during load tests, raise to 10–15 and re-run the failing stage. 2)

loadThreshold: 0.75

— acceptable but not configurable in Cloud dispatch • Setting a lower

loadThreshold

helps a self-hosted agent reject new jobs before overload. LiveKit Cloud manages load and dispatch itself, so this field may have limited effect in Cloud. Keep it as a guard if your agent code evaluates it locally. 3)

jobMemoryLimitMB: 1200

— reasonable safeguard • 1200 MB per job is conservative and should cover VAD + light STT work (VAD ~400MB). If you preload heavy models, increase accordingly. Monitor OOMs and per-process RSS to tune. 4) Other practical checks to avoid "job is unresponsive" and APIConnectionError • Ensure agents run on non-burstable, CPU-heavy instances (16+ cores recommended for large concurrent STT). • Prewarm heavy resources in your prewarm/setup function so job handlers stay non-blocking. • Add timeouts/backoff on external API calls (STT) so retries don’t block the session loop. • Emit metrics for idle-process exhaustion, job start latency, and per-process CPU/memory. 5) What to monitor immediately after deploy • Idle processes over time (are they reaching 0?). • Job queue depth and TimeToJobStart. • Agent pod CPU, memory, and thread utilization. • Repeated APIConnectionError and STT retry failures in logs. 6) Next concrete step I recommend now • Keep

numIdleProcesses = 5

. Run the staged load test (100 → 200 → 250). If you see idle processes hit 0 or timeouts at any stage, increase to 10 and retest. Share a short log excerpt (30–60s) showing the first timeouts and the output of `ps`/memory for that process if you want exact tuning. If you want, paste your full prewarm/setup function and I’ll suggest small code changes to make startup non-blocking and more resilient.

2811 jobMemoryLimitMB: 1200, // Per-job memory limit (VAD needs ~400MB only) can you configure this for livekit cloud or no ?

You cannot set

jobMemoryLimitMB

in LiveKit Cloud. Cloud manages memory and resource allocation for agent jobs automatically; that low‑level per‑job memory limit is only configurable when you self‑host agents (e.g., in Kubernetes or your own VMs). What you can do now • Keep

jobMemoryLimitMB

in your agent code for self‑hosted runs, but it has no effect on LiveKit Cloud. • Follow these recommendations to avoid memory issues in Cloud: ◦ Provision larger agent instances (LiveKit recommends multi‑core, 8GB+ RAM per agent server for heavy voice workloads). ◦ Prewarm heavy models in your setup/prewarm hook so jobs don’t allocate large memory on demand. ◦ Monitor agent memory usage and OOM logs via the Cloud dashboard and

lk agent logs

. ◦ If you need strict control over memory limits or custom scaling policies, move to a self‑hosted deployment where you can configure container memory limits and autoscaling (I can provide Kubernetes HPA/manifest examples if you want). Would you like example Kubernetes YAML for setting per-job memory limits, or help interpreting current memory metrics from your logs? Sources: Deploying to custom environments | LiveKit docs

what about - shutdownProcessTimeout: 60000 can we configure this and nah we will stay with cloud for now

You cannot configure

shutdownProcessTimeout

in LiveKit Cloud. Cloud manages instance lifecycle and graceful shutdown timing itself; low-level process shutdown timeouts are not exposed for customers to change. What you can do to improve graceful shutdown behavior • Prewarm resources in your agent’s setup hook so work completes quickly when a shutdown begins. • Make your job handlers idempotent and implement short, bounded cleanup steps so sessions can finish within the platform’s drain window. • Add fast fail/cleanup paths for unrecoverable errors (close websockets, cancel long-running retries) so processes don’t hang during drain. • Emit a final status/log line when a job begins shutdown to help correlate shutdown timing in logs: ◦ e.g., log "shutdown requested — finishing work" and then "shutdown complete". • Monitor Cloud dashboard and agent logs for drain/shutdown events to confirm sessions finish gracefully. If you want, paste your prewarm/setup and shutdown handling code and I’ll suggest minimal changes to ensure clean, quick shutdowns under LiveKit Cloud’s managed drain window.

what is the common sweet spot for idle proceesess and should my vad model be shared between all of the idle processes or a new instanation for each one

Good question — short answer and the why: • Sweet spot: start with 5–10 idle processes per worker. ◦ Use 5 as a baseline for most voice workloads. Increase to 10–15 only if staged load tests still show idle processes hitting zero or you preload heavy models. • VAD model: load once in prewarm and share it across the worker’s processes (store on

proc.userData

). ◦ Why share: a VAD model is memory- and CPU‑heavy to load; creating one per idle process multiplies memory usage and slows startups. Preloading once gives fast job starts and far lower memory footprint. Quick Node-style example

export default defineAgent({
  prewarm: async (proc) => {
    proc.userData.vad = await loadVADModel(); // load once per worker
  },

  entry: async (ctx) => {
    const vad = ctx.proc.userData.vad;
    // use shared vad for this job
  },

  numIdleProcesses: 5,
});

Tuning checklist • Run staged load tests (100 → 200 → 250). If idle processes reach 0 or jobs timeout, raise idle processes to 10, then 15. • Monitor per-process RSS and overall worker memory; if memory climbs too high, increase instance size instead of creating more VAD instances. • Prewarm other heavy models (STT/TTS) similarly to avoid blocking during job start. If you share observed memory per process and results from a short 100→200 test (or the logs showing idle processes exhaustion), I’ll recommend exact

numIdleProcesses

and instance sizing. Sources: Agent server options | LiveKit docs | Silero VAD plugin | LiveKit docs | Events and error handling | LiveKit docs | Turn detection and interruptions | LiveKit Docs