how should we implement a worker pool?

Thanks for reaching out, Utkarsh. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

i am using python, then how can i create a worker pool and maintain its lifecycle properly?

Thanks — I’m responding to your question about implementing a worker pool in Python and keeping its lifecycle healthy inside the LiveKit Agents environment. Below are practical patterns you can use, with code examples and lifecycle best practices you can apply immediately. Which part are you most interested in? • A process-level prewarmed pool for heavy resources (recommended), or • An explicit async task queue worker pool for fine-grained job scheduling? Tell me which and any details about your workload (CPU- or I/O-bound, expected job duration, whether tasks must preserve ordering) and I’ll tailor the example. Process-level prewarmed workers (recommended) • What it is: Load heavy resources (ML models, DB/HTTP clients, etc.) once per worker process and store them in

proc.userdata

. All sessions running in that process reuse those resources. • Why use it: Avoids repeated expensive initialization per session; keeps resource lifecycle tied to process lifecycle which is easier to manage. Example pattern 1. In your AgentServer

setup_fnc

, create and attach the shared resource(s) to

proc.userdata

. 2. Use those prewarmed resources inside session handlers via

ctx.proc.userdata

. 3. Clean up in process shutdown if needed. Sketch:

# in server setup
def prewarm(proc: JobProcess):
    proc.userdata["model"] = MyHeavyModel.load()
    # optional: create a WorkerPool object here and start it
    proc.userdata["pool"] = WorkerPool(num_workers=4)
    asyncio.create_task(proc.userdata["pool"].start())

server = AgentServer(setup_fnc=prewarm)

Session usage:

@server.rtc_session(agent_name="worker-agent")
async def entrypoint(ctx: JobContext):
    model = ctx.proc.userdata["model"]
    pool = ctx.proc.userdata["pool"]

    # submit async work to the pool or run directly
    await pool.submit(lambda: model.predict(some_data))

    # ensure graceful cleanup for session-local tasks
    async def on_shutdown():
        await pool.shutdown()  # or other cleanup actions

    ctx.add_shutdown_callback(on_shutdown)
    ...

Explicit asyncio.Queue worker pool (fine-grained control) • What it is: An async in-process queue with N worker coroutines consuming jobs. Useful when you need task scheduling, backpressure, or to limit concurrency for I/O-bound tasks. • Where to put it: Create it in

setup_fnc

and store it on

proc.userdata

, so it’s shared by sessions in that process. • Lifecycle: Start it once when process starts; on process/session shutdown, drain the queue then cancel worker coroutines. WorkerPool class (pattern)

import asyncio

class WorkerPool:
    def __init__(self, num_workers):
        self.queue = asyncio.Queue()
        self.workers = []
        self.num_workers = num_workers
        self._started = False

    async def worker(self):
        while True:
            job = await self.queue.get()
            try:
                await job()
            except asyncio.CancelledError:
                break
            except Exception as e:
                # handle job error (logging, metrics)
                pass
            finally:
                self.queue.task_done()

    async def start(self):
        if self._started:
            return
        self._started = True
        for _ in range(self.num_workers):
            self.workers.append(asyncio.create_task(self.worker()))

    async def submit(self, coro_func):
        await self.queue.put(coro_func)

    async def shutdown(self, wait_timeout: float = 10.0):
        # wait for queued work to finish
        try:
            await asyncio.wait_for(self.queue.join(), timeout=wait_timeout)
        except asyncio.TimeoutError:
            # not all tasks finished in time
            pass
        # cancel workers
        for w in self.workers:
            w.cancel()
        await asyncio.gather(*self.workers, return_exceptions=True)

Lifecycle integration tips • Start pool in

setup_fnc

so it is created once per process. Use

asyncio.create_task(pool.start())

. • Use

ctx.add_shutdown_callback()

inside sessions to cancel any session-specific background tasks and to ensure orderly shutdown. • For process termination (e.g., container termination), ensure your environment allows a grace period (Kubernetes

terminationGracePeriodSeconds

) so

shutdown()

can drain queued jobs. • Avoid creating separate thread pools per session — prefer sharing per-process resources via

proc.userdata

. Sizing and tuning • Workers: For I/O-bound tasks, you can have more workers than CPU cores; for CPU-bound tasks, use roughly

num_cpus

workers. • Backpressure: Limit queue size (wrap

submit()

to reject or block when queue is full) if producers can outpace consumers. • Job size: If jobs are short and frequent, prefer many small async tasks; if jobs are long/blocking, run them via

run_in_executor

or isolate in separate processes. Error handling and observability • Ensure errors inside jobs are caught and logged; consider a poison-job policy to avoid repeated failures. • Add metrics for queue length, job latency, worker errors so you can tune pool size and timeouts. • If you need tracing per job, propagate context or use a lightweight job wrapper that attaches trace IDs. Graceful shutdown flow 1. Stop accepting new jobs (e.g., set a flag or stop producers). 2. Wait for queue to drain with a timeout. 3. Cancel worker coroutines. 4. Close shared resources (model, DB connections, HTTP clients). 5. Exit process. Next steps I can help with • Provide a complete runnable example integrating this pool into your AgentServer and session code. • Recommend sizing and queue limits if you share whether tasks are CPU- or I/O-bound and an average task duration. • Show an example that uses

run_in_executor

for blocking CPU-bound work. Tell me which option you want (prewarmed resources vs. explicit queue pool) and a few details about your workload, and I’ll produce a concrete, runnable example tailored to your use case. Sources: Gemini Realtime Agent with Live Vision | LiveKit Documentation | Collect recording consent with tasks | LiveKit Documentation | Job lifecycle | LiveKit Documentation | Self-hosted deployments | LiveKit Documentation

if i have 500k tokens/minute available at LLM and i have 2gb ram and .5 cpu core, how many calls i can handle if i have 2 pods running the AgentServer with the same config of 2gb ram and .5 cpu core?

Thanks — I’m responding to your question about how many LLM calls your two AgentServer pods can handle given 500k tokens/min available and each pod having 2 GB RAM and 0.5 CPU. I can give a useful estimate, but I need one key detail from you to be accurate: what is the average number of tokens per call (including prompt + response)? A few example choices you could give: • Average tokens per call (e.g., 100 tokens/call) • Or distribution: e.g., 70% are ~50 tokens, 25% are ~200 tokens, 5% are ~1000 tokens What I can estimate now (with assumptions) • If you only consider token throughput (500,000 tokens/min total from your LLM provider): ◦ Calls per minute = 500,000 / (avg tokens per call). ◦ With 2 pods sharing the same load, each pod would handle half the calls (assuming even distribution and no other bottlenecks). • CPU and RAM can become limiting factors depending on per-call overhead (concurrency, model streaming, local preprocessing, or memory used for request buffers). With only 0.5 CPU and 2 GB RAM per pod, you may be limited by: ◦ How many concurrent sessions each pod can maintain (memory per session). ◦ How much CPU time each call requires for preprocessing/postprocessing and I/O. • If you don’t supply avg tokens/call, here are quick examples: ◦ 100 tokens/call -> 5,000 calls/min total -> ~2,500 calls/min per pod ◦ 500 tokens/call -> 1,000 calls/min total -> ~500 calls/min per pod ◦ 1,000 tokens/call -> 500 calls/min total -> ~250 calls/min per pod Follow-ups I need to refine this into a practical capacity plan 1. Average tokens per call (or distribution). 2. Is each call synchronous and blocking until LLM responds, or do you stream results? (Streaming reduces peak memory but may increase concurrent open connections.) 3. Typical CPU/RAM usage per request on the AgentServer (if you have measurements or approximate values). 4. Any extra per-call work (e.g., heavy preprocessing, embeddings, caching) that uses CPU or memory? 5. Target concurrency goals (e.g., do you want headroom for spikes, or run near 100% token capacity?) If you give the average tokens per call and indicate whether you stream responses, I’ll produce: • A calls-per-minute and concurrent-calls estimate per pod. • Suggested concurrency limits and memory/CPU safety margins. • Practical tuning tips (queue sizes, prewarming, batching) for your 0.5 CPU / 2 GB pods.

with 500k tokens/minute available at the LLM, how many pods are ideal?

Thanks — I’m responding to your question about how many pods are ideal to handle 500k tokens/minute at the LLM. I can give a clear recommendation, but I need one key piece of information to produce an accurate pod count: what is the average number of tokens per LLM call (prompt + response)? If you don’t have an exact number, an estimate or distribution (e.g., 70% 50 tokens, 25% 200 tokens, 5% 1000 tokens) is enough. Why I need this • Total calls/minute = 500,000 tokens / (avg tokens per call). Without avg tokens/call we can’t convert tokens into request rate. • Pod count depends on both the request rate and how many concurrent calls a pod can handle (which depends on CPU, RAM, and per-call processing). If you give me avg tokens per call, I’ll return: • Calls per minute and calls per second total. • Recommended number of pods for the token throughput, split evenly across pods. • Guidance on whether your pods’ 0.5 CPU / 2 GB RAM will be the bottleneck and practical safety margins. • Tuning suggestions (concurrency limits, queueing/backpressure, streaming vs. blocking calls) and autoscaling tips. Quick examples to illustrate (assume token-only bottleneck, evenly distributed load): • If avg = 100 tokens/call → 500,000/100 = 5,000 calls/min → 83.3 calls/sec. ◦ With 2 pods: 2,500 calls/min (~41.7 calls/sec per pod). ◦ To increase headroom, use more pods (e.g., 4 pods → 1,250 calls/min per pod). • If avg = 500 tokens/call → 1,000 calls/min total → 500 calls/min per pod with 2 pods. • If avg = 1,000 tokens/call → 500 calls/min total → 250 calls/min per pod with 2 pods. Important resource considerations beyond token capacity • 0.5 CPU and 2 GB RAM per pod is small. If your AgentServer does any nontrivial preprocessing/postprocessing, maintains many concurrent sessions, or holds buffers while streaming, CPU and memory can become the limiting factor before token quota. • For CPU-bound work, aim for roughly 1 worker per core; for I/O-bound (network/LLM) work, you can sustain more concurrent requests per CPU. • Prefer autoscaling (horizontal pods) based on real metrics (CPU, request latency, or custom token-usage metrics) and leave headroom for spikes. If you share: 1) Average tokens per call (or a distribution), and 2) Whether calls are streamed or blocking, plus any per-call CPU/RAM estimates, I’ll compute an ideal pod count with conservative headroom and give concrete autoscaling and tuning recommendations. Which of those can you provide? Sources: Build voice, video, and physical AI | LiveKit | Self-hosted deployments | LiveKit Documentation | Troubleshooting Latency and Timeout Errors with Turn Detection on AWS

per call we are consuming 10000 tokens and average call duration is 35 seconds

Thanks — I’m responding to your capacity question using the numbers you gave: 500,000 tokens/min total available from the LLM, 10,000 tokens per call, and average call duration 35 seconds. I’ll show the math, give a recommended pod count, and call out resource constraints and practical autoscaling/tuning advice for your AgentServer pods (2 GB RAM / 0.5 CPU each). 1) Token → calls math • Calls per minute = 500,000 tokens / 10,000 tokens per call = 50 calls/minute total. • Calls per second = 50 / 60 ≈ 0.833 calls/sec. 2) Concurrency implied by call duration • Each call lasts 35 seconds, so average concurrent calls = calls_per_second * duration = 0.833 * 35 ≈ 29.17 concurrent calls (≈29–30 concurrent sessions) across the whole system. 3) With 2 identical pods (evenly split) • Concurrent calls per pod ≈ 29.17 / 2 ≈ 14.6 → roughly 14–15 concurrent calls per pod. • Calls per minute per pod = 50 / 2 = 25 calls/minute → ~0.417 calls/sec per pod. 4) Resource fit check (2 GB RAM, 0.5 CPU per pod) • 0.5 CPU is very small for ~15 concurrent calls if any nontrivial per-call CPU work (pre/postprocessing, parsing, encoding/decoding) is done. CPU will likely be a bottleneck if calls require CPU-bound work. • 2 GB RAM needs to hold runtime, session state, buffers, and any pooled resources. If each call requires substantial in-memory buffers (streaming chunks, prompt contexts), memory could become a bottleneck as well. • If calls are mostly I/O-bound (the pod waits on LLM network/IO and does minimal CPU work), 0.5 CPU may suffice for ~14 concurrent calls. If calls involve any CPU-heavy tasks, expect CPU saturation and degraded latency. 5) Recommended pod count and options Option A — Minimal pods (based on tokens alone) • You can run with 2 pods as you described; token capacity supports it (each pod handles ~25 calls/min and ~14–15 concurrent calls). • Risk: With only 0.5 CPU / 2 GB per pod, you may see CPU or memory saturation under normal processing. This is high risk unless per-call CPU/memory usage is confirmed to be very small. Option B — Safer baseline (recommended) • Increase pods so concurrency per pod is lower and there’s CPU headroom. Target ~4 pods: ◦ Concurrent calls per pod ≈ 29.17 / 4 ≈ 7.3 (7–8 concurrent). ◦ Calls/min per pod = 12.5. ◦ This reduces per-pod load and gives more resiliency to spikes and background tasks. • If 0.5 CPU is fixed and per-call CPU work is nontrivial, consider using 4–6 pods to reduce contention. Option C — Resource-first approach (strongly recommended) • Rather than fixing pods at 0.5 CPU, increase pod sizing to something like 1 CPU / 2–4 GB RAM or 2 CPU / 4–8 GB RAM, then use fewer pods. Example baseline from prior LiveKit sizing: 4 CPU / 8 GB handled multiple concurrent jobs comfortably. If you can move to 1–2 CPU per pod, you can run fewer pods with better latency and simpler autoscaling. 6) Autoscaling & operational tips • Autoscale horizontally using real metrics: ◦ Primary metric: CPU utilization (if per-call CPU matters). ◦ Secondary: queue length, request latency, or concurrent sessions. ◦ If you can emit token-usage or requests/min metrics, use them for a custom HPA so scaling tracks LLM token consumption. • Leave headroom: scale before pods hit 70% CPU. For short spikes, allow buffer because draining sessions on pod termination takes time. • Graceful shutdown: ensure terminationGracePeriodSeconds is long enough to let in-flight calls finish (35s average → add safety margin). • Limit concurrency per pod: enforce a per-pod max concurrent calls (e.g., 6–8) to protect CPU/ram. Use an in-process pool or Kubernetes concurrency limits. • Backpressure: if token quota is reached, reject/queue requests upstream rather than overloading pods. 7) Quick recommended next steps • Measure per-call CPU and memory on a representative pod (run a load test that simulates a 35s, 10k-token call). That will tell you whether 0.5 CPU / 2 GB is viable. • If you can’t increase pod CPU, start with 4 pods (lower concurrency/pod) and monitor CPU/latency. • Implement autoscaling with CPU + custom request metrics and set graceful termination >= 60s so calls can finish. If you want, I can: • Produce a small load-test plan (requests/sec, concurrency targets) and scripts to measure per-call CPU/RAM so you can pick the right pod size. • Recommend specific HPA settings and per-pod concurrency limits given measured per-call resource usage. Which would you like next — a load test script to measure per-call resource use, or a recommended autoscaler configuration assuming 0.5 CPU/2 GB pods and a target safety margin?

if i have to run 10k calls per day and my llm quota is 500k tokens/minute, how many pods are required and what should be the configuration for each pod?

Thanks — I’m responding to size pods for running 10,000 calls/day given your LLM quota (500k tokens/min) and the per-call numbers you previously shared (10,000 tokens per call, avg duration 35s). I’ll show the math, give a recommended number of pods, and propose pod configuration and operational recommendations. 1) Workload math • Calls per day: 10,000 (given). • Tokens per call: 10,000 → total tokens/day = 10,000 * 10,000 = 100,000,000 tokens/day. • Your LLM quota is 500,000 tokens/minute (this is a rate limit, not a daily cap). Translate the daily workload into required sustained and peak throughput: • Average calls per minute across a full day = 10,000 calls / 1,440 minutes ≈ 6.94 calls/min. • Average token usage per minute (spread evenly) = 6.94 * 10,000 = 69,444 tokens/min — well below the 500k tokens/min quota. • However, you are more likely concerned about concurrency and peak behavior because each call lasts 35s. If calls are uniformly distributed (no spikes): • Average calls per second = 6.94 / 60 ≈ 0.116 calls/sec. • Concurrent calls (steady state) = calls_per_second * 35s ≈ 4.06 concurrent calls across the system. If traffic is bursty (peak windows), you must size for peak RPS rather than the daily average. Example: if 50% of calls occur in a 2-hour window: • Calls in 2 hours = 5,000 calls → calls per minute during peak = 5,000 / 120 ≈ 41.7 calls/min. • Tokens/min during that peak = 41.7 * 10,000 ≈ 417,000 tokens/min — still under 500k quota but much closer. • Concurrent calls at that peak = (41.7 / 60) * 35 ≈ 24.3 concurrent calls. Deciding which scenario to size for depends on your traffic pattern. I’ll give recommendations for both uniform and a reasonable peak case (50% of daily traffic in 2 hours). Tell me if you expect different burstiness. 2) Pod count — two sizing approaches A. Minimal pods for uniform distribution (steady low load) • System-wide concurrent calls ≈ 4. • If you keep pods small (example: 0.5 CPU / 2 GB), you could run this on a single pod safely from a tokens-rate perspective. • But 0.5 CPU / 2 GB is tight if any per-call CPU/RAM work exists. For resilience and maintenance, use at least 2 pods. Recommendation (uniform): 2 pods • With 2 pods, concurrent calls per pod ≈ 2, giving strong headroom. B. Safer sizing for a peak window (50% of calls in 2 hours) — recommended if traffic is bursty • Peak concurrent calls ≈ 24. • Choose a target concurrent calls per pod to avoid CPU/RAM saturation. With small pods (0.5 CPU / 2 GB), keep concurrency low (e.g., 6–8 concurrent calls/pod). With this target: ◦ Required pods = ceil(24 / 8) = 3 pods (if you allow ~8 concurrent calls/pod). ◦ For more headroom, pick 4 pods → 6 concurrent calls/pod. • If per-call CPU work is nontrivial, use more pods or larger pods. Recommendation (peak): 3–4 pods (minimum 3, safer 4) 3) Pod configuration guidance (practical recommendations) Given per-call duration and tokens, the real bottleneck is likely CPU and memory, not token quota. Choose one of two practical options: Option A — Small, many pods (if CPU per call is low / mostly I/O-bound) • Pod size: 0.5 CPU, 2 GB RAM (your current spec). • Concurrency cap per pod: 6–8 concurrent calls (enforced by in-process semaphore or server concurrency limit). • Estimated capacity per pod under bursty peak: ~6–8 concurrent calls → 4 pods handle ~24–32 concurrent calls. • Use Kubernetes HPA on CPU + custom concurrency metric. Keep terminationGracePeriodSeconds >= 60s so in-flight calls (avg 35s) can finish. Option B — Larger pods, fewer instances (recommended if any CPU-bound work) • Pod size: 1–2 CPU, 4 GB RAM (prefer 2 CPU/4–8 GB for more headroom). • Concurrency cap per pod: 10–20 depending on CPU (I/O-bound allows higher). • With 2 CPU/4 GB, 2 pods could comfortably handle the peak 24 concurrent calls with good latency. • Easier to autoscale and manage fewer pods; better for CPU-heavy preprocessing. 4) Autoscaling and operational best practices • Autoscale horizontally using CPU utilization and a custom concurrency or request metric tied to in-flight calls. Scale before pods reach ~70% CPU. • Implement a per-pod concurrency limit (semaphore) so pods don’t accept more concurrent calls than they can handle. • Graceful shutdown: set Kubernetes terminationGracePeriodSeconds >= 60s to allow 35s average calls to finish. • Backpressure: if peak load would exceed tokens/min quota (e.g., sudden spike), queue or reject new requests rather than overprovisioning LLM usage. • Monitoring: emit metrics for in-flight calls, tokens/min used, request latencies, CPU/RAM per pod. 5) Practical recommendation (summary) • If traffic is uniform: 2 pods (but 2 is mainly for HA; 1 could technically handle the load). • If traffic is bursty (example 50% of calls in 2 hours): 3–4 pods of 0.5 CPU/2 GB to limit concurrency to ~6–8 per pod; better: 2 pods sized at 2 CPU/4 GB (or 1–2 CPU/4 GB each) to reduce pods and increase headroom. • If you can choose only one config and want safety: pick 4 pods at 0.5 CPU/2 GB, with a per-pod concurrency cap of 6 and autoscaling in place. That balances cost, resilience, and headroom. 6) Next steps I can help with • Produce an autoscaler (HPA) config example using CPU + custom concurrent requests metric. • Give a small load test script to measure per-call CPU/RAM on a representative pod so sizing is data-driven. • Recommend exact concurrency limits per pod after you run a short load test. Which would you like next — an HPA example or a load-test script to validate how many concurrent calls a pod can actually sustain?

what is the num_idle_processes for livekit in self hosted mode?

Thanks — I’m responding to your question about the

num_idle_processes

setting for LiveKit in self‑hosted mode. Short answer: • There is no documented default value for

num_idle_processes

in the sources. If you don’t explicitly configure it, the system behaves as if

num_idle_processes = 0

(no prewarmed processes kept ready). • You must set

num_idle_processes

in your AgentServer configuration (or equivalent deployment config) to keep prewarmed worker processes available for faster session start. What this parameter does •

num_idle_processes

controls how many prewarmed job processes are kept idle and ready to handle incoming jobs. • Prewarming is useful for loading heavy resources once per process (models, clients, connection pools) via the

prewarm

/

setup_fnc

hook, reducing per-job startup latency. Practical guidance for self‑hosted deployments • If you want faster session startup and can afford the memory/CPU, set

num_idle_processes

> 0 so processes load models at startup and stay ready. • If you leave it unset (or set to 0), each job will start fresh and perform its own initialization, which increases per-job latency but reduces idle resource usage. • Tune the value based on: ◦ Memory/CPU cost per prewarmed process (models loaded, connection pools). ◦ Expected incoming job rate and acceptable startup latency. ◦ Pod sizing and instance limits in your deployment environment. If you’d like, I can: • Show a code snippet for where to set

num_idle_processes

in your AgentServer setup (Python or Node), or • Help pick a starting value based on your per-process memory footprint and expected request rate. Which would you prefer? Sources: Server options | LiveKit Documentation | Simple Content Filter | LiveKit Documentation | Agents (Python) | livekit/livekit_composite | DeepWiki

is it mandatory to set, what if i do not configure num_idle_processes, then how many workers are spun?

Thanks — I’m responding to whether

num_idle_processes

is mandatory and what happens if you don’t configure it. • It is not mandatory to set

num_idle_processes

. • If you do not configure it, the system behaves as if `num_idle_processes = 0`: no prewarmed idle job processes are kept. Jobs will start fresh and perform their own initialization when dispatched. • That means no idle/prewarmed worker processes are spun just for waiting; workers are created on demand per job (subject to whatever process lifecycle your AgentServer deployment or runtime enforces). Practical notes • Leaving it at 0 reduces idle memory/CPU usage but increases per-job startup latency (since heavy resources like models must be loaded at job start). • If you want faster session startup, set

num_idle_processes

> 0 and size it based on the memory/CPU cost per prewarmed process and your expected job arrival rate. If you’d like, I can show a short code snippet for setting

num_idle_processes

in your AgentServer setup (Python or Node) and help pick a starting value based on your per-process resource footprint. Which language do you use?

does

num_idle_processes

sets the numbers of workers?

Thanks — I’m responding to whether

num_idle_processes

sets the number of worker processes. Short answer •

num_idle_processes

controls how many prewarmed (idle) job processes the AgentServer keeps ready to accept new jobs. In that sense it does set how many idle worker processes are kept spun up. • It does not necessarily define the absolute total number of worker processes the server may use over time; it only specifies how many idle/prewarmed processes are maintained for immediate use. Additional processes may still be created on demand according to the server’s runtime behavior (i.e., jobs started without an idle process will spawn processes as needed). Behavior summary • If you set `num_idle_processes = N`: the server will prewarm and maintain N idle job processes (each running your `prewarm`/`setup_fnc`) so new sessions can start with low latency. • If you do not set it (or set it to 0): no prewarmed idle processes are kept. Workers are created on demand when jobs arrive, which increases startup latency but conserves idle CPU/RAM. • It is not mandatory to set this value. Practical note • Treat

num_idle_processes

as the number of ready-to-serve processes you want kept warm. Pick it based on the memory/CPU cost of a prewarmed process and your expected job arrival rate (more arrivals → larger value for lower latency; constrained resources → smaller or zero). • If you need, I can show a short code snippet for setting

num_idle_processes

in your AgentServer setup (Python or Node) and help pick a starting value based on your process memory footprint and expected arrival rate. Which language do you use?

what is the relation between Agent Server, workers and

num_idle_processes

. How do they work together and what is their dependence

Thanks — I’m responding to how the AgentServer, worker processes, and

num_idle_processes

relate, how they work together, and what depends on what. Below is a concise, practical explanation with lifecycle and operational notes so you can tune behavior for latency, cost, and resiliency. High-level roles • AgentServer ◦ The central process that accepts incoming jobs/sessions (RTC sessions, agent tasks) and coordinates dispatch. ◦ Responsible for lifecycle management of jobs and for applying your

setup_fnc

/

prewarm

logic when workers start. • Worker processes (job processes) ◦ Separate child processes (or runtime units) that actually run agent sessions and execute job logic. ◦ Each worker loads any heavy resources in its prewarm/setup step (models, clients, connection pools) and holds per-process state in

proc.userdata

. ◦ Workers execute sessions in isolation from each other so one session’s crash or memory pressure won’t necessarily affect others in a different process. • num_idle_processes ◦ A configuration that tells the AgentServer how many worker processes to keep prewarmed and idle, ready to accept new jobs immediately. ◦ These idle workers run your

setup_fnc

so heavy resources are already loaded, reducing per-job startup latency. How they work together (lifecycle) 1. Startup ◦ AgentServer starts. If

num_idle_processes > 0

, it spawns that many worker processes and runs the prewarm/setup function in each. Those processes sit idle, waiting for assignments. 1. Job arrival ◦ When a new job/session arrives, AgentServer: ▪︎ Preferentially assigns it to an idle/prewarmed worker (fast path). ▪︎ If no idle worker is available, it may spawn a new worker on demand (slow path) or queue the job depending on your runtime’s behavior. 1. During job execution ◦ The assigned worker runs the job; heavy resources from prewarm are reused via

proc.userdata

. 1. Job completion ◦ After the job finishes, the worker either: ▪︎ Returns to the idle pool (if you keep it alive), or ▪︎ Terminates (if it was created for that job and not kept idle). ◦ AgentServer keeps

num_idle_processes

workers ready. If a worker died or was used up, AgentServer may spawn new idle workers to maintain that count. 1. Shutdown / scaling down ◦ AgentServer or orchestrator terminates workers; ensure graceful shutdown callbacks are used to stop background tasks and allow in-flight jobs to finish. Dependencies and important interactions • AgentServer depends on workers to run actual sessions; it orchestrates spawning, assignment, and cleanup. • Workers depend on AgentServer for job assignments and for lifecycle signals (start, shutdown). •

num_idle_processes

affects: ◦ Latency: higher values reduce cold-start time per job because workers are already warmed. ◦ Resource usage: each idle worker consumes RAM/CPU (models, threads, file descriptors). More idle processes → higher idle cost. ◦ Throughput readiness: if jobs arrive in bursts, a larger idle pool absorbs bursts without spawning new processes. • Worker lifecycle depends on process-level prewarm and your cleanup hooks; make sure you register shutdown callbacks so workers stop background tasks and free resources. Practical trade-offs and tuning • Latency vs. cost ◦ Set

num_idle_processes

> 0 to reduce startup latency at the expense of idle resource usage. ◦ Set it to 0 to minimize idle cost but accept higher latency as workers are created on demand. • Resource sizing ◦ Estimate memory/CPU consumed by a prewarmed worker (models, clients). Divide available host resources by that to get a practical upper bound on prewarmed workers per node. • Arrival pattern ◦ If traffic is steady, smaller idle pools work because workers will be reused. ◦ If traffic is bursty, keep enough idle workers to cover typical burst size, or combine a small idle pool with fast on-demand worker creation. • Failure and resilience ◦ Workers are isolated: one worker crash normally doesn’t crash AgentServer or other workers, but AgentServer must detect the crash and replace the worker to maintain

num_idle_processes

. • Graceful shutdown ◦ Use shutdown callbacks in workers and set container terminationGracePeriodSeconds long enough to finish in-flight jobs (e.g., > average job duration). Concrete examples • Low-cost, low-latency need:

num_idle_processes = 2–4

per AgentServer instance if each prewarmed worker uses modest RAM/CPU. • High-burst workload: set idle count to match expected burst concurrency, or use autoscaling to add more AgentServer instances plus idle workers. • Memory-constrained node: if one prewarmed worker uses 1.5 GB, then

num_idle_processes

should be 0 or 1 on a 2 GB node to avoid OOMs. Next step • If you want, I can: ◦ Show the code snippet for setting

num_idle_processes

in Python or Node. ◦ Help pick a starting value given the per-worker memory/CPU footprint you see in a quick local test. ◦ Recommend an autoscaling approach (HPA rules + custom concurrency metrics) to combine prewarmed workers with horizontal scaling. Which would you like to do next?

how many workers can a AgentServer spawn?

Thanks — I’m responding to your question about how many worker processes an AgentServer can spawn. Short answer • There isn’t a single hard-coded “maximum workers” value in the AgentServer itself documented in the sources. The number of worker processes an AgentServer can spawn is determined by deployment configuration and the resources available on the host (CPU, RAM), plus any orchestration limits you put in place (container resource limits, pod/node capacity, or autoscaler rules). •

num_idle_processes

only controls how many prewarmed (idle) workers are kept ready; it does not by itself cap the total workers that can be created on demand. How it works in practice • Prewarmed idle workers: ◦ If you set

num_idle_processes = N

, the AgentServer will pre-spawn and keep N worker processes warmed (running your `prewarm`/`setup_fnc`) so incoming jobs can be assigned immediately. ◦ Those N processes consume RAM/CPU while idle. • On-demand workers: ◦ When no idle worker is available, the AgentServer may spawn additional worker processes to handle incoming jobs (slow path). The sources describe this behavior conceptually but do not specify a fixed upper bound. ◦ How many on-demand workers can be started depends on: ▪︎ Host resources (how many processes your node can run without OOM/CPU exhaustion). ▪︎ Any process- or orchestration-level limits you configure (max pods per node, container CPU/memory requests & limits, quotas). ▪︎ Your AgentServer or deployment configuration (some deployments may impose a cap or use autoscaling to add more AgentServer instances instead of unlimited per-instance workers). • Worker lifecycle: ◦ After a job completes, a worker may return to the idle pool (to maintain

num_idle_processes

) or exit depending on how you’ve configured lifecycle behavior and resource policies. Practical guidance for sizing and limits • Estimate per-worker resource use (RAM/CPU with prewarmed models). Divide node capacity by that footprint to determine a safe upper bound of concurrent worker processes on that host. • Use

num_idle_processes

to trade latency vs. idle cost: choose N so prewarmed workers cover expected bursts without exceeding host capacity. • Protect the node: set container resource requests/limits and use orchestration-level quotas so worker spawning cannot exhaust the node. • Prefer horizontal scaling: instead of letting one AgentServer spawn an unbounded number of workers, scale AgentServer pods horizontally (with each maintaining a reasonable idle pool). Use autoscaling policies (CPU, concurrency, or custom metrics) to keep latency and resource usage controlled. • Add a concurrency limit inside the AgentServer or orchestration rules so it won’t create more workers than the node can sustain. If you want, I can: • Help compute a safe per-node worker cap if you share the per-worker RAM/CPU footprint. • Provide an example config pattern (num_idle_processes, container limits, and HPA) that balances prewarm latency with node stability. Which would you like?

can you create a mermaid flow chart explaining the working of AgentServer, Workers and num_idle_processes?

Thanks — I’m responding with a Mermaid flowchart that explains how AgentServer, worker processes, and

num_idle_processes

interact, plus a short textual explanation. Copy the Mermaid code below into any Mermaid renderer (e.g., Mermaid Live Editor, VS Code Mermaid preview, or documentation that supports Mermaid) to view the diagram. Mermaid flowchart

flowchart TD
    subgraph AgentServerNode["AgentServer"]
        A[Start AgentServer]
        B{num_idle_processes > 0?}
        C[Spawn N idle workers\n(run prewarm/setup_fnc)]
        D[Listen for incoming jobs]
    end

    subgraph WorkerPool["Worker Processes"]
        direction TB
        I[Idle Worker (prewarmed)\n(proc.userdata loaded)]
        J[On-demand Worker (spawned)\n(initializes at job start)]
        K[Running Job\n(session/job executing)]
        L[Return to idle pool\nor exit]
    end

    subgraph Orchestration["K8s / Host / Limits"]
        O[Host CPU / RAM limits]
        P[Container resource limits / quotas]
        Q[Autoscaler / HPA]
    end

    A --> B
    B -- yes --> C
    B -- no --> D
    C --> I
    D -->|job arrives| R{Is idle worker available?}
    R -- yes --> I --> K
    R -- no --> S[Spawn on-demand worker] --> J --> K
    K --> T{Job complete}
    T -- keep warm --> L --> I
    T -- terminate worker --> L -->|exit| U[Worker process ends]
    U --> V[AgentServer may spawn new idle worker to maintain count]
    O -->|limits workers| S
    P -->|enforces limits| S
    Q -->|scale AgentServer instances| A

    %% Explanatory labels
    R:::decision
    S:::action

    classDef decision fill:#f9f,stroke:#333,stroke-width:1px;
    classDef action fill:#bbf,stroke:#333,stroke-width:1px;

Short explanation (how the pieces work together) • AgentServer is the central coordinator that accepts jobs and manages worker processes. • If

num_idle_processes

is set, AgentServer pre-spawns that many idle workers and runs your `prewarm`/`setup_fnc` so heavy resources (models, clients) are loaded into

proc.userdata

. • When a job arrives, AgentServer assigns it to an idle worker if available (fast path). If none are idle, it may spawn an on-demand worker (slow path) which initializes at job start. • After a job completes, the worker either returns to the idle pool (to maintain

num_idle_processes

) or exits, freeing resources. • The total number of workers that can exist depends on host/container resource limits and any autoscaling rules;

num_idle_processes

only sets how many are kept prewarmed and ready. • Kubernetes / host limits and HPA policies influence how many workers/AgentServer instances you can practically run. If you’d like, I can: • Produce a variant that shows lifecycle timing (startup, job assignment, shutdown) or • Produce a version with recommended values/annotations (e.g., where to set

terminationGracePeriodSeconds

, where to enforce concurrency limits). Which would you prefer?

how can i know if the create_dispatch was rejected by the livekit server? currently i am doing dispatch from scheduler as below:

try:
            await asyncio.wait_for(
                self.lkapi.agent_dispatch.create_dispatch(
                    CreateAgentDispatchRequest(
                        agent_name=self.agent_name,
                        room=room_name,
                        metadata=json.dumps({
                            "trunk_id": trunk_id,
                            "phone_number": phone_number,
                            "customer_data": customer_row,
                            "caller_phone_id": phone_number_id,
                            "resource_type": "phone"
                        })
                    )
                ),
                timeout=10.0
            )
            
            <http://logger.info|logger.info>(
                "✅ Call successfully dispatched to LiveKit for customer %s (room: %s)",
                customer_id,
                room_name
            )
            return True

Thanks — I’m responding to how you can detect if your

create_dispatch

call was rejected by LiveKit and how to handle the possible outcomes in your scheduler code. Short summary • A server-side rejection or error when calling

create_dispatch

will surface as an exception from the SDK/network layer. You must catch and inspect exceptions (and timeouts) to know the call failed. • A successful

create_dispatch

only means the dispatch request was accepted by LiveKit’s dispatch system (job queued/created). It does not guarantee the agent will accept the job later — agent-level rejection is a separate runtime event inside the agent and isn’t reported by

create_dispatch

. • To detect agent-level rejection, you must observe job lifecycle events (logs, callbacks, or monitoring) after the dispatch is created. Concrete changes to your snippet 1) Add exception handlers for timeout and API errors. 2) Log error details (message, status/code, metadata) so you can distinguish rejected vs transient network errors. 3) Optionally inspect the returned response (on success) for confirmation data. Example pattern (adapt to the exact exception types your SDK exposes):

from asyncio import TimeoutError
# replace ApiError with the SDK-specific error class (e.g., TwirpError)
from livekit import api  # or correct module for your SDK

try:
    resp = await asyncio.wait_for(
        self.lkapi.agent_dispatch.create_dispatch(
            CreateAgentDispatchRequest(... )
        ),
        timeout=10.0
    )

    <http://logger.info|logger.info>("✅ Call successfully dispatched to LiveKit for customer %s (room: %s)",
                customer_id, room_name)
    logger.debug("Dispatch response: %s", resp)
    return True

except TimeoutError:
    logger.error("❌ create_dispatch timed out after 10s for customer %s (room: %s)",
                 customer_id, room_name)
    return False

except api.TwirpError as e:
    # SDK-specific structured error: log code and metadata if available
    logger.error("❌ create_dispatch rejected by server: %s; code=%s; meta=%s",
                 getattr(e, "message", str(e)),
                 getattr(e, "code", None),
                 getattr(e, "metadata", None))
    return False

except Exception as e:
    # catch-all for network errors, serialization errors, etc.
    logger.exception("❌ Unexpected error when calling create_dispatch: %s", e)
    return False

Notes about agent-level rejection vs dispatch failure • create_dispatch success: means the request was accepted (job created/queued). It does not mean the agent processed the job successfully. • Agent runtime rejection: if the agent code rejects the job (for example,

job.reject(...)

), that rejection happens after dispatch and is not returned by

create_dispatch

. To detect that: ◦ Monitor agent logs/metrics for job rejection events. ◦ Use any job lifecycle callbacks or webhooks your deployment exposes to report job acceptance/rejection. ◦ Add tracing/metadata you can correlate between the dispatch response and later agent events. If you need help identifying the right exception class • Tell me which LiveKit Python package or client you’re using (package name and version). I’ll show the exact exception types to catch (e.g., TwirpError or a client-specific APIError) and example logging fields to inspect. Want help adding retries or backoff? • If you’d like, I can give a retry/backoff pattern for transient failures (timeouts/network errors) while avoiding retries for hard rejections (HTTP 4xx-like errors). Which behavior do you prefer? Sources: Agent dispatch | LiveKit Documentation | Agents telephony integration | LiveKit Documentation | sip | livekit/livekit_composite | DeepWiki | Why do jobs stop after agent rejection? | LiveKit

should load function of AgentServer depends on the number of pods?

Thanks — I’m responding to whether the AgentServer

load

function should depend on the number of pods. Short answer • The

load

function is evaluated per AgentServer instance (per pod). It should reflect the local pod’s current load and capacity, not the total number of pods. • Cluster-level scaling decisions should aggregate per-pod load signals (or use a separate metric) to decide how many pods to run. Why per-pod load matters • The

load

function tells the scheduler how busy a particular AgentServer pod is (e.g., how many active jobs, CPU usage, memory pressure, token throughput). • The scheduler or dispatcher uses that per-pod load to choose where to dispatch new jobs so work is balanced across available pods. • Keeping

load

scoped to the pod makes scheduling decisions local and immediate, and avoids coupling the scheduler to cluster topology. How pod count fits in • Pod count is an outcome of autoscaling or your deployment plan — it is not something a single pod’s

load

function should rely on. • An autoscaler (HPA, custom controller) should observe per-pod loads (or aggregated metrics) and increase/decrease pod count when needed. That controller uses the per-pod load metrics to scale horizontally. Practical guidance — what your

load

function should report • Report metrics that reflect the pod’s capacity to accept more jobs: ◦ Active/in-flight job count or fraction of max concurrent jobs. ◦ CPU utilization or memory pressure. ◦ Token consumption rate relative to local or global quotas (if relevant). ◦ Any custom capacity constraints (e.g., open connections, model instances). • Return a normalized value the dispatcher understands (for example: 0.0 = idle, 1.0 = full). • Keep the calculation local and conservative (avoid returning overly optimistic low load). How to use those signals for scaling • Aggregate per-pod loads (average, p95, or custom) in your metrics backend. • Configure autoscaling rules to: ◦ Scale out when average load exceeds a threshold (e.g., 0.7) or when queue length grows. ◦ Scale in only when load is low for a sustained period and in-flight jobs are near zero (to allow graceful draining). • For bursty workloads, consider combining a small prewarmed

num_idle_processes

per pod with autoscaling to add pods for larger bursts. Example patterns • Per-pod

load

= active_jobs / max_concurrent_jobs. Dispatcher avoids pods near load=1.0. • HPA uses the average of that metric across pods; if avg_load > 0.7 → add pods. • Use terminationGracePeriodSeconds >= longest expected job duration so pods can drain safely when scaling in. When (and how) pod count can influence local decisions • You might expose a cluster-level hint (e.g., global token quota left) so each pod can be more conservative when global capacity is low. But this is an external signal — it shouldn’t be baked into the per-pod

load

logic. • Example: if global LLM tokens remaining are low, pods can raise their reported load artificially so dispatcher routes fewer jobs, avoiding a global quota breach. Bottom line • Keep

load

per-pod and focused on local capacity. Use aggregated metrics and autoscaling logic to decide how many pods are needed overall. • If you want, I can suggest a specific

load

function formula (normalized value) based on your metrics (in-flight jobs, CPU, memory, tokens/sec). Share which signals you have available and I’ll recommend one. Which signals can you expose?