what is the uses of manual dispatch agent ? and when to not use it ?

Thanks for reaching out, simo.dev. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

In a WebRTC call where each participant can independently trigger video processing, what is the recommended approach in LiveKit: • Dispatch a dedicated agent instance per participant when they enable processing of their video track, • or run a single shared agent and dynamically toggle processing per participant (e.g., via flags/RPC)? Goal: ensure per-participant control (opt-in), minimal latency, and efficient resource usage while handling video frame processing on demand.

Thanks — I’m replying about which approach to use for per-participant, opt-in video processing in LiveKit. Short recommendation • Use a dedicated agent per participant (dispatch as a PUBLISHER job) when a participant opts in. ◦ Gives true per-participant isolation, simple lifecycle (agent runs only when needed), lower contention and predictable latency, and easy scaling horizontally. • Use a single shared agent with dynamic toggles only for very lightweight, low-concurrency workloads where you can tolerate added complexity and potential latency/throughput bottlenecks. Why the per-participant (PUBLISHER) approach is preferred • Opt-in control: each agent is only created for participants who enable processing, so you avoid wasting CPU/GPU on everyone. • Latency: isolated agents can be colocated with inference services and avoid queuing from other users’ work. • Scalability: you scale by adding more agent workers rather than making one process handle many independent streams. • Simpler state: no cross-user flags or complex routing inside a single agent. When to consider a shared agent • Processing is extremely lightweight (e.g., occasional metadata tagging or sampling). • Very small scale / prototype where operational simplicity matters more than performance. • You cannot spin up per-publisher agents for operational reasons (but this adds state/coordination complexity). If you’d like, tell me: • Typical concurrency (how many participants may enable processing simultaneously), and • Whether processing is heavy (GPU inference) or light (CPU sampling). I’ll suggest a concrete dispatch pattern and resource-sizing guidance based on those details. Sources: Server options | LiveKit Documentation | Agent server overview | LiveKit Documentation | Video | LiveKit Documentation | An open source stack for real-time multimodal AI | LiveKit | Bringing Zoom's end-to-end optimizations to WebRTC | LiveKit, with interactive elements

we can have 10-30 participant

Thanks — with 10–30 participants potentially opting in, here’s a concise recommendation. Recommendation • Use a dedicated agent per participant (dispatch as a PUBLISHER job). ◦ This gives per-participant opt-in control, predictable low latency, and horizontal scalability for 10–30 concurrent processors. Why this fits 10–30 concurrents • Isolation prevents one participant’s load from impacting others. • You can autoscale agent workers to match concurrency (add workers when more participants opt in). • Latency stays low because each agent handles a single stream and can be colocated with inference services. Resource and operational considerations • If processing is heavy (GPU models, high-res or high-FPS), provision 1 GPU per N agents (N depends on model throughput) or run agents on GPU nodes with autoscaling. • If processing is light (frame sampling, lightweight CPU inference), smaller CPU instances can host multiple agents; still prefer separate agents for simpler logic and isolation. • Implement quick teardown: stop/terminate agent when participant disables processing to free resources. Practical patterns to implement • Dispatch agents on-demand via JobRequest or access-token metadata when a participant enables processing. • Keep per-agent lifecycle short and stateless where possible; persist needed state externally. • Monitor per-agent CPU/GPU, latency, and memory to set autoscaling thresholds. Quick follow-up question • Is your processing heavy (GPU inference, large models) or light (CPU sampling/simple models)? I’ll give concrete sizing and autoscaling guidance based on that., with interactive elements

can i run only one agent x but when triggering rpc we need to trigger another agent y and pass to it the ctx from the agent x ?

Yes — you can run Agent X and, on RPC/tool call, start Agent Y and pass X’s context to it. What’s supported • Agents can perform a handoff: Agent X can programmatically start Agent Y (or return a new agent instance) during an RPC / function_tool call. • You can pass conversational/operational context via

chat_ctx

and/or

userdata

on the new agent session so Agent Y continues with Agent X’s state. • Use

chat_ctx.copy(exclude_instructions=True)

to preserve prior turns without re-sending system instructions. Typical pattern 1. Agent X receives an RPC or tool invocation to delegate. 2. Agent X calls the handoff API / function_tool to instantiate Agent Y, supplying: ◦

chat_ctx=self.chat_ctx.copy(...)

for conversation history ◦

userdata

for any custom state 1. Agent Y starts and resumes processing with the provided context. When to use this • Role specialization (e.g., lightweight controller agent delegates heavy inference to a worker agent). • Stateful transitions where preserving conversation or session data matters. When not to • If you need ultra-low latency per-frame processing: spinning up a new agent may add start-up cost. For strict real-time needs, prefer pre-warmed agents or per-publisher agents dispatched up-front. If you want, tell me whether you need a short code example in Python or Node.js for invoking the handoff and passing `chat_ctx`/`userdata`. Sources: Agents and handoffs | LiveKit Documentation | Chat context | LiveKit Documentation | Building Multi-Agent Architectures with LiveKit Agents, with interactive elements