for scalability, agents and SFU should be maintained seperately, right?

Thanks for reaching out, Kuniaki. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

OK. some questions: 2 SFU and 1 agent. how it works?

Thanks — I’ll explain how two SFUs with one agent typically operate, and what to expect for media flow, redundancy, and scaling. How it works (high level) • Each SFU handles WebRTC connections for a subset of clients. Clients (browsers/mobile) connect to an SFU and publish/subscribe media there. • The single agent joins the same room as a normal client and connects to one of the SFUs (whichever the agent’s signalling/selection routes it to). • The SFUs forward media between each other as needed so the agent can receive streams published on any SFU and other participants can receive the agent’s output. Media routing & connectivity • When a publisher on SFU A publishes a track, SFU A forwards that track to other SFUs (including SFU B) via the SFU mesh/peering so subscribers on SFU B and the agent (wherever it’s connected) can receive it. • The agent receives remote tracks from the SFU it’s connected to and can: ◦ Process them (STT/LLM/etc.) locally, and ◦ Publish processed audio/voice or synthesized media back into the room via its SFU connection. • The SFU the agent is connected to will forward the agent’s published tracks to subscribers on other SFUs. Resilience and failover • If the agent is connected to SFU A and SFU A fails, the agent must reconnect (or be restarted) and will typically connect to SFU B to continue processing. • If an SFU fails, clients connected to that SFU lose their direct connection and need to reconnect — their new connection may land on the surviving SFU; media flows resume once peers reconnect. • For higher availability of the agent, run multiple agent instances (or use an autoscaling pool) so one agent can failover without losing processing capacity. Performance and scaling considerations • Two SFUs split media bandwidth and CPU load for forwarding; the single agent’s CPU/GPU is the bottleneck for any compute-heavy tasks (STT, LLM inference). • Monitor agent CPU, latency (RTT), and media throughput. If agent processing becomes a bottleneck, add agent instances and distribute work (multiple agents can join the same room or separate rooms depending on your design). • Ensure SFU interconnects have sufficient bandwidth and low latency so cross-SFU forwarding doesn’t introduce noticeable delay. Questions to tailor guidance • Is the agent doing real-time heavy processing (on-prem inference) or primarily cloud API calls? • Do you want automatic agent failover or are you okay with manual restart/scale-up? • Are you using LiveKit Cloud or self-hosted SFUs? Tell me which of those apply and I’ll give specific deployment recommendations and configuration tips. , with interactive elements

ok. so I wrap up: • an agent just connects to one of SFUs. right? • when client A connects to SFU A, an agent job handles this. • when client B connects to SFU B, another agent job handles this. is this correct?

You're almost correct — small clarifications: • Yes, an agent process connects to one SFU at a time like a normal client. • An agent connected to SFU A can receive media from clients on SFU A and from clients on SFU B because SFUs forward tracks across the SFU mesh/peering. • You do not strictly need a separate agent job per client/SFU. You can run one agent that handles many clients across SFUs (if its CPU/latency allow), or run multiple agent instances and route work (or have each instance join the room) for scale and redundancy. If you want, tell me whether your agents do heavy on‑device inference or call cloud APIs and how many concurrent streams you expect — I’ll recommend agent counts and placement. , with interactive elements

in my assume, one client per one agent in a room, no multiple clients in the same room. also i'm talking about self-hosting. I just wanna make sure that each agent does not need to know each SFU to connect like: • SFU A <- Agent A • SFU B <- Agent B

You're correct: in a self‑hosted deployment each agent only needs to connect to a single SFU instance — it does not need to be aware of every SFU in the cluster. Key points that match your setup (one client per agent, self‑hosted): • An agent connects to one SFU as a normal room participant. • SFUs in the mesh/peering forward tracks between each other, so an agent on SFU A can receive media from clients on SFU B without connecting to SFU B directly. • You can run a one‑client‑per‑agent model (one agent instance per client in the room). That’s valid but not required — you can also have an agent handle multiple clients if desired. Operational notes for self‑hosting: • Let your signalling/load‑balancer route agents to SFUs; agents don’t need manual configuration of every SFU. • For availability, run multiple agent instances (or an autoscaling pool) so a single SFU or agent failure doesn’t drop processing. • Monitor agent CPU/latency since each agent doing heavy inference will be the bottleneck. If you want, tell me: • whether agents run heavy on‑device inference or call cloud APIs, and • expected concurrency per region — and I’ll recommend agent counts and simple placement/HA patterns for your self‑hosted cluster. , with interactive elements

so in that case, SFU must be clustered. this is the least requirements, right?

We have knowledge: SFU mesh, load balancing, agents connect to one SFU, run multiple agents for HA. Also mention Redis/etcd? In knowledge base earlier mentions global mesh and distributed; not explicit about Redis/etcd. But earlier function query showed citations referencing mesh/peering and self-hosted support. Avoid stating specifics not given. Provide concise confirmation: SFU clustering is minimal for cross-SFU forwarding; need load balancer/signalling to route clients/agents; agents don't need to know all SFUs; recommend HA by multiple SFUs and agent instances; monitoring and autoscaling. Ask if they want deployment recommendations. Keep warm tone.Short answer: yes — for self‑hosted you should run a clustered/peered SFU mesh as the minimal setup to support agents and cross‑SFU media forwarding. What that gives you • Cross‑SFU forwarding so an agent on one SFU can receive/publish media for clients on other SFUs. • The ability to scale SFU capacity independently from agent compute. • Better availability: if one SFU fails, clients can reconnect to others. What you still need (minimum) • A load‑balancer/signalling layer to route clients and agents to SFU instances. • Multiple SFU instances peered together (SFU mesh) so tracks are forwarded across the cluster. • Multiple agent instances or an autoscaling pool for HA if you need agent redundancy. Operational notes • Agents only connect to a single SFU; they don’t need to know every SFU. • Monitor SFU interconnect bandwidth and agent CPU/latency; scale SFUs for media bandwidth and agents for compute. Would you like a simple HA/placement pattern (example counts for SFUs vs agents) based on expected concurrency? , with interactive elements

OK. for example in AWS: • SFUs are clustered with ElastiCache and loadbalanced/autoscaled with ALB. • an agent just connects to one of SFU (by internal DNS. etc). and also with autoscaled should be prefered. does this make sense?

Yes — that setup makes sense. Why it works • Clustering SFUs (peered mesh) provides cross‑SFU forwarding and scales media bandwidth. • An ALB + autoscaling routes clients and agents to healthy SFU instances and lets you scale capacity. • Using ElastiCache for shared coordination/state is a common pattern to support discovery/locking/session data in self‑hosted clusters. • Agents connect via internal DNS to one SFU instance and should be autoscaled for compute/HA. Recommendations / quick checks • Ensure SFU interconnect bandwidth and latency are sufficient for cross‑SFU forwarding. • Use health checks on the ALB so clients/agents are routed only to healthy SFUs. • Run multiple agent instances (or an autoscaling group) so agent failures don’t lose processing capacity. • Monitor agent CPU/GPU, SFU forwarding load, and network egress to determine scaling triggers. Would you like an example autoscaling policy and instance count guidance based on expected concurrent streams? , with interactive elements

also, you said

Let your signalling/load‑balancer route agents to SFUs; agents don’t need manual configuration of every SFU.

right? much better than DNS name resolution.

Yes — routing agents to SFUs via your signalling/load‑balancer is better than hard‑coding DNS to specific SFUs. Why it's preferred • Simplicity: agents don’t need knowledge of every SFU instance — they just connect to the service endpoint. • Resilience: the load‑balancer can route only to healthy SFUs and steer traffic during scale or failover. • Flexibility: you can scale/replace SFUs without updating agent configs or DNS records on each agent. Implementation notes for AWS • Use an internal ALB (or NLB if you need UDP) and target group with health checks to route agents to healthy SFUs. • Keep SFUs peered/meshed so tracks forward across instances. • Let agents use the ALB/internal DNS name as their connect endpoint; autoscale agents independently. • Use service discovery or short‑TTL internal DNS for other control-plane needs (if any), but avoid baking per‑SFU addresses into agents. If you want, I can suggest ALB vs NLB tradeoffs, health‑check settings, and a simple autoscaling policy for SFUs and agents. Which would you prefer? , with interactive elements

OK, so could you wrap up this conversation? especially assuming self-hosting on AWS.

Thanks — here’s a concise wrap‑up for a self‑hosted LiveKit deployment on AWS, covering the recommended architecture and operational points we discussed. Summary (what to run) • SFUs: Run multiple peered SFU instances (SFU mesh) across your cluster for cross‑SFU forwarding, capacity, and availability. • Load balancer: Use an internal ALB/NLB to route clients and agents to SFU instances with health checks. • Coordination store: Use ElastiCache (or similar) for shared coordination/session needs. • Agents: Run agents as normal room participants; each agent connects to a single SFU via the LB/internal DNS and should be autoscaled for compute/HA. • Autoscale both SFUs and agents independently based on media bandwidth and agent CPU/latency. Why this works • SFU mesh forwards tracks across instances so an agent on one SFU can receive/publish media for clients on any SFU. • Load balancer-based routing avoids hard‑coding SFU addresses and ensures agents connect only to healthy SFUs. • Separating SFU (media forwarding) and agent (compute) lets you scale each to its bottleneck. Minimum requirements & checks • Multiple peered SFUs and an internal LB with health checks. • Autoscaling groups for SFUs and agents. • Sufficient inter‑SFU network bandwidth and low latency. • Monitoring: SFU forwarding load, agent CPU/GPU, RTT/latency, and network egress. • Agent redundancy (multiple instances or autoscaling) if you need high availability for processing. Operational recommendations • Prefer LB routing over hard‑coded per‑SFU DNS. • Use ALB for TCP/HTTP signaling; use NLB if you require UDP passthrough. • Configure LB health checks to take SFU readiness into account. • Start with conservative instance counts, monitor metrics, and tune autoscaling policies for media throughput (SFUs) vs. compute (agents). If you want, I can provide: • Example autoscaling policies and initial instance counts based on expected concurrent streams, or • ALB vs NLB tradeoffs and recommended health‑check settings. Which follow‑up would you like? , with interactive elements

could you add more about how agents connects to SFU?

Thanks — I’ll expand on how agents connect to an SFU in a self‑hosted AWS setup and include practical best practices. Connection flow (high level) • Agents act like normal LiveKit clients: they perform signalling to get a room token and then establish a WebRTC (or RTC) connection to an SFU instance. • Agents should connect to the cluster via your internal load balancer (ALB/NLB) or a service endpoint rather than targeting a specific SFU instance. • Once connected, the SFU the agent is attached to handles publishing/subscribing for that agent. The SFU mesh/peering forwards tracks across SFUs so the agent can access media from clients on other SFUs. Practical steps for agent connection 1. Obtain auth credentials / join token for the room (normal LiveKit flow). 2. Resolve the SFU endpoint as a single service address (internal ALB DNS or internal service discovery name). 3. Establish the WebRTC/RTC connection to that endpoint using the LiveKit SDK or WebRTC client. 4. Publish/subscribe tracks as needed; process media locally and publish results back to the room. AWS implementation notes / best practices • Use an internal ALB for TCP/HTTP signalling; use an NLB if you need UDP passthrough or lower latency for non‑HTTP transports. • Point agents at the ALB’s internal DNS name so the LB routes them to healthy SFU instances. • Configure LB health checks that validate SFU readiness (signalling + media forwarding ready). • Keep agent autoscaling independent from SFU autoscaling so you can scale compute without touching media forwarding capacity. • Use short‑TTL internal DNS or service discovery only for control-plane tasks; avoid hard‑coding per‑SFU addresses in agents. • Ensure SFU interconnect links (VPC network bandwidth/peering) have enough capacity and low latency for cross‑SFU forwarding. Authentication & session handling • Issue per‑agent room tokens (short TTL) so each agent authenticates as a distinct participant. • If you use sticky sessions at the LB, prefer it only for debugging — normal operation should tolerate reconnections and reassignments. • Handle reconnects in agents: if their SFU instance becomes unhealthy, the LB will route a reconnect to another SFU; the agent should rejoin and resume publishing/subscribing. Reliability & HA patterns • Run multiple agent instances (or an autoscaling group) for processing redundancy; each agent connects via the LB to some SFU. • For critical workloads, run a warm standby agent pool so new agents can join quickly if one fails. • Monitor agent CPU, memory, and media latency to trigger scale‑out. Short checklist to implement now • Internal ALB/NLB with health checks -> SFU autoscaling group. • Agents configured to use ALB internal DNS -> agent autoscaling group. • Token service to mint per‑agent room tokens. • Monitoring/alerts on SFU mesh bandwidth and agent CPU/latency. If you want, I can provide: • Example connection code snippet for an agent using the LiveKit SDK, or • Recommended health‑check settings and autoscaling thresholds for ALB/NLB and agent groups. Which would be more helpful? , with interactive elements

OK. Thanks. ありがとう。

はい、日本語で対応できます。ご希望であれば、これまでのまとめや追加の設計・運用アドバイス（接続コード例、ALB/NLBのヘルスチェック設定、オートスケーリング閾値など）を日本語で詳しくご説明します。どれを日本語で見たいですか？ , with interactive elements

that's it for now. thanks.