I wanted to setup self hosting of live kit . What do you think with plugin hybrid system or with self hosting the latency is good ? I am planning 10 to 20 calls at a minute.

Thanks for reaching out, Waleed. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

Yes I need strict latency because we do sales call

Thanks — I hear you need strict, low latency for sales calls. I can help dial this in. Quick follow-ups so I can give concrete recommendations (instance sizes, autoscaling, and deployment patterns): 1. Where will STT / LLM / TTS run? ◦ Third‑party providers (OpenAI, Deepgram, etc.) or self‑hosted in your VPC/region? 1. Peak concurrency vs steady rate: ◦ You said 10–20 calls per minute — is peak concurrent sessions ~10–20, or could bursts exceed that (e.g., 40–50 concurrent)? 1. Target SLA: ◦ What maximum end‑to‑end latency do you require (e.g., <300 ms, <500 ms, <1 s)? 1. Region / geography: ◦ Where are most callers located (one region or global)? 1. Infrastructure constraints: ◦ Any restrictions on cloud provider, instance types, or budget for reserved instances? Answer these and I’ll recommend a specific self‑host vs hybrid approach, instance types, autoscaling rules, co‑location strategy, and concrete tuning steps to meet your SLA. , with interactive elements

Target SLA is undr 500 and in total ww have budget around 2000$ for a month. And mostly we use inworld and cartesia ,

Thanks — you need <500 ms end‑to‑end with Inworld and Cartesia and ~$2,000/month. Below are concise, practical recommendations and next steps to meet that SLA. Recommended approach • Hybrid (self‑hosted agents + cloud provider/model co‑location) — gives best control over latency while keeping cost predictable. Host LiveKit agents in regions closest to Inworld/Cartesia endpoints you use. Key configuration to hit <500 ms • Co‑locate agents and model/STT/TTS endpoints in same cloud region to minimize network RTT. • Use fully streaming mode for STT → LLM → TTS (no batching). • Keep agents warm (avoid cold starts) so no multi‑second startup delays. • Monitor per‑turn metrics (e2e_latency, llm ttft, tts ttfb) and set alerts at 250 ms to catch regressions early. • Prefer smaller/faster model variants for quicker TTFT when acceptable. Instance sizing & autoscaling (starting guidance) • Use steady performance VMs (no burstable types). Example starting pool: ◦ 2 x m5.large (or equivalent) as warm workers handling baseline sessions ◦ Autoscale up to 6–8 workers on CPU/memory or connection count triggers • Autoscale triggers: ◦ CPU >70% for 1–2 min OR ◦ Active sessions per worker > recommended session capacity (benchmark early) • Benchmark to find sessions/worker; assume conservative 4–8 concurrent voice agent sessions per worker initially. Multi‑region & traffic routing • Deploy agents in the primary caller region(s). If callers are global, deploy in 2–3 regions and route users to nearest region. • Ensure network egress/ingress routes to Inworld/Cartesia are low latency (same cloud provider or peered regions if possible). Cost guidance (rough) • $2,000/month can cover a modest multi‑region hybrid setup if you: ◦ Use 2–3 steady workers per region (reserve instances where helpful) ◦ Minimize cross‑region egress and choose region pricing carefully ◦ Optimize model usage (smaller/faster models, token limits) • Exact cost depends on chosen cloud sizes, region pricing, and model provider billing — run a short load test to estimate real costs. Immediate next steps I can help with • Recommend specific VM families/sizes for your preferred cloud and regions. • Draft autoscaling rules and monitoring dashboards (metrics to track). • Create a short benchmarking plan to validate sessions/worker and real latency. Which cloud provider and regions do you plan to use for hosting agents? I’ll give concrete VM sizes, autoscaling thresholds, and an estimated cost split to fit your $2,000/mo budget. , with interactive elements