https://livekit.io logo
Join Slack
Powered by
Hi everyone, We recently migrated from LiveKit Clo...
# ask-ai
f
Hi everyone, We recently migrated from LiveKit Cloud to self-hosted LiveKit on Azure and are trying to understand expected Egress scalability and CPU usage patterns. Current setup: • Azure VM: D8as_v5 (8 vCPU, 32 GB RAM) • Ubuntu • Self-hosted LiveKit + SIP + Redis + Egress + Caddy + CoTURN • Using room composite Egress for MP4 call recordings • Azure Blob upload for recordings • Docker Compose deployment • Currently running 3–5 Egress replicas Issue: Calls work perfectly: • SIP calling works • audio/media works • agents talk normally • rooms are created correctly But under concurrent load, some recordings fail to upload or never get created even though calls themselves complete successfully. Earlier logs showed: • "can not accept request" • "Start signal not received" • "no response from servers" • occasional egress_aborted After scaling Egress replicas and increasing shm_size, things improved significantly. However, one thing we are confused about: Even a single active room composite recording causes around 35–40% CPU usage on one Egress container (docker stats). Questions: 1. Is this expected CPU usage for Chromium-based room composite Egress? 2. Roughly how many concurrent room composite recordings should an 8 vCPU / 32 GB VM realistically handle? 3. Is scaling Egress horizontally the recommended approach, or should we separate Egress onto dedicated VMs? 4. Are there recommended production architectures for higher concurrency self-hosted recording setups? 5. In LiveKit Cloud, ~$50 usage handled significantly higher recording concurrency for us. Is Cloud using internal autoscaling/distributed Egress workers behind the scenes? 6. Any recommended tuning for: Chromium shm_size CPU pinning Redis Kubernetes autoscaling GPU acceleration file upload throughput Would really appreciate guidance from teams running self-hosted LiveKit Egress at scale. Thanks!
✅ 1
t
Thanks for reaching out, Anjani. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — this is very clear and helpful context. I’ll answer each of your numbered questions and finish with concise tuning and architecture recommendations you can act on. Acknowledgement: you’re asking about expected CPU for Chromium-based room-composite Egress, how many recordings an 8 vCPU/32 GB VM can handle, whether to scale horizontally or isolate Egress, production architecture patterns, LiveKit Cloud behavior, and tuning knobs (Chromium, shm_size, CPU pinning, Redis, k8s autoscaling, GPU, upload throughput). I’ll address each below. 1) Is ~35–40% CPU per single room composite expected? • Yes. Room-composite Egress runs a headless Chromium instance plus real-time encoding (GStreamer/FFmpeg), which is CPU-intensive. Seeing ~30–45% CPU on an 8‑vCPU VM for one composite recording is within expected behavior. 2) How many concurrent room-composite recordings on 8 vCPU / 32 GB? • Rough estimate: ~6–10 concurrent room-composite recordings per 8 vCPU / 32 GB VM when properly tuned and Egress is isolated. Actual capacity depends on viewport/resolution, encoding settings, other co-located services, and upload network. Test under your exact workload to find your sustainable number. 3) Scale horizontally or use dedicated VMs? • Horizontal scaling (more Egress replicas across nodes) is the recommended approach. Also run Egress on dedicated nodes (separate from SIP/signaling/SFU) to avoid resource contention. Prefer many Egress-only workers across multiple VMs rather than packing everything onto few large machines. 4) Recommended production architectures for higher concurrency • Run dedicated Egress workers (Egress-only service) separate from LiveKit SFU/SIP. • Use Redis-based request distribution / queueing so available workers pick up Egress jobs. • Deploy multiple smaller nodes (Egress pods/VMs) with autoscaling rather than one big host. • Ensure durable local storage or fast temporary disk for recordings, then multipart upload to blob storage. • Monitor per-worker CPU, memory, queue depth, and upload latencies; use those metrics to scale. 5) LiveKit Cloud handling higher concurrency for ~$50 • LiveKit Cloud uses autoscaling and distributed Egress workers and pools capacity across infrastructure. That distributed autoscaling and worker pooling is why Cloud can achieve higher concurrency for the same spend compared to a single self-hosted VM. 6) Recommended tuning (practical items to try) • shm_size: ◦ Increase 
--shm-size
for containers (512MB–1GB or more per Chrome instance) to avoid Chrome shared-memory issues. • Chromium flags: ◦ Run Chrome headless with flags like 
--disable-dev-shm-usage --disable-gpu --no-sandbox --disable-software-rasterizer
and set a fixed viewport to reduce overhead. • CPU pinning / cgroups: ◦ Pin or limit CPU for Egress containers/pods so each Chrome process has stable cores; use CPU requests/limits in k8s to avoid noisy neighbor contention. • Redis: ◦ Use a low-latency Redis (same region), monitor Pub/Sub queue depth, and ensure HA (replication/failover) so Egress job dispatch is reliable. • Kubernetes autoscaling: ◦ Autoscale based on CPU (e.g., trigger scale up >70% CPU for sustained period) and queue-depth metrics. Set a generous 
terminationGracePeriodSeconds
so active recordings can finish. ◦ Use node taints/tolerations or separate node pools to dedicate nodes for Egress workloads. • GPU acceleration: ◦ GPU can reduce CPU by offloading encoding (e.g., NVENC), but it’s advanced: requires GPU-enabled nodes, matching GStreamer/encoder pipeline, and additional ops complexity. Not officially turnkey. • File upload throughput: ◦ Use the cloud provider’s SDKs and multipart uploads (Azure Blob multipart) for resilience and throughput. ◦ Ensure sufficient network bandwidth on Egress nodes and monitor upload latencies; avoid uploading directly from overloaded nodes if possible (or stream to a durable buffer). • shm_size, memory, container limits: ◦ Ensure each Egress replica has adequate memory and shm; set container memory limits high enough to avoid OOM kills. Quick operational checklist to reduce failures you observed • Isolate Egress to dedicated replicas/nodes (don’t co-locate heavy services). • Increase 
shm_size
to >= 512MB–1GB for each Egress container. • Add more Egress replicas and ensure Redis is healthy and low-latency. • Add CPU-based autoscaling and monitor queue depth; scale before CPU saturates. • Use multipart uploads and ensure upload bandwidth is sufficient. • Capture and monitor these metrics per worker: CPU, memory, shm usage, container restarts, Redis queue depth, and upload error rates — use them to tune replica count and VM sizing. If you want, I can: • Suggest concrete container CPU/memory requests & limits and 
shm_size
values to test for your environment. • Propose a simple autoscaling policy (CPU + Redis queue depth) for k8s. Which would you prefer me to provide next? Sources: Universal Egress | LiveKit | Universal Egress | LiveKit | LiveKit Inference | Fast, reliable access to voice AI models | LiveKit, with interactive elements
f
one more thing i create sip trunks in sef hosted version when my VM restarts or soemthng all of my sip trunks are gettingdeleted? how to make it be rthere/
t
Thanks — I’m sorry you’re losing SIP trunks after VM restarts. I can help get this fixed. A few quick questions so I give the right steps: • How are you creating/configuring the SIP trunks today? (LiveKit API call, UI, IaC/script, or manual in some config file) • How is LiveKit deployed: Docker Compose on that VM, or Kubernetes/Helm? • Do you have an external DB (CockroachDB or other) configured, or is LiveKit using local/ephemeral storage? • Are you restoring from any startup script that could be resetting state? Short answer: SIP trunks are stored by the LiveKit server in its persistent backend. If trunks disappear after restarts it means the server state isn’t being persisted (ephemeral filesystem or no external DB). The fix is to ensure LiveKit’s state is stored on persistent storage or an external database and to persist any container data across restarts. Recommended next steps (pick what matches your setup): • Docker Compose on single VM: ◦ Mount a host volume for LiveKit’s data directory so database/files aren’t ephemeral. ◦ Ensure compose defines volumes for the service and any DB container. • Kubernetes: ◦ Use PVCs/StatefulSet for the LiveKit server and any DB so state survives pod/node restarts. • Use an external DB: ◦ Configure LiveKit to use an external persistent DB (e.g., CockroachDB) so trunk configs persist across instances. • Backup or script: ◦ Export or script trunk creation (API calls) so you can reapply automatically if needed. • Verify on restart: ◦ After making persistence changes, restart the VM and confirm trunks remain. Tell me which deployment method you’re using and how trunks are currently created and I’ll give concrete commands / compose/helm snippets and the exact paths or DB settings to change. Sources: Security Overview | LiveKit | Self-hosted deployments | LiveKit Documentation, with interactive elements
f
I have only an ai agent and a human participant with screen sharing, camera enabled in the room, how much concurrency I can achieve on a 4vcpu, 8gb ram egress vm? Or should I use browser side javascript recording in react?
t
Thanks — I’ll address exactly what you asked: expected concurrency on a 4 vCPU / 8 GB Egress VM for a room with an AI agent + human (camera + screen share), and whether to use server-side Egress or browser-side recording. Short answer • A single Room Composite (Chromium-based) recording with screen-share + camera is CPU‑heavy and a 4 vCPU / 8 GB VM may only reliably handle 1 room-composite at a time. It could handle more only if you switch to lighter egress types or lower resolution/bitrate. • If you need higher concurrency on that VM, prefer Track Egress (export raw tracks) or Track Composite rather than Room Composite. • Browser-side (client) recording can offload server CPU but is variable in reliability and quality; use it only if you accept client-device variability and you don’t need centralized, guaranteed recordings. What to consider (concise) • Room Composite uses headless Chromium + real‑time encoding — expect high CPU per recording (Chromium can consume ~30–45% of an 8‑vCPU host as a reference; on 4 vCPU expect proportionally larger impact). • Screen sharing increases encoding cost (larger frame, higher bitrate). • Track Egress (raw track export) is far lighter than Room Composite and scales much better on small VMs. • Client-side recording: ◦ Pros: offloads server CPU, lower infra cost. ◦ Cons: dependent on client device CPU, browser limitations, upload reliability, and no centralized control or guaranteed quality. Practical recommendations • If you need reliable, centralized high-quality recordings: use server-side Egress but ◦ On 4 vCPU/8 GB, avoid Room Composite for concurrency — reserve that VM for 1 composite or use it for Track Egress only. ◦ Test with your exact video resolution/bitrate to find real limits. • If you can accept some variability and want to reduce server cost: consider client-side recording for the human participant (MediaRecorder in the browser) while using server-side for the AI agent or as a fallback. • Hybrid approach: client-side recording + server-side Track Egress for redundancy or post-join stitching. • If you need multiple concurrent Room Composites, scale horizontally (more Egress workers) or use larger/dedicated Egress nodes. If you want, I can: • Suggest which Egress type to try first for your workload (Track vs Track Composite vs Room Composite) and example settings (resolution/bitrate) to test. • Propose a simple load-test plan to measure concurrency on your 4 vCPU VM. Which would you like next? Sources: Self-hosted deployments | LiveKit Documentation | Universal Egress | LiveKit | Universal Egress | LiveKit, with interactive elements
f
How can I achieve the same type of recording with TrackComposite recording?
t
Acknowledgement: you asked how to get the same screen+camera recording you get from Room Composite using Track Composite. Short answer: • Track Composite cannot natively composit multiple video sources (camera + screen) or multiple participants into one layout. It records a single participant’s synchronized audio+video track. • To achieve the same result you have two practical options: Options • Client-side compositing + Track Composite: a. In the publisher (browser), merge camera + screen into one outgoing video track (e.g., draw camera over screen to a canvas and use 
canvas.captureStream()
). b. Publish that single composited track. c. Use Track Composite to record that merged track (gives you one MP4 without server-side Chromium). ◦ Pros: much lower server CPU (no headless Chrome). Cons: depends on publisher device CPU and network. • Multiple Track Composite recordings (workaround): ◦ Start separate Track Composite egresses: one for the camera track + audio and another for the screen track + audio. ◦ Post-process (server-side) to combine them into a single file (e.g., ffmpeg overlay) if you need one file. ◦ Pros: keeps recording server-side; Cons: extra storage/processing and complexity. • Stick with Room Composite: ◦ If you need server-controlled layout, dynamic UI overlays, or multi-participant layouts, Room Composite is the direct solution (uses headless Chromium). Recommendation: • If you want to reduce Egress CPU and can control the publisher: use client-side compositing + Track Composite. • If you cannot rely on publisher devices or need server-side layout/UI, continue using Room Composite and scale Egress accordingly. If you want, I can: • Provide a small example (React + canvas) to publish a composited track. • Or show the Track Composite request format to record a specific track and a simple ffmpeg command to merge two recordings. Which would you like? Sources: Universal Egress | LiveKit | Streaming video from a canvas with WebRTC and React | LiveKit, with interactive elements
f
Give me a load test plan and small example of react + canvas to publish composited track
t
Acknowledgement: I’ll give a concise load‑test plan you can run against your 4 vCPU / 8 GB Egress VM and a minimal React + canvas example that composites camera over screen and publishes that merged track (suitable for Track Composite recording). Load test plan (goal: measure how many concurrent composited recordings your 4vCPU/8GB Egress VM can sustain) 1) Test goals • Determine max concurrent Room Composite-equivalent recordings when clients publish a single composited track (client-side) and server does Track Composite. • Measure CPU, memory, shm usage, upload throughput, recording success rate, and end-to-end latency/errors. 2) Workload definition • Scenario: 2 participants per room (AI agent simulated by a headless publisher or prerecorded track, and one human client). Each human client publishes: ◦ A composited video track (screen + small camera overlay) at target resolution (e.g., 1280x720) and bitrate (2–4 Mbps). ◦ Mono audio at 48 kHz, 64 kbps. • Egress type: Track Composite recording the published composited track to MP4 and uploading to Azure Blob. 3) Test tooling • Use automated headless publishers for scaled clients. Options: ◦ Puppeteer / Playwright scripts that capture a prerecorded video and a synthetic “screen” canvas and publish via LiveKit JS. ◦ headless Chrome browsers launched with increased shm_size and proper flags. • Use a load orchestrator (k6, locust, or a simple node script) to start N simultaneous publishers and trigger Track Composite egress jobs via the LiveKit server API. • Collect metrics with Prometheus + Grafana or simple cloud VM monitoring; capture container 
docker stats
or k8s metrics. 4) Steps 1. Baseline: ◦ Start 1 client publisher + 1 Track Composite recording. Verify successful MP4 creation and upload. ◦ Record baseline CPU/memory/shm/upload times. 1. Ramp: ◦ Increase concurrent publishers in steps: 1, 2, 4, 6, 8, 10… until failure rate rises or CPU ~80–90%. ◦ At each step, run for 5–10 minutes to measure steady state. 1. Observe: ◦ Metrics: CPU, memory, shm, container restarts, Redis queue depth, egress_aborted/errors, upload failure rate, time-to-upload. 1. Failure validation: ◦ When failures appear, capture logs from Egress containers (start signal, can not accept request, egress_aborted). 1. Repeat with variations: ◦ Lower resolution/bitrate; switch from Room Composite to Track Composite; try multipart uploads; try different shm_size. 5) Success criteria • Recording success rate >= 99% over 10 minutes at target concurrency. • CPU per node < 80% sustained (headroom for peaks). • Upload latency acceptable (e.g., <30s for MP4 uploads) and no container OOMs/restarts. 6) Actions based on results • If CPU saturates: switch to more nodes (horizontal scale), reduce resolution/bitrate, or use client-side compositing (which you are testing). • If uploads fail: increase network bandwidth, use multipart uploads, or offload uploads to a separate uploader service. Small React + canvas example (minimal, focused on compositing camera over screen and publishing one merged track) Notes: • This example assumes you already have LiveKit JS installed and a token/URL from your server. • It uses 
canvas.captureStream()
to create a single outgoing video track combining screen and camera. 1) Key idea (pseudocode) • Get camera MediaStream (getUserMedia) • Get screen MediaStream (getDisplayMedia) • Draw screen frames to a canvas, draw camera as a small overlay, update at 30 FPS • Use 
canvas.captureStream(30)
to get a composited stream • Add composited video track + original microphone audio track to LiveKit room and publish 2) Minimal React component (JSX-like, abbreviated)
Copy code
import React, { useRef, useEffect } from "react";
import { connect } from "livekit-client"; // placeholder import name

export default function CompositedPublisher({ url, token }) {
  const videoRef = useRef(null);
  const canvasRef = useRef(null);
  const rafRef = useRef(null);

  useEffect(() => {
    let pc;
    let cameraStream, screenStream, micTrack, compositedStream;

    async function start() {
      // 1. connect to LiveKit
      const room = await connect(url, token);

      // 2. get camera and mic
      cameraStream = await navigator.mediaDevices.getUserMedia({ video: true, audio: true });
      const cameraTrack = cameraStream.getVideoTracks()[0];
      micTrack = cameraStream.getAudioTracks()[0];

      // 3. get screen
      screenStream = await navigator.mediaDevices.getDisplayMedia({ video: true });
      const screenTrack = screenStream.getVideoTracks()[0];

      // 4. setup canvas
      const canvas = document.createElement("canvas");
      canvas.width = 1280; // target resolution
      canvas.height = 720;
      canvasRef.current = canvas;
      const ctx = canvas.getContext("2d");

      // 5. preview (optional)
      if (videoRef.current) videoRef.current.srcObject = canvas.captureStream(30);

      // 6. draw loop
      function draw() {
        // draw screen frame
        if (screenStream) {
          const [sVideo] = screenStream.getVideoTracks();
          // draw video frame by drawing an offscreen video element
        }
        // For simplicity, create hidden video elements
        // (Simplified; in production reuse elements and check readyState)
        // draw screen full
        // draw camera small overlay at bottom-right
        rafRef.current = requestAnimationFrame(draw);
      }
      draw();

      // 7. capture composited stream
      compositedStream = canvas.captureStream(30);
      const compositedTrack = compositedStream.getVideoTracks()[0];

      // 8. publish composited video + mic
      await room.localParticipant.publishTrack(compositedTrack);
      await room.localParticipant.publishTrack(micTrack);
    }

    start();

    return () => {
      cancelAnimationFrame(rafRef.current);
      // stop tracks and disconnect
    };
  }, [url, token]);

  return (
    <div>
      <video ref={videoRef} autoPlay muted playsInline style={{ width: 640 }} />
    </div>
  );
}
3) Practical notes for production • Reuse hidden video elements to render camera and screen frames into canvas (avoid creating new elements each frame). • Set canvas size to target recording resolution to avoid runtime scaling. • Use 
canvas.captureStream()
with an appropriate FPS (e.g., 24 or 30) and set bitrate constraints via RTCRtpSender parameters if needed. • Test CPU usage on publisher devices — compositing is CPU-heavy on client side. • Publish only the composited track (and mic) to simplify server-side recording. If you want, I can: • Provide a complete, copy-paste React component with real draw loop using hidden video elements and cleanup logic. • Provide example LiveKit Track Composite API request body to record the published track. Which would you like next? , with interactive elements
3 Views
Open in Slack
PreviousNext
Powered by