Hi AI Assistant, I’m building an interview platform where I need to record candidates answering specific questions one by one. I would love a second pair of eyes on my architecture and configuration, as I am hitting severe stability issues in production that I didn't see during local testing. The Setup I am running a "Single Box" self-hosted setup. LiveKit Server + Egress + Redis are all running on the same machine via Docker Compose ( followed this guy's tutorial for the setup ) • Hardware: 8 vCPUs / 16 GB RAM. • Workflow: a. Candidate connects. b. For each question, I trigger Stop then Start of 2 Egresses via API. This is to ensure ti have separability between recordings that is align to when the question is displayed 1. I run two separate Egresses simultaneously for each question: ▪︎ 1x

ParticipantEgress

for Camera + Audio (MP4). ▪︎ 1x

ParticipantEgress

for Screen Share (Images/Screenshots). The Issues In production, I am seeing significant degradation: 1. Random A/V Desync: i don't know the reason and it happens randomly sometimes even with a fast connection. Logs show tracks being ignored due to delays sometimes reaching >1.5 seconds and even more. Sync is rarely recovered. 2. Data Loss: Videos sometimes start with black frames or have no audio at all. I played around with Egress params but it seems it created other problems ( like sometimes audio keep going off for few seconds and come back, Cf. the

egress.yaml

config i put at the end of the message ) Below are my configurations. I would really appreciate a feedback in the setup and especially if am doing some trivial mistakes that i can correct immediatly.

Python Code Snippet for the Egresses ( excluding the stop and start logic )

# 1. Camera + Audio Egress
camera_audio_output = EncodedFileOutput(
    filepath=camera_storage_path,
    file_type=EncodedFileType.MP4,
    disable_manifest=True,
)
camera_audio_request = ParticipantEgressRequest(
    room_name=room_name,
    identity=participant_identity,
    screen_share=False,  # Camera + audio only
    file_outputs=[camera_audio_output],
    preset=EncodingOptionsPreset.H264_720P_30
)

# 2. Screen Egress (only if question requires screen recording)
screen_output = ImageOutput(
    capture_interval=int(1 / settings.ocr_extraction_fps),
    filename_prefix=screen_storage_prefix  # e.g. "frame_000000.jpg", "frame_000001.jpg", etc.
)
screen_request = ParticipantEgressRequest(
    room_name=room_name,
    identity=participant_identity,
    screen_share=True,  # Record screen share
    image_outputs=[screen_output]
)

React Frontend snippet that publishes the streams at the beginning of the interviews

const publishPromises: Promise<void>[] = [];
      const userMediaStreamName = 'user-camera-mic';
      
      // Prepare camera + audio with synchronized stream (same stream name for A/V sync)
      if (preGrantedStreams.cameraStream) {
        const videoTrack = preGrantedStreams.cameraStream.getVideoTracks()[0];
        const audioTrack = preGrantedStreams.cameraStream.getAudioTracks()[0];
        
        if (videoTrack && audioTrack) {
          // Publish both camera and microphone simultaneously for better synchronization
          // Using the same stream name ensures they are placed in the same MediaStream
          publishPromises.push(
            room.localParticipant.publishTrack(videoTrack, {
              name: 'camera',
              source: Track.Source.Camera,
              stream: userMediaStreamName,
            }).then(() => {
              console.log('[handleStartInterview] ✓ Camera video track published');
            })
          );
          publishPromises.push(
            room.localParticipant.publishTrack(audioTrack, {
              name: 'microphone',
              source: Track.Source.Microphone,
              stream: userMediaStreamName,
            }).then(() => {
              console.log('[handleStartInterview] ✓ Microphone audio track published');
            })
          );
        } else if (videoTrack) {
          publishPromises.push(
            room.localParticipant.publishTrack(videoTrack, {
              name: 'camera',
              source: Track.Source.Camera,
              stream: userMediaStreamName,
            }).then(() => {
              console.log('[handleStartInterview] ✓ Camera video track published');
            })
          );
        }
      }

      // Prepare screen share
      if (preGrantedStreams.screenStream) {
        const screenTrack = preGrantedStreams.screenStream.getVideoTracks()[0];
        
        if (screenTrack) {
          publishPromises.push(
            room.localParticipant.publishTrack(screenTrack, {
              name: 'screen_share',
              source: Track.Source.ScreenShare,
              simulcast: false, // Disable simulcast for faster egress start
            }).then(() => {
              console.log('[handleStartInterview] ✓ Screen share track published (single quality layer)');
            })
          );
        }
      }

      // Publish all tracks simultaneously for better synchronization
      await Promise.all(publishPromises);

      // Mark as started
      setIsInterviewStarted(true);
      setSetupComplete(true);
      showNotification('Interview started successfully!', 'success');

My Current Egress Configuration ( I am quite aware that i might set some params too high ).

redis:
    address: <REDIS_ADDR>

storage:
    gcp:
        bucket: <GCS_BUKCKET>

cpu_cost:
  participant_cpu_cost: 1

latency:
  rtcp_sender_report_rebase_enabled: true 
  rtp_max_drift_adjustment: 3s
  jitter_buffer_latency: 3s

And last my docker compose for running the live-kit services

services:
  caddy:
    image: livekit/caddyl4
    command: run --config /etc/caddy.yaml --adapter yaml
    restart: unless-stopped
    network_mode: "host"
    volumes:
      - ./caddy.yaml:/etc/caddy.yaml
      - ./caddy_data:/data
  livekit:
    image: livekit/livekit-server:latest
    command: --config /etc/livekit.yaml
    restart: unless-stopped
    network_mode: "host"
    volumes:
      - ./livekit.yaml:/etc/livekit.yaml
  redis:
    image: redis:7-alpine
    command: redis-server /etc/redis.conf
    restart: unless-stopped
    network_mode: "host"
    volumes:
      - ./redis.conf:/etc/redis.conf
  egress:
    image: livekit/egress:latest
    restart: unless-stopped
    environment:
      - EGRESS_CONFIG_FILE=/etc/egress.yaml
    network_mode: "host"
    volumes:
      - ./egress.yaml:/etc/egress.yaml
    cap_add:
      - CAP_SYS_ADMIN

Thanks for reaching out, Adam. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

when i first noticed the deyncs i have this configuration, that' why i tried playing with the params and actually it did reduce the desyncs for a while

redis:
    address: <REDIS_ADDR>

storage:
    gcp:
        bucket: <GCS_BUKCKET>

cpu_cost:
  participant_cpu_cost: 0.5

Thanks — I see you reverted

participant_cpu_cost

to 0.5 and want to know how that affects the symptoms and what to change. Short answer • Setting

participant_cpu_cost: 0.5

lowers the CPU budget the egress scheduler assigns per participant. That can cause the egress worker to under-allocate CPU to encoding/compositing work, which increases encoding latency, frame drops, and contributes to A/V drift and black/silent starts. Increase it (or remove the low value) and use other fixes below. Why this matters (concise) • Egress tasks (especially ParticipantEgress with a compositor/headless Chrome) require CPU for encoding and I/O. If the scheduler thinks each participant only needs 0.5 CPU, it may pack more concurrent jobs on the host than the machine can handle, causing CPU contention, delayed frames, encoder stalls, and missed keyframes. • Underprovisioning amplifies issues you already have from jitter buffer and frequent stop/start: slow startups miss first keyframes (black video), audio buffers underflow (no audio), and drift grows. Concrete recommendations 1. Set

participant_cpu_cost

to a realistic value ◦ For ParticipantEgress (Chrome compositing): use >= 1.0 (try 1.5 if you still run many concurrent jobs). ◦ For TrackComposite/TrackEgress (GStreamer): 0.5–1.0 may be adequate depending on resolution/bitrate. ◦ Start with

participant_cpu_cost: 1.0

and monitor. 1. Combine this with the latency fixes ◦ Set

jitter_buffer_latency: 200ms

and

rtp_max_drift_adjustment: 300ms

(as suggested earlier). Both lower latency tolerance and correct CPU allocation are needed together. 1. Prefer track-level egress to reduce CPU needs ◦ Switch from ParticipantEgress to TrackComposite/TrackEgress where possible; those are far lighter and less likely to need a high

participant_cpu_cost

. 1. Don’t run many concurrent heavy egress jobs on the same host ◦ If you must keep ParticipantEgress, move egress to a dedicated host or increase host CPU/RAM. Monitor (`docker stats`/htop) and ensure CPU isn’t pegged. 1. Use fewer start/stop cycles ◦ Long-running egress or splitting files post-hoc reduces churn and CPU spikes. Quick checklist to apply now • Change

participant_cpu_cost: 1.0

(or remove the custom value). • Update

egress.yaml

jitter settings to 200ms/300ms. • Restart egress service and run a load test while watching CPU usage. If you want, I can supply: • A recommended

egress.yaml

snippet for Participant vs Track egress, or • A short plan to test and tune

participant_cpu_cost

values under load. Which would you prefer?

A recommended egress value please

Thanks — here’s a concise, recommended

participant_cpu_cost

plus a minimal

egress.yaml

snippet and brief rationale. Recommended value • participant_cpu_cost: 1.0 — start here for ParticipantEgress on your single-box (8 vCPU / 16GB). Increase to 1.5 if you still run several concurrent heavy egress jobs (Chrome compositor) or see CPU contention. Why • 1.0 gives each egress job a realistic CPU budget so the scheduler won’t overcommit the host. Too-low values (0.5) allow too many concurrent jobs, causing encoding stalls, missed keyframes, and A/V drift. Full minimal egress.yaml snippet to apply (merge with your existing config)

redis:
  address: <REDIS_ADDR>

storage:
  gcp:
    bucket: <GCS_BUCKET>

cpu_cost:
  participant_cpu_cost: 1.0

latency:
  rtcp_sender_report_rebase_enabled: true
  rtp_max_drift_adjustment: 300ms
  jitter_buffer_latency: 200ms

Next steps • Restart the egress service after applying. • Run a production-like test while monitoring CPU and memory (

docker stats

/

htop

). If CPU usage is high and you run many concurrent ParticipantEgress jobs, raise to

1.5

or move egress to a dedicated host and consider switching to track-level egress for lower CPU usage.

What do you recommend as crash test setup ( what should i test , what should i monito, what parameters to finetune, and when will i have enough informations to be sure the setup is stable )

Thanks — below is a concise, actionable crash-test plan tailored to your interview workflow (participant egress + screen images) that covers what to test, what to monitor, which parameters to tune, and clear pass/fail criteria. Test scenarios (run progressively, repeat each 3×) 1. Baseline single interview ◦ 1 room, 1 candidate publishing camera+mic + optional screen share. ◦ Purpose: verify correctness after config changes. 1. Scale ramp (functional) ◦ Ramp participants: 1 → 10 → 25 → 50 in a single room; at each step run 10–15 minutes. ◦ Simulate typical publish mix: 100% audio, 25% camera+mic, 10% screen share. 1. Concurrency stress ◦ Simultaneously start 10 interviews (separate rooms) each with camera+mic and screen capture. ◦ For each interview, trigger egress start/stop pattern you expect (or long-running egress if you changed workflow). 1. Burst churn ◦ 10–20 participants join/leave within 30s; repeat several times to exercise join/leave/egress lifecycle. 1. Network impairment ◦ Add network conditions per client: packet loss 1–5%, jitter 50–200ms, bandwidth caps (upload 1–2 Mbps). ◦ Run 15–30 minute sessions under these conditions. 1. Long-run soak ◦ 60–120 minute session at expected peak load to detect memory leaks/gradual degradation. What to monitor (live + logs) • Host metrics (per host) ◦ CPU (per-core) and total: aim <80% sustained; spikes ok if brief. ◦ Memory: no OOMs; docker/container restarts. ◦ Disk I/O and free disk for recording outputs. ◦ Network I/O and interface errors. • LiveKit / Egress metrics ◦ Egress process CPU & memory per job. ◦ Number of concurrent egress jobs and job start latency. ◦ Egress error/failure rate and reasons (missing keyframe, timeout, crash). ◦ Track-level telemetry: packet loss %, jitter, inbound/outbound bitrate, frame rate. ◦ RTCP stats: dropped packets, rtx/retransmits. • Client metrics ◦ ICE connection success / time to connect. ◦ getStats() per track: framerate, jitter, packetLoss, currentBitrate. ◦ Time between publish and first frame/keyframe. • Application-level ◦ Recording artifacts: black-frame occurrences, initial audio silence, A/V drift seconds per recording. ◦ File integrity: ensure MP4s are playable, not truncated. ◦ Error logs from livekit-server and livekit-egress. How to validate recording quality (A/V sync & content) • Automated check per recorded file: ◦ Verify file duration matches wall-clock interval (±1s). ◦ Detect long initial black frames (>300ms) or silent audio (>300ms) at start. ◦ Compute A/V drift by comparing audio timestamps vs video timestamps (if available) or by sampling a clapper-like event (manual or synthetic short beep/frame) and measuring offset. • Keep a PCM-level or short beacon at question boundaries to measure consistent offsets. Parameters to tune (order) 1. egress.yaml latency: ◦ jitter_buffer_latency: start 200ms. ◦ rtp_max_drift_adjustment: start 300ms. 1. cpu_cost: ◦ participant_cpu_cost: start 1.0 for ParticipantEgress; 0.5–1.0 for TrackComposite. ◦ If CPU saturates, raise value or move egress off box. 1. Publishing limits: ◦ Max publish resolution 720p, 24–30fps. ◦ Limit screen capture fps (e.g., 1–2 fps for screenshots). 1. Client simulcast/adaptive: ◦ Enable simulcast/adaptive stream to save bandwidth during scale tests. 1. Egress lifecycle: ◦ Prefer long-running egress or track-level egress. If you must stop/start, add a short warm-up delay (1–2s) after publish before starting egress. Fine-tuning steps (iterative) 1. Run a scenario, observe CPU and egress error rate. 2. If CPU >80% or OOMs: move egress off the box or increase participant_cpu_cost and/or host resources. 3. If recordings show initial black/silent frames: add wait-for-first-frame on client and delay egress start 500–1000ms after publish confirmation. 4. If A/V drift persists: reduce jitter_buffer_latency/rpt_max_drift and re-run network-impaired tests; if that breaks recovery, increase CPU allocation or move to track-level egress. 5. If packet loss triggers layer switching often: tune simulcast bitrates and client adaptive settings. Pass/fail criteria (when you have enough info to call it stable) • Functional stability: ◦ 3 repeated runs of your peak test scenario with the same results. ◦ No crashes, restarts, or OOMs during soak tests. • Quality thresholds (per run) ◦ Recording success rate ≥ 99% (files created and playable). ◦ Initial black/silent start events < 0.5% of recordings. ◦ A/V drift ≤ 200ms for ≥ 99% of recordings (aligns with jitter buffer). ◦ Client disconnect rate < 0.5% and ICE failures < 0.5%. • Resource stability: ◦ CPU sustained <80% per host; memory stable (no leak over 2-hour soak). ◦ Egress job start latency predictable (e.g., <2s for TrackComposite, acceptable higher for ParticipantEgress). Practical tips • Use automated bots to reproduce consistent load (avoid human variability). • Capture precise timestamps at question boundaries (server and client) for post-test analysis. • Keep one test matrix where you change only one variable at a time (egress type, cpu_cost, jitter settings). • Keep logs centralized (ELK/Cloud logging) and keep the recorded outputs for offline verification. • If you need to debug a problematic recording, preserve the corresponding egress job ID and logs immediately. If you want, I can: • Provide a short script/example to automate the "publish → waitForFirstFrame → start egress" sequence. • Provide a TrackCompositeEgress example payload (Python). • Draft a minimal automated test plan (bots + network impairment) that you can run on your infra. Which follow-up would be most helpful: the automation script, the TrackComposite example, or the runnable test matrix? Sources: Going beyond a single-core | An introduction to WebRTC Simulcast | Diagnosing Connection Errors with Connection Test Utility | And…we’re Live(Kit)! | Announcing LiveKit Cloud