https://livekit.io logo
Join Slack
Powered by
Hi LiveKit Team, I’m building an interview platfo...
# egress
c
Hi LiveKit Team, I’m building an interview platform where I need to record candidates answering specific questions one by one. I would love a second pair of eyes on my architecture and configuration, as I am hitting severe stability issues in production that I didn't see during local testing. The Setup I am running a "Single Box" self-hosted setup. LiveKit Server + Egress + Redis are all running on the same machine via Docker Compose ( followed this guy's tutorial for the setup ) • Hardware: 8 vCPUs / 16 GB RAM. • Workflow: a. Candidate connects. b. For each question, I trigger Stop then Start of 2 Egresses via API. This is to ensure ti have separability between recordings that is align to when the question is displayed 1. I run two separate Egresses simultaneously for each question: ▪︎ 1x 
ParticipantEgress
for Camera + Audio (MP4). ▪︎ 1x 
ParticipantEgress
for Screen Share (Images/Screenshots). The Issues In production, I am seeing significant degradation: 1. Random A/V Desync: i don't know the reason and it happens randomly sometimes even with a fast connection. Logs show tracks being ignored due to delays sometimes reaching >1.5 seconds and even more. Sync is rarely recovered. 2. Data Loss: Videos sometimes start with black frames or have no audio at all. I played around with Egress params but it seems it created other problems ( like sometimes audio keep going off for few seconds and come back, Cf. the 
egress.yaml
config i put at the end of the message ) Below are my configurations. I would really appreciate a feedback in the setup and especially if am doing some trivial mistakes that i can correct immediatly. > Python Code Snippet for the Egresses ( excluding the stop and start logic )
Copy code
# 1. Camera + Audio Egress
camera_audio_output = EncodedFileOutput(
    filepath=camera_storage_path,
    file_type=EncodedFileType.MP4,
    disable_manifest=True,
)
camera_audio_request = ParticipantEgressRequest(
    room_name=room_name,
    identity=participant_identity,
    screen_share=False,  # Camera + audio only
    file_outputs=[camera_audio_output],
    preset=EncodingOptionsPreset.H264_720P_30
)

# 2. Screen Egress (only if question requires screen recording)
screen_output = ImageOutput(
    capture_interval=int(1 / settings.ocr_extraction_fps),
    filename_prefix=screen_storage_prefix  # e.g. "frame_000000.jpg", "frame_000001.jpg", etc.
)
screen_request = ParticipantEgressRequest(
    room_name=room_name,
    identity=participant_identity,
    screen_share=True,  # Record screen share
    image_outputs=[screen_output]
)
> React Frontend snippet that publishes the streams at the beginning of the interviews
Copy code
const publishPromises: Promise<void>[] = [];
      const userMediaStreamName = 'user-camera-mic';
      
      // Prepare camera + audio with synchronized stream (same stream name for A/V sync)
      if (preGrantedStreams.cameraStream) {
        const videoTrack = preGrantedStreams.cameraStream.getVideoTracks()[0];
        const audioTrack = preGrantedStreams.cameraStream.getAudioTracks()[0];
        
        if (videoTrack && audioTrack) {
          // Publish both camera and microphone simultaneously for better synchronization
          // Using the same stream name ensures they are placed in the same MediaStream
          publishPromises.push(
            room.localParticipant.publishTrack(videoTrack, {
              name: 'camera',
              source: Track.Source.Camera,
              stream: userMediaStreamName,
            }).then(() => {
              console.log('[handleStartInterview] ✓ Camera video track published');
            })
          );
          publishPromises.push(
            room.localParticipant.publishTrack(audioTrack, {
              name: 'microphone',
              source: Track.Source.Microphone,
              stream: userMediaStreamName,
            }).then(() => {
              console.log('[handleStartInterview] ✓ Microphone audio track published');
            })
          );
        } else if (videoTrack) {
          publishPromises.push(
            room.localParticipant.publishTrack(videoTrack, {
              name: 'camera',
              source: Track.Source.Camera,
              stream: userMediaStreamName,
            }).then(() => {
              console.log('[handleStartInterview] ✓ Camera video track published');
            })
          );
        }
      }

      // Prepare screen share
      if (preGrantedStreams.screenStream) {
        const screenTrack = preGrantedStreams.screenStream.getVideoTracks()[0];
        
        if (screenTrack) {
          publishPromises.push(
            room.localParticipant.publishTrack(screenTrack, {
              name: 'screen_share',
              source: Track.Source.ScreenShare,
              simulcast: false, // Disable simulcast for faster egress start
            }).then(() => {
              console.log('[handleStartInterview] ✓ Screen share track published (single quality layer)');
            })
          );
        }
      }

      // Publish all tracks simultaneously for better synchronization
      await Promise.all(publishPromises);

      // Mark as started
      setIsInterviewStarted(true);
      setSetupComplete(true);
      showNotification('Interview started successfully!', 'success');
> My Current Egress Configuration ( I am quite aware that i might set some params too high ).
Copy code
redis:
    address: <REDIS_ADDR>

storage:
    gcp:
        bucket: <GCS_BUKCKET>

cpu_cost:
  participant_cpu_cost: 1

latency:
  rtcp_sender_report_rebase_enabled: true 
  rtp_max_drift_adjustment: 3s
  jitter_buffer_latency: 3s
> And last my docker compose for running the live-kit services
Copy code
services:
  caddy:
    image: livekit/caddyl4
    command: run --config /etc/caddy.yaml --adapter yaml
    restart: unless-stopped
    network_mode: "host"
    volumes:
      - ./caddy.yaml:/etc/caddy.yaml
      - ./caddy_data:/data
  livekit:
    image: livekit/livekit-server:latest
    command: --config /etc/livekit.yaml
    restart: unless-stopped
    network_mode: "host"
    volumes:
      - ./livekit.yaml:/etc/livekit.yaml
  redis:
    image: redis:7-alpine
    command: redis-server /etc/redis.conf
    restart: unless-stopped
    network_mode: "host"
    volumes:
      - ./redis.conf:/etc/redis.conf
  egress:
    image: livekit/egress:latest
    restart: unless-stopped
    environment:
      - EGRESS_CONFIG_FILE=/etc/egress.yaml
    network_mode: "host"
    volumes:
      - ./egress.yaml:/etc/egress.yaml
    cap_add:
      - CAP_SYS_ADMIN
c
Hey @colossal-bear-85704 it would be great to have egress logs so that we can see what's going on there. Make sure you have debug level logging enabled. When it comes to latency config - bare in mind it's experimental, one thing you could try - in addition to the values you already have there, is adding
Copy code
latency:
  pre_jitter_buffer_receive_time_enabled: true
  packet_burst_estimator_enabled: true
These are experimental settings but might improve a/v sync issues though we can't be sure without seeing logs and what's causing it
c
Hey @clever-nail-49292 thanks for replying I just restarted the service some hours ago , not seing logs now but will transfer as soon as they come. Okay thanks for letting me know that the section is experimental, what i do remember seing in the logs, that before i put the latency configs i use to have synchronizer.go reports being ignored because the RTP adjusment asked for was too high ( i was seing 1.5 seconds ) , before my config was simple ( I think maybe i had a low participant_cpu_cost ) Otherwise is that something i would experience if i try LiveKit Cloud for example ? and keep the default parameters as is ( just adding the gcs bucket as destination )
Copy code
redis:
    address: <REDIS_ADDR>

storage:
    gcp:
        bucket: <GCS_BUKCKET>

cpu_cost:
  participant_cpu_cost: 0.5
c
For LiveKit Cloud - we are maintaining infra and making sure all services have enough resources and are configured properly (including egress) so you don't need to worry about this. We still don't know what's causing issues on your end without logs - but setting these 2 latency config flags I outlined for your on-prem build could help (and will likely be part of the default config soon)
c
Hello @clever-nail-49292 I just saw the logs after putting those new configurations in the egress, saw some data loss in the recorded video ( black frames apprearing 2 times in a video of 40 seconds, and voice not recorded as well at the same times those blach frames appear ) these are the egress logs :
Copy code
2026-01-02T11:00:56.259Z        INFO    egress  server/server_rpc.go:59 request received        {"nodeID": "NE_hKG5KcUNyB75", "clusterID": "", "egressID": "EG_sTLKtcjADf3S"}
2026-01-02T11:00:56.259Z        INFO    egress  server/server_rpc.go:69 request validated       {"nodeID": "NE_hKG5KcUNyB75", "clusterID": "", "egressID": "EG_sTLKtcjADf3S", "requestType": "participant", "sourceType": "EGRESS_SOURCE_TYPE_SDK", "outputType": "file", "room": "interview-a2dc38b5-c917-48f1-a418-40d6d6e76d08", "request": {"Participant":{"room_name":"interview-a2dc38b5-c917-48f1-a418-40d6d6e76d08","identity":"candidate-a2a1aa14-cb08-4a72-babe-a6c37c3c857e","Options":{"Preset":0},"file_outputs":[{"file_type":1,"filepath":"interviews/a2dc38b5-c917-48f1-a418-40d6d6e76d08/processed_segments/camera/q_0/video_audio_slice_q0.mp4","disable_manifest":true,"Output":null}]}}}
2026-01-02T11:00:56.286Z        INFO    egress  redis/redis.go:142      connecting to redis     {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_kjsMKmuY7w9Q", "clusterID": "", "egressID": "EG_sTLKtcjADf3S", "simple": true, "addr": "localhost:6379"}
2026-01-02T11:00:56.370Z        INFO    egress  source/sdk.go:412       subscribing to track    {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_kjsMKmuY7w9Q", "clusterID": "", "egressID": "EG_sTLKtcjADf3S", "trackID": "TR_VCGyKibz7QTVzg"}
2026-01-02T11:00:56.371Z        INFO    egress  source/sdk.go:412       subscribing to track    {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_kjsMKmuY7w9Q", "clusterID": "", "egressID": "EG_sTLKtcjADf3S", "trackID": "TR_AM6jxaBz6zNnSM"}
2026-01-02T11:00:56.371Z        INFO    egress  source/sdk.go:579       ignoring track  {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_kjsMKmuY7w9Q", "clusterID": "", "egressID": "EG_sTLKtcjADf3S", "reason": "source SCREEN_SHARE"}
2026-01-02T11:00:56.441Z        INFO    egress  server/server_rpc.go:59 request received        {"nodeID": "NE_hKG5KcUNyB75", "clusterID": "", "egressID": "EG_NrptS4zeqfMY"}
2026-01-02T11:00:56.442Z        INFO    egress  server/server_rpc.go:69 request validated       {"nodeID": "NE_hKG5KcUNyB75", "clusterID": "", "egressID": "EG_NrptS4zeqfMY", "requestType": "participant", "sourceType": "EGRESS_SOURCE_TYPE_SDK", "outputType": "images", "room": "interview-a2dc38b5-c917-48f1-a418-40d6d6e76d08", "request": {"Participant":{"room_name":"interview-a2dc38b5-c917-48f1-a418-40d6d6e76d08","identity":"candidate-a2a1aa14-cb08-4a72-babe-a6c37c3c857e","screen_share":true,"Options":null,"image_outputs":[{"capture_interval":5,"filename_prefix":"interviews/a2dc38b5-c917-48f1-a418-40d6d6e76d08/processed_segments/screen/q_0_frames/frame","Output":null}]}}}
2026-01-02T11:00:56.466Z        INFO    egress  redis/redis.go:142      connecting to redis     {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_eGGCpVMBZMc6", "clusterID": "", "egressID": "EG_NrptS4zeqfMY", "simple": true, "addr": "localhost:6379"}
2026-01-02T11:00:56.503Z        INFO    egress  source/sdk.go:579       ignoring track  {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_eGGCpVMBZMc6", "clusterID": "", "egressID": "EG_NrptS4zeqfMY", "reason": "source CAMERA"}
2026-01-02T11:00:56.503Z        INFO    egress  source/sdk.go:579       ignoring track  {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_eGGCpVMBZMc6", "clusterID": "", "egressID": "EG_NrptS4zeqfMY", "reason": "source MICROPHONE"}
2026-01-02T11:00:56.503Z        INFO    egress  source/sdk.go:412       subscribing to track    {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_eGGCpVMBZMc6", "clusterID": "", "egressID": "EG_NrptS4zeqfMY", "trackID": "TR_VSdRQmMe4LChZW"}
2026-01-02T11:00:56.630Z        INFO    egress  v2@v2.8.2-0.20250515012651-36db9bf95ff3/remoteparticipant.go:118       track subscribed        {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_kjsMKmuY7w9Q", "clusterID": "", "egressID": "EG_sTLKtcjADf3S", "participant": "candidate-a2a1aa14-cb08-4a72-babe-a6c37c3c857e", "track": "TR_VCGyKibz7QTVzg", "kind": "video"}
2026-01-02T11:00:56.630Z        INFO    egress  v2@v2.8.2-0.20250515012651-36db9bf95ff3/remoteparticipant.go:118       track subscribed        {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_kjsMKmuY7w9Q", "clusterID": "", "egressID": "EG_sTLKtcjADf3S", "participant": "candidate-a2a1aa14-cb08-4a72-babe-a6c37c3c857e", "track": "TR_AM6jxaBz6zNnSM", "kind": "audio"}
2026-01-02T11:00:56.651Z        INFO    egress  pipeline/watch.go:250   TR_AM6jxaBz6zNnSM playing       {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_kjsMKmuY7w9Q", "clusterID": "", "egressID": "EG_sTLKtcjADf3S"}
2026-01-02T11:00:56.651Z        INFO    egress  pipeline/watch.go:250   TR_VCGyKibz7QTVzg playing       {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_kjsMKmuY7w9Q", "clusterID": "", "egressID": "EG_sTLKtcjADf3S"}
2026-01-02T11:00:56.761Z        INFO    egress  v2@v2.8.2-0.20250515012651-36db9bf95ff3/remoteparticipant.go:118       track subscribed        {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_eGGCpVMBZMc6", "clusterID": "", "egressID": "EG_NrptS4zeqfMY", "participant": "candidate-a2a1aa14-cb08-4a72-babe-a6c37c3c857e", "track": "TR_VSdRQmMe4LChZW", "kind": "video"}
2026-01-02T11:00:56.779Z        INFO    egress  pipeline/watch.go:250   TR_VSdRQmMe4LChZW playing       {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_eGGCpVMBZMc6", "clusterID": "", "egressID": "EG_NrptS4zeqfMY"}
2026-01-02T11:00:58.123Z        INFO    egress  pipeline/watch.go:245   pipeline playing        {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_eGGCpVMBZMc6", "clusterID": "", "egressID": "EG_NrptS4zeqfMY"}
2026-01-02T11:00:58.127Z        INFO    egress  info/io.go:230  egress_active   {"nodeID": "NE_hKG5KcUNyB75", "clusterID": "", "egressID": "EG_NrptS4zeqfMY", "requestType": "participant", "outputType": "images", "error": "", "code": 0, "details": ""}
2026-01-02T11:00:58.185Z        INFO    egress  pipeline/watch.go:245   pipeline playing        {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_kjsMKmuY7w9Q", "clusterID": "", "egressID": "EG_sTLKtcjADf3S"}
2026-01-02T11:00:58.191Z        INFO    egress  info/io.go:230  egress_active   {"nodeID": "NE_hKG5KcUNyB75", "clusterID": "", "egressID": "EG_sTLKtcjADf3S", "requestType": "participant", "outputType": "file", "error": "", "code": 0, "details": ""}
2026-01-02T11:01:12.898Z        WARN    egress  jitter/buffer.go:274    buffer full, dropping sample    {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_kjsMKmuY7w9Q", "clusterID": "", "egressID": "EG_sTLKtcjADf3S", "trackID": "TR_AM6jxaBz6zNnSM", "kind": "audio"}
2026-01-02T11:01:12.898Z        WARN    egress  jitter/buffer.go:274    buffer full, dropping sample    {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_kjsMKmuY7w9Q", "clusterID": "", "egressID": "EG_sTLKtcjADf3S", "trackID": "TR_AM6jxaBz6zNnSM", "kind": "audio"}
2026-01-02T11:01:12.898Z        WARN    egress  jitter/buffer.go:274    buffer full, dropping sample    {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_kjsMKmuY7w9Q", "clusterID": "", "egressID": "EG_sTLKtcjADf3S", "trackID": "TR_AM6jxaBz6zNnSM", "kind": "audio"}
2026-01-02T11:01:12.898Z        WARN    egress  jitter/buffer.go:274    buffer full, dropping sample    {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_kjsMKmuY7w9Q", "clusterID": "", "egressID": "EG_sTLKtcjADf3S", "trackID": "TR_AM6jxaBz6zNnSM", "kind": "audio"}
2026-01-02T11:01:12.898Z        WARN    egress  jitter/buffer.go:274    buffer full, dropping sample    {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_kjsMKmuY7w9Q", "clusterID": "", "egressID": "EG_sTLKtcjADf3S", "trackID": "TR_AM6jxaBz6zNnSM", "kind": "audio"}
2026-01-02T11:01:22.113Z        INFO    egress  info/io.go:230  egress_ending   {"nodeID": "NE_hKG5KcUNyB75", "clusterID": "", "egressID": "EG_sTLKtcjADf3S", "requestType": "participant", "outputType": "file", "error": "", "code": 0, "details": "End reason: StopEgress API"}
2026-01-02T11:01:22.208Z        INFO    egress  info/io.go:230  egress_ending   {"nodeID": "NE_hKG5KcUNyB75", "clusterID": "", "egressID": "EG_NrptS4zeqfMY", "requestType": "participant", "outputType": "images", "error": "", "code": 0, "details": "End reason: StopEgress API"}
2026-01-02T11:01:22.237Z        INFO    egress  sdk/appwriter.go:182    writer finished {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_kjsMKmuY7w9Q", "clusterID": "", "egressID": "EG_sTLKtcjADf3S", "trackID": "TR_VCGyKibz7QTVzg", "kind": "video"}
2026-01-02T11:01:22.265Z        INFO    egress  sdk/appwriter.go:182    writer finished {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_kjsMKmuY7w9Q", "clusterID": "", "egressID": "EG_sTLKtcjADf3S", "trackID": "TR_AM6jxaBz6zNnSM", "kind": "audio"}
2026-01-02T11:01:22.324Z        INFO    egress  pipeline/watch.go:115   pipeline received EOS   {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_kjsMKmuY7w9Q", "clusterID": "", "egressID": "EG_sTLKtcjADf3S"}
2026-01-02T11:01:22.335Z        INFO    egress  v2@v2.8.2-0.20250515012651-36db9bf95ff3/engine.go:790   server initiated leave {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_kjsMKmuY7w9Q", "clusterID": "", "egressID": "EG_sTLKtcjADf3S", "reason": "CLIENT_INITIATED", "canReconnect": false}
2026-01-02T11:01:22.335Z        WARN    egress  source/sdk.go:654       disconnected from room  {"nodeID": "NE_hKG5KcUNyB75", "handlerID": "EGH_kjsMKmuY7w9Q", "clusterID": "", "egressID": "EG_sTLKtcjADf3S"}
2026-01-02T11:01:22.356Z        INFO    egress  sdk/appwriter.go:182    writer finished
I think it has something to do with the jitter buffer : 
WARN    egress  jitter/buffer.go:274    buffer full, dropping sample
7 Views
Open in Slack
PreviousNext
Powered by