Thanks — this is a great description and I can help. I’ll respond to two parts you highlighted: (A) the agent worker crashes / voice lag under concurrent calls, and (B) publishing TTS / hold music into rooms from a NestJS backend without spinning up a full agent session. I’ll also ask a few targeted questions so I can give precise next steps or sample code. A — Agent worker performance, crashes, and voice lag • What this suggests: repeated audio lag and the process errors you listed ("process did not exit in time, killing process", "sending SIGUSR1", "process exited with non-zero exit code -10") commonly indicate resource exhaustion (CPU, memory), starving real-time threads, or ungraceful shutdowns of worker subprocesses. • What you can try quickly: ◦ Scale horizontally: run multiple agent worker instances and spread jobs across them rather than packing many concurrent calls into one container. ◦ Limit per-worker concurrency: set a load function / load_threshold so each worker accepts only N active jobs (for example, max 8–10) instead of being overwhelmed. ◦ Pre-warm heavy models and dependencies at worker startup so per-call setup cost is reduced. ◦ Increase graceful shutdown timeout so long running jobs are drained instead of abruptly killed (the default force-kill timeout can be too short for audio pipelines). ◦ Check container resource limits: raise CPU and memory for the pod/container, and ensure realtime scheduling and thread priorities are not being starved by OS limits. ◦ Check logs around the SIGUSR1 and SIGSEGV — if the crashes are coming from native libraries (FFmpeg or audio codecs), they may point to codec mismatches or native memory issues. • If the problem is reproducible locally or on a single worker: ◦ Run the worker with monitoring (CPU, memory, thread count) while you ramp concurrent calls to identify which resource is maxing out. ◦ Enable verbose logging for the audio/codec libraries and for child processes to capture the exact failure point. • Useful operational changes: ◦ Auto-scale worker replicas on CPU/memory or on a custom metric like active job count. ◦ Ensure each agent job runs in an isolated process (so one job crash won’t bring others down) and increase process kill timeout if needed. ◦ If you’re using VAD/STT/LLM models, offload heavy tasks (LLM inference, large VAD models) to separate services where possible. B — Publishing TTS / background music into rooms without creating a full agent session You have three practical paths; pick one depending on reliability and complexity tradeoffs: Option 1 — Ingress (recommended for static/dynamic audio playout) • Ingress is designed to bring external audio into a LiveKit room via RTMP/WHIP and is a good fit for hold music or TTS playouts. • Typical flow: a. From your NestJS backend, create an ingress session for the target room (use the server-side SDK / ingress API). b. Stream audio into the provided ingress endpoint with FFmpeg (RTMP or WHIP). c. Stop the ingress when you no longer need the hold music. • FFmpeg tips (common causes of SIGSEGV): ◦ Use a stable FFmpeg binary compatible with your container’s architecture. ◦ Encode to a supported audio format (Opus is typical for WebRTC). Test locally with verbose logging: add

-loglevel debug

or

-report

to capture stack traces. ◦ Example basic looped stream command: ffmpeg -stream_loop -1 -i hold_music.mp3 -c:a libopus -b:a 128k -f flv rtmp://<ingress-url> • If FFmpeg crashes with SIGSEGV, inspect the exact command you ran, the binary version, and any filters/plugins. Native crashes often point to a codec or container mismatch or a problem with a binary built against different libs. • Benefits: efficient, scalable for static or pre-generated TTS, avoids keeping a full participant connection open. Option 2 — Lightweight "audio-only" agent participant (alternative) • Run a minimal participant (could be a tiny Node or Python process) that connects to the room as a user and publishes a single audio track for hold music or TTS. • This is effectively a small client connection instead of the heavier Agent job: easier to publish dynamic/real-time TTS from your backend (generate audio and push to the participant’s audio track). • Tradeoff: you must manage the connection lifecycle and token refreshes, but works well if Ingress is unstable for your environment. Option 3 — Use an Agent with only playout responsibilities • Spawn a minimal agent job whose sole job is to publish audio (no STT/LLM). This allows you to integrate with existing agent lifecycle but keeps CPU/LLM cost low. • Useful if you want lifecycle hooks and the ability to later expand behavior. Implementation help I can provide • If you want an Ingress-based solution, I can provide: ◦ A sample NestJS flow that requests/creates ingress and returns the ingress URL/credentials. ◦ A sample FFmpeg command tuned for Opus audio streaming into ingress. • If you want a Node or Python “audio participant” example, I can provide a small reproducible script that: ◦ Connects to LiveKit using a server token ◦ Publishes an audio track from a local file or a TTS stream ◦ Handles lifecycle (start/stop) • If you prefer an agent-lite approach, I can provide sample agent code that only handles audio playout. Targeted questions so I can produce exact sample code / diagnosis 1. Which approach would you like sample code for: Ingress (FFmpeg), Node/Python audio participant, or minimal agent job? 2. Please paste the exact FFmpeg command you tried and the container/FFmpeg version; also paste the ingress configuration you used. That will help diagnose the SIGSEGV. 3. What environment are the workers running in (Kubernetes, Docker on VM, other)? What CPU & memory limits are set on the failing container? 4. Rough scale: how many concurrent calls per worker do you expect in production? This helps me recommend sensible default load thresholds. 5. For TTS, which provider or format do you produce (MP3, WAV, raw PCM)? That will determine the most reliable FFmpeg pipeline. If you tell me which option you prefer (and provide FFmpeg command + environment info if you want help debugging SIGSEGV), I’ll send a concise, copy-pasteable sample: either NestJS + ingress creation + FFmpeg command, or a Node/Python participant script to publish audio into a room. Sources: An open source stack for real-time multimodal AI | Job lifecycle | LiveKit Documentation | Self-hosted deployments | LiveKit Documentation | Server options | LiveKit Documentation | Playing Audio | LiveKit Documentation