"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":6734,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986633,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":6702,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986634,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":6670,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986635,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":6638,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986636,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":6606,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986636,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":6574,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986639,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":6544,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986641,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":6512,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986641,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":6480,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986642,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":6448,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986643,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":6416,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986644,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":6384,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986644,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":6352,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986645,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":6320,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986646,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":6288,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986647,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":6256,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986648,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":6224,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986648,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":6192,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986649,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":6160,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986650,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":6128,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986651,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":6096,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986651,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":6064,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986703,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":6082,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986706,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":6053,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986708,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":6022,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986709,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":5990,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986710,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":5958,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986711,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":5926,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986711,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":5894,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986713,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":5862,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986714,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":5831,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986717,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":5801,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986719,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":5770,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986720,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":5738,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986721,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":5706,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986722,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":5674,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986723,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":5642,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986724,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":5610,"msg":"inference is slower than realtime"} {"level":40,"time":1766420986725,"pid":1504,"hostname":"deployment-p-551rhlf5aqv-ca-ubuzhpvxljvl-658d4cf5c7-tgspg","delay":5578,"msg":"inference is slower than realtime"} what causes this error

Thanks for reaching out, Bobvi. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

we are using livekit cloud and we are gettign this error here 💥 [CRASH-DIAG] UNHANDLED REJECTION: { reason: { message: 'EBADF: bad file descriptor, read', stack: 'Error: EBADF: bad file descriptor, read' }, activeJobCount: 0, activeJobs: [], memoryUsage: { rss: 490115072, heapTotal: 60424192, heapUsed: 50971000, external: 8303871, arrayBuffers: 4470525 }, pid: 142, timestamp: '2025-12-22T163348.994Z' } 💥 [CRASH-DIAG] UNHANDLED REJECTION: { reason: { message: 'EBADF: bad file descriptor, read', stack: 'Error: EBADF: bad file descriptor, read' }, activeJobCount: 0, activeJobs: [], memoryUsage: { rss: 490115072, heapTotal: 60424192, heapUsed: 50996048, external: 8436147, arrayBuffers: 4602801 }, pid: 142, timestamp: '2025-12-22T163349.318Z' } what is going on

Thanks — I’m responding to your crash logs showing repeated "EBADF: bad file descriptor, read" which means your agent tried to read from a file descriptor that is invalid or already closed. This typically comes from your agent code (not LiveKit Cloud itself). Below are targeted troubleshooting steps and a couple quick questions so I can help narrow it down. What to check now • Confirm runtime: Are you running a Node.js or Python agent? • Find the origin: search recent logs immediately before the crash for stack traces or filenames referenced by I/O calls. • Look for async file/stream usage that outlives the handle: ◦ Node:

fs.read()

,

fs.createReadStream()

or reopened/closed descriptors used in callbacks/promises. ◦ Python:

wave.open

, file objects used outside

with

blocks or in background tasks. • Ensure graceful shutdown handling: add SIGTERM/SIGINT handlers to stop background jobs, close streams, and await pending I/O before exit. • Wrap file I/O with error handling to avoid unhandled rejections: ◦ Node: check callback/Promise errors from

fs

APIs. ◦ Python: catch `OSError`/`ValueError` around reads. • Avoid using closed descriptors: ensure you don’t close a stream and then use its buffer later asynchronously. • Check for lifecycle events: LiveKit Cloud can scale/drain instances — verify your agent cleans up on termination to avoid operations during container shutdown. • If you read audio files repeatedly, consider loading into memory (buffer) once and reuse instead of reopening files concurrently. Immediate quick fixes to try • Add try/catch (or .catch) around all file reads and stream operations to prevent unhandled rejections. • Add logging right before every file read to capture which path/descriptor is being used when the error occurs. • Add a process.on('unhandledRejection') handler (Node) that logs stack/context to help identify caller. Questions to help me pinpoint the cause 1. Which runtime is the agent using (Node.js or Python) and which version? 2. Does the crash happen during startup, during a session, or on shutdown/scale events? 3. Can you paste a small code snippet that performs file I/O or any stack trace lines that reference files/streams? 4. Are you seeing any autoscaling/drain or SIGTERM logs just before the EBADF events? If you share the runtime and a short snippet or the log lines immediately preceding the crash, I’ll point to the exact change to prevent it and a minimal patch you can apply. Sources: Deploying to production | LiveKit Docs | Agent server options | LiveKit docs | Playing Audio | LiveKit docs | Examples | livekit/livekit_composite | DeepWiki | Still Have Questions

node js agent

Thanks — since you’re running a Node.js agent, I’ll focus on actionable, Node-specific steps for the EBADF crash and a short checklist for the “inference is slower than realtime” problem. Acknowledgement • I see repeated crashes with:

EBADF: bad file descriptor, read

and earlier logs showing inference lag. Let’s tackle the EBADF first (likely causing the crash), then confirm any inference tuning you still need. Node.js — targeted checks and fixes for EBADF 1. Add centralized handlers to capture context • Add these to log stack/context (temporary) so we can see where reads originate:

process.on('unhandledRejection', (err) => {
  console.error('UNHANDLED REJECTION', err);
});
process.on('uncaughtException', (err) => {
  console.error('UNCAUGHT EXCEPTION', err);
});

1. Identify problematic file/stream usage • Search for

fs.read

,

fs.readFile

,

fs.createReadStream

, or any native addon that reads file descriptors. • Add a brief log before each read/stream open to capture the filename/path and a unique operation id. 1. Fix common async/stream mistakes • Do not use file descriptors after closing them. If you call

fd = fs.openSync(...)

ensure you

fs.closeSync(fd)

only after all async work with that fd is done. • When using streams, ensure you don’t call

stream.destroy()

or

stream.close()

while other async handlers still assume it’s readable. • Example safe pattern for read streams:

const rs = fs.createReadStream(path);
rs.on('error', (err) => console.error('readstream error', err));
rs.on('open', () => console.log('opened', path));
rs.on('end', () => console.log('ended', path));

1. Guard against operations during shutdown/drain • Handle termination signals and stop starting new reads; wait for inflight I/O to finish:

let shuttingDown = false;
process.on('SIGTERM', async () => {
  shuttingDown = true;
  console.log('SIGTERM received — waiting for I/O to complete');
  await shutdownCleanup(); // close streams, stop jobs
  process.exit(0);
});

• In read paths, skip/abort if

shuttingDown

is true. 1. Wrap all file I/O promises and callbacks • Ensure every Promise-based fs call has a

.catch()

and callbacks check

err

. Unhandled promise rejections can mask origin.

fs.promises.readFile(path)
  .then(data => { /* use data */ })
  .catch(err => console.error('readFile failed', path, err));

1. Check third-party native modules • If you use native addons or binaries (audio libs, ffmpeg wrappers), they may close fds unexpectedly. Add try/catch and vendor-specific error handling. 1. Reduce concurrent file descriptor usage • If you open many files simultaneously, hitting OS fd limits or race conditions can expose EBADF. Use a pool/queue to limit concurrency. Quick debugging steps to run now • Deploy a build with the centralized handlers and extra logs around all read/open calls. • Reproduce the crash and paste the new logs showing which file/path and stack triggered EBADF. • If feasible, run the agent locally with the same workload to reproduce under a debugger. Short checklist for “inference is slower than realtime” (reminder) • Confirm you are not on burstable instances (t3/t4); CPU throttling causes big latency spikes. • Measure per-stage latency (STT, LLM token streaming, TTS, and predict_end_of_turn) to find the slow component. • Stream LLM output and start TTS on partial text to reduce end-to-end latency. • Consider moving to provisioned inference (LiveKit Inference) or dedicated instance types (

c5

,

m5

). Next steps from me • If you can paste: (A) the new logs after adding the handlers, (B) snippets where you open/read audio or files, and (C) whether the crash aligns with shutdown/scale events — I’ll point to the exact line to patch and provide a minimal code fix. Would you like me to show examples for common patterns you use (fs.promises, stream, or native addon)?