For telephony, I see that voicemail detection is recommended to be done via tool call. Does that mean there is no built in AMD in livekit?

Thanks for reaching out, Karan. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

The LLM tool call seems non determinsitic and unreliable

Thanks — I hear you: you’re worried the LLM tool call approach for voicemail detection is nondeterministic and therefore unreliable. That’s a valid concern. I’ll explain the tradeoffs, practical ways to improve reliability if you want to try LLMs, and more deterministic alternatives you can use instead. Then tell me which direction you want to explore and I’ll give a concrete implementation plan. Why LLMs feel nondeterministic • LLMs produce probabilistic outputs; the same audio or transcript can yield different classifications across runs. • They may be sensitive to prompt phrasing, minor audio/transcription differences, and domain-specific phrasing (e.g., “please leave a message” vs a live person saying similar words). • Reported accuracy for LLM-based voicemail detection is around 70–80% — useful for many workflows but not guaranteed for 100% precision or recall. How to improve reliability if you keep an LLM step • Combine signals (don’t rely on the LLM alone): ◦ Use silence detection (e.g., >5s of silence before/after beep) as a strong heuristic. ◦ Use VAD /

user_started_speaking

events to measure speech timing and patterns. ◦ Use DTMF or call progress events from SIP when available. • Stabilize input to the LLM: ◦ Send structured features instead of raw audio when possible (e.g., silence durations, speech timestamps, presence of a beep, transcript snippets). ◦ Normalize transcripts (lowercase, remove filler tokens). ◦ Use clear, consistent prompts and include examples (few-shot) targeted to your audio patterns. • Apply thresholds and ensemble rules: ◦ Only mark voicemail if multiple signals (LLM score + silence + no human reply) cross thresholds. ◦ Require higher LLM confidence to take automated actions (e.g., leave a message) and lower confidence for human review. • Log and retrain prompts: ◦ Track misclassifications to refine prompts or examples. ◦ Use a small task-specific classifier trained on your labeled call audio/transcripts for higher consistency. More deterministic alternatives (recommended if you need reliability) • Rule-based detection using silence/VAD + timing: ◦ Heuristic: If call is answered and first speech starts after a long delay (e.g., >2–5s) with sustained silence earlier, treat as voicemail. ◦ Heuristic: If speech contains known answering machine phrases (“please leave a message”, “after the tone”), or if a tone/beep is detected, mark voicemail. • Classic AMD libraries / signal-processing: ◦ Use dedicated Answering Machine Detection algorithms or DSP libraries that analyze audio features (energy, pitch, cadence) and are deterministic. • Hybrid: deterministic front-end + ML fallback ◦ Use deterministic heuristics first. If inconclusive, then call the LLM (or a small ML classifier) as a tie-breaker. • Human-in-the-loop: ◦ For high-value calls, route low-confidence results to an agent to confirm. Which option should we detail next? • I can give a short step‑by‑step rule-based flow (silence/VAD + beep/transcript checks). • I can provide a hybrid flow with suggested thresholds and where to insert an LLM tie‑breaker. • I can sketch pseudocode / event sequence using LiveKit events (

user_started_speaking

, silence timers, tool calls) for whichever option you pick. Which would be most helpful — deterministic rule set, hybrid flow, or example pseudocode?

is there an AMD library compatible with livekit?

Thanks — I’m responding to your question about whether there’s an AMD library that’s already compatible with LiveKit. Short answer: LiveKit does not include a dedicated third‑party AMD library out of the box. However, LiveKit provides building blocks and recommended approaches you can use, and you can integrate external AMD solutions if you prefer a deterministic or specialized library. What LiveKit provides you can use for AMD: • Silence detection / timers — configurable silence handling (e.g., cut/mark call after ~5s) to detect likely voicemail. • VAD / speaking events —

user_started_speaking

and related events to observe timing and speech patterns. • LLM‑based tool calls — suggested for richer classification (approx. 70–80% accuracy), useful as a fallback or tie‑breaker. • End‑of‑Utterance (EOU) transformer — an open model LiveKit has published for turn/utterance detection that can be adapted to improve detection logic. How to integrate a third‑party or deterministic AMD library • Use LiveKit to capture or stream the call audio (or transcripts and timing features). • Run that audio through your chosen AMD library or DSP algorithm (locally or as a separate service). • Feed the AMD result back into your call flow (tool call, SIP routing, or application logic) to decide whether to leave a message, route to an agent, or hang up. Recommended patterns • Deterministic first: run rule‑based checks (silence + beep/dtmf/timing + phrase matching). • If inconclusive, run a specialized AMD library or ML classifier (or use LLM as a tie‑breaker). • For high‑value calls, route low‑confidence results to a human agent. If you want, I can: • Suggest specific open‑source AMD/DSP projects you could try (tell me if you prefer Python, Node, or native C++ libraries). • Sketch a concrete integration flow with LiveKit events and where to call the library (including example pseudocode). • Propose thresholds and signals to use for a deterministic-first approach. Which would you like next — recommendations for libraries by language, or an integration pseudocode flow? Sources: How to detect and handle silence during calls | examples/drive-thru | livekit/livekit_composite | DeepWiki | Using a transformer to improve end of turn detection | LiveKit | Using a transformer to improve end of turn detection | LiveKit