Thanks — I’m responding to your request for a detailed, implementable explanation of LiveKit’s human‑in‑the‑loop (HITL) system, including the logic, escalation criteria, agent transfer (warm transfer) flow, APIs, and best practices so you can reproduce the pattern for your agent. Summary — core idea • HITL lets an AI agent run routine tasks autonomously but pauses for human review/approval on high‑risk, ambiguous, or regulated actions. • Principle: propose → commit. The AI proposes an action; it does not execute irreversible or high‑risk steps until a human approves. When to escalate (escalation logic) • Risk level: high financial, legal, or health impact. • Model confidence: low LLM intent/entity confidence. • Complexity: multi‑step or ambiguous reasoning beyond agent scope. • Regulatory mandates: KYC, HIPAA, GDPR, PCI. • Sentiment/emotion: frustrated, angry, or the user explicitly requests a human. • Domain boundary breaches: request outside agent authorization. Implement escalation as a callable tool in your agent loop (treat “ask human” as a function the model can call). Encode the criteria in the tool description so the LLM can reason about when to call it. Warm transfer (agent transfer) — step‑by‑step workflow 1. Notify caller and set expectations ◦ Example: “Please hold while I connect you with a specialist. I’ll share everything we’ve discussed.” ◦ Use non‑interruptible playback to avoid dead air. 1. Place caller on hold ◦ Disable caller audio I/O and play hold music/filler speech. 1. Create consultation room (private briefing space) ◦ Spawn a private room where the agent can brief the human supervisor with full context. 1. Dial supervisor via SIP/outbound call ◦ Use an API to create a SIP participant (phone number or SIP URI) and ring the human. 1. Pass full context to human (briefing) ◦ Provide a complete evidence pack: ▪︎ Conversation transcript ▪︎ Detected intent + model confidence ▪︎ Extracted entities (account IDs, dates, amounts) ▪︎ Sentiment and compliance flags ▪︎ Customer history and policy notes 1. Connect supervisor to caller ◦ Move the supervisor participant from the consultation room into the caller’s room (or bridge them). Agent can introduce the human, then disconnect. 1. Agent disconnects; human and caller continue ◦ Leave a direct human-to-human call. Optionally support handback later. Key APIs / primitives you’ll need (conceptual) • WarmTransferTask (or your orchestration component): runs entire warm transfer flow end‑to‑end. • CreateSIPParticipant / outbound dialing: to call supervisors via SIP/phone. • MoveParticipant (or equivalent): to move/connect participants between rooms. • TransferSIPParticipant: for cold transfers (no briefing) if desired. • Chat/context object: single structured payload containing transcript, detections, entities, metadata. Example escalation function (concept) • Provide a callable function the agent invokes when escalation criteria are met. It should: a. Tell the caller you’re escalating. b. Wait for the message to finish. c. Run WarmTransferTask with chat context and target supervisor details. • Ensure allow_interruptions=False during the transfer announcement and wait for playout to finish before starting hold music. Blocking vs asynchronous HITL • Blocking (synchronous): Agent halts and waits for immediate human approval — used for live voice interactions. • Non‑blocking (asynchronous): Agent queues a review (e.g., via Slack/email) and continues / pauses depending on use case — used for backend approvals. Sub‑patterns and options • Interrupt & resume: Agent pauses, human provides input, agent resumes. • Human‑as‑tool: “Ask human” is modeled as a tool call inside reasoning. • Approval gate: Agent proposes action and waits for explicit human approval before commit. • Sampled approvals: Randomly sample low‑risk actions for human review to detect drift. • Exception‑only review: Only escalate when policy flags trigger. Bidirectional handoff • Humans can hand the session back to the agent for routine follow‑ups. Implement by watching participant lifecycle events and reactivating the agent when the human leaves. Avoiding context loss (critical) • Always pass the full evidence pack with every escalation. Context loss is the top UX failure. With full context, approval times drop dramatically and customers aren’t forced to repeat information. Production best practices • Use filler speech to avoid dead air; announce what you’re doing. • Implement SLAs and timeouts (e.g., supervisor response timeout → fallback routing). • Use idempotency keys to prevent duplicate actions on retries. • Pre‑handoff verification: confirm intent before transfer (“I’m connecting you to billing…”). • Log every escalation for observability and training. Feedback loop and continuous improvement • Store: original query, agent proposal, human correction/response. • Use that data to retrain or update prompts/heuristics. Expect escalation rates to drop over time as the agent improves. Implementation checklist to reproduce HITL • 1) Define escalation criteria and thresholds (confidence, risk, sentiment). • 2) Implement a callable escalate_to_human function/tool in your agent. • 3) Build orchestration: hold, consultation room, SIP dialing, context packaging, participant move. • 4) Ensure robust context packaging (transcript, intents, entities, flags). • 5) Add SLAs/timeouts, idempotency, and logging/observability. • 6) Implement human→agent handback lifecycle. • 7) Instrument and collect human correction data for retraining. Resources to follow (examples & testing) • Start from a warm transfer example repository and test in an agent console or agent builder. • Log and observe escalations, then iterate thresholds and briefing content. If you want, I can: • Provide a concrete code example for your stack (Python or Node) implementing the escalate_to_human function and WarmTransfer orchestration. • Draft the exact JSON schema for the “evidence pack” (transcript, intent/confidence, entities, flags). • Help design escalation criteria thresholds and sample prompt/tool descriptions for the LLM. Which of those would you like next (code example, evidence‑pack schema, or escalation threshold templates)? Sources: The Human-in-the-Loop (HITL) Pattern for Voice Agents | LiveKit, with interactive elements