Hi i am new to livekit and am way too confused with the deployment

Thanks for reaching out, Vinay. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

I am building a multilingual AI voice agent system and need help with architecture and implementation. My Current Stack - Telephony handled by Twilio. - Considering LiveKit for agent orchestration and state management. - LLM for reasoning and tool calling. - Speech recognition transcripts passed to the LLM. - Text-to-speech responses returned to the caller. What I Already Know Twilio capabilities - Twilio uses TwiML for voice responses ("<Say>", "<Gather>", etc.). - Language can be specified in "<Say language="xx-XX">". - Speech recognition language must be defined in "<Gather input="speech" language="...">". - Twilio does NOT auto-detect languages during speech recognition. - Language switching can only happen between gathers, not mid-recognition. - Twilio can produce Hinglish-style transcripts when using the "en-IN" recognition model. - Hindi transcripts appear in Devanagari when using "hi-IN". - Twilio webhooks ("/voice", "/process-audio") are stateless, so context must normally be stored externally. My current problem My current architecture looks like this: Caller → Twilio Twilio → "/voice" webhook Speech gather → "/process-audio" webhook LLM reasoning → TwiML response This leads to several problems: - Each webhook request is stateless. - Conversation context must be manually stored. - Tool execution becomes complicated. - Conversation history must be reloaded every request. - Managing state across "/voice", "/process-audio", and other actions is messy. Why I am interested in LiveKit I want to use LiveKit because it allows: - Persistent agent sessions - Tool definitions for the LLM - Automatic context handling during a call - Streaming audio conversations - Real-time agent loops instead of stateless HTTP webhooks My understanding is that the ideal architecture is: Phone Call → Twilio → Media Streams → LiveKit Room → LiveKit Agent Then inside the LiveKit agent: Speech → STT → LLM reasoning → Tool calls → Response generation → TTS audio back to caller This gives: - persistent context window - simpler tool definitions - continuous conversation loop - no stateless webhook orchestration My Goal I want to design a clean architecture for a multilingual AI voice agent that: - handles Hinglish and Indian regional languages - maintains context during the entire call session - allows easy definition of tools for the LLM - integrates Twilio telephony with LiveKit agents - streams audio for low latency interactions - avoids complex session management between webhooks Additional constraints - I currently have $14 Twilio trial credits for development. - I want a system that is easy to test and iterate on. - I want the agent to be able to call tools such as: - checking order status - cancelling orders - retrieving user data - external API calls What I want help with Please help me with: 1. The best architecture for Twilio + LiveKit integration 2. How to maintain context and memory across a call session 3. How to define and manage LLM tools effectively 4. How to structure the LiveKit agent loop 5. Best practices for multilingual voice agents 6. How to minimize complexity compared to Twilio-only IVR systems Assume I am comfortable with backend development and system architecture. Provide practical, implementation-oriented guidance. Now i want to create a python sdk script foor this agent tell me the complete steps

Thanks — this is a great, detailed brief. I’ll walk you through a practical, implementation‑oriented plan and then give a complete Python SDK script you can run and iterate on. I’ll address: • Recommended architecture for Twilio + LiveKit • How to keep context and session memory • Practical tool definition for LLMs • The LiveKit agent loop structure • Multilingual best practices (Hinglish/Indian languages) • How this reduces complexity vs Twilio-only IVR • A runnable Python SDK example with step-by-step instructions If anything below is not what you meant (for example you want a lighter local-only demo first or Kubernetes deployment), tell me and I’ll adjust. 1) Recommended architecture (high level) • Caller → Twilio (programmable voice / Media Streams or SIP) → LiveKit inbound trunk → LiveKit Room (one persistent room per call) → LiveKit Agent (session/stateful) • Inside the LiveKit Agent: ◦ Incoming audio → STT (multilingual model) ◦ STT text → LLM reasoning (with function/tool calling) ◦ LLM may call tools (order lookup, cancel, external APIs) ◦ LLM text → TTS → audio back to caller via LiveKit • Why: LiveKit Rooms and AgentSessions give you a persistent session (conversation history, tools, session-scoped state and audio streaming) instead of stateless Twilio webhooks and TwiML gather loops. 2) How to maintain context and memory across a call session • Use the LiveKit AgentSession for in-call state: it preserves conversation history, tool invocation history, and session variables for the life of the call. • Keep ephemeral state in the session (no DB roundtrips per turn). Persist only long-term data (user profile, order history) in your DB. • For cross-call memory (e.g., returning customers), store a key like customer_id and load related context at session start. • If you ever need to rehydrate an AgentSession (e.g., reconnect or resume), store minimal session metadata externally (session id → user id / call metadata) to restore or link records. 3) How to define and manage LLM tools effectively • Make each business capability its own tool (check_order_status, cancel_order, fetch_user_profile, external_api_call). • Tools should: ◦ Accept clear typed arguments. ◦ Return structured results (JSON or a clearly formatted string). ◦ Handle errors and timeouts gracefully. • Implement tools inside your Agent class so they can access session state and emit side effects (logs/events). • Let the LLM decide when to call a tool using function-calling. On return, include tool output in the LLM context for follow-up replies. 4) How to structure the LiveKit agent loop • Use an AgentSession abstraction that handles: a. Streaming audio → VAD/segments → STT b. Feeding transcripts + session history to the LLM c. LLM may call tools — execute them and return results to the LLM d. LLM generates response text → TTS → play back e. Loop until call ends • Use VAD (silero or similar) to detect turns and interruptions for natural conversation. • Start the session with a short greeting or language detection probe (see multilingual tips). • Reactively handle DTMF or explicit IVR-like inputs by mapping them to tools or state transitions if needed. 5) Best practices for multilingual voice (Hinglish & regional languages) • STT: ◦ Use a multilingual STT model that supports code-switching (Hinglish). If you must rely on Twilio’s STT, specify recognition language per gather; Twilio will not auto-detect language mid-gather. ◦ For better code-switch handling, prefer an STT model labeled as multi-language or specialized for the region. • Language detection: ◦ At session start, run a short language probe (ask caller to say a phrase) or detect language from the first 5–10 seconds of transcript. ◦ Save preferred language in the AgentSession to guide LLM and TTS. • LLM: ◦ Use a model that handles multiple languages reliably; include the preferred language in system prompt. ◦ For Hinglish, pass transcripts as-is (code-switched) and instruct the LLM to reply in appropriate language or script. • TTS: ◦ Use TTS voices supporting target languages; if script differs (e.g., Devanagari), ensure TTS supports correct orthography or transliteration. • Turn-taking: ◦ Use VAD plus short utterance buffering to avoid truncation. ◦ Support interrupts: if user speaks while TTS is playing, immediately stop playback and pass the audio to the model. 6) How this reduces complexity vs Twilio-only IVR • Removes webhook orchestration: LiveKit session manages conversation; TwiML gather loops are no longer needed for each turn. • Keeps single source of truth (session state inside AgentSession), simplifying tool integration and state transitions. • Lower latency for real-time interactions via WebRTC streaming rather than repeated HTTP uploads/downloads. • Easier iteration and testing: change agent behavior without reworking TwiML flows. Now — step-by-step Python SDK script and deployment guide Below is a practical script and instructions to get a working agent. This example: • Uses a multilingual STT model • Uses function-style tools for order lookup/cancel • Runs an AgentServer that will be invoked when a call routes into a LiveKit room Requirements & assumptions before starting: • You have a LiveKit account or self-hosted LiveKit and an API key/secret. • You can configure Twilio to forward calls as SIP to your LiveKit inbound trunk (or use LiveKit phone number setup). • You have environment access for running the Python agent server during testing. • You will replace placeholder API calls (order service, DB, and external credentials) with your implementations. Step 0 — Install dependencies 1. Create a Python venv and activate it. 2. Install required packages: ◦ pip install "livekit-agents[silero,deepgram,openai,cartesia]" python-dotenv Step 1 — Prepare environment variables Create a .env file (or set env vars) with: • LIVEKIT_URL (e.g., wss://your-livekit-host) • LIVEKIT_API_KEY • LIVEKIT_API_SECRET • OPENAI_API_KEY (or your LLM provider key) • Any third-party STT/TTS keys if required Example .env: LIVEKIT_URL=https://your-livekit-host LIVEKIT_API_KEY=lk_api_key LIVEKIT_API_SECRET=lk_api_secret OPENAI_API_KEY=sk-... Step 2 — Create agent code Save the following to a file named agent.py

# agent.py
import asyncio
import logging
from dotenv import load_dotenv
from livekit.agents import AgentServer, AgentSession, Agent, JobContext
from livekit.agents.tools import function_tool
from livekit.plugins import silero

load_dotenv()
logging.basicConfig(level=<http://logging.INFO|logging.INFO>)
logger = logging.getLogger("multilingual_agent")

server = AgentServer()

# Optional: preload VAD/other heavy models to speed cold starts
def prewarm(proc):
    proc.userdata["vad"] = silero.VAD.load()

server.setup_fnc = prewarm

class VoiceAssistant(Agent):
    def __init__(self):
        super().__init__(
            instructions=(
                "You are a helpful multilingual voice assistant for an e-commerce service. "
                "You can speak Hindi and English and handle code-switched (Hinglish) input."
            )
        )

    @function_tool
    async def check_order_status(self, order_id: str) -> str:
        # TODO: replace with real API call to your order system
        # Simulate lookup
        return f"Order {order_id} is out for delivery and will arrive tomorrow."

    @function_tool
    async def cancel_order(self, order_id: str) -> str:
        # TODO: call your cancel API and handle errors
        return f"Order {order_id} has been canceled successfully."

    async def on_enter(self):
        # welcome the caller
        await self.session.generate_reply(instructions="Welcome! How can I help you today?")

@server.rtc_session(agent_name="multilingual-agent")
async def entrypoint(ctx: JobContext):
    # ctx.room is the LiveKit room for this call
    <http://logger.info|logger.info>("Starting agent session for room: %s", ctx.room.name)

    session = AgentSession(
        stt="deepgram/nova-3:multi",
        llm="openai/gpt-4.1-mini",
        tts="cartesia/sonic-3",
        vad=ctx.proc.userdata.get("vad"),
    )

    await session.start(room=ctx.room, agent=VoiceAssistant())
    await ctx.connect()  # keep session alive while connected
    <http://logger.info|logger.info>("Agent session ended for room: %s", ctx.room.name)

if __name__ == '__main__':
    server.run()

Step 3 — Run the agent server locally • python agent.py • The server should prewarm VAD and listen for LiveKit session connections. Step 4 — Configure LiveKit inbound trunk and Twilio • In LiveKit dashboard, create an inbound SIP trunk and add a dispatch rule to route incoming calls to a room/agent (use the agent name configured above). • In Twilio, create a SIP or SIP Trunk configuration (or use Twilio Media Streams) to send calls to the LiveKit SIP endpoint or phone number forwarding LiveKit. • For Sources: LiveKit Phone Numbers | Connect voice agents to phone lines | LiveKit | Creating SIP Inbound Trunks and Dispatch Rules with Python SDK | Agent session | LiveKit Documentation | An open source stack for real-time multimodal AI | LiveKit | examples/bank-ivr | livekit/livekit_composite | DeepWiki

currently i am just building this for testing and want to make a script that will run in my system and will use my system mic to interact with em and save the conversation data in a formated json file just the way i tell it

Thanks — that’s a clear goal and a great way to iterate quickly. Below I’ll give a concise, practical Python script you can run locally that: • Captures audio from your system microphone • Sends audio to a multilingual STT (local or cloud) • Passes transcripts to a simple agent loop (LLM or rule-based) • Produces TTS audio played to your speakers • Logs each turn (user + assistant) in a formatted JSON file you control Before you run it, quick clarifying questions so I tailor the script: • Do you want to use a local STT (e.g., VOSK/silero) or a cloud STT (Deepgram, OpenAI Whisper via API)? Cloud is easier but needs keys; local avoids API costs. • Same for TTS: prefer local playback (pyttsx3) or cloud TTS (higher quality)? If you answer now I can adjust. I’ll provide a cloud-optional script (works with local fallback) so you can test immediately. What this example does • Records short user turns using your mic (VAD or push-to-talk) • Sends audio to STT (local Whisper/Deepgram fallback) • Sends transcript to an LLM function (placeholder — you can wire OpenAI) • Converts LLM reply to TTS and plays it • Appends structured turns to a JSON array and writes to a file on exit Requirements • Python 3.8+ • Basic packages: sounddevice, soundfile, requests (or SDKs for Deepgram/OpenAI), pyttsx3 (local TTS) or playsound • Microphone and speakers connected Minimal environment setup 1. Create and activate a venv: ◦ python -m venv venv && source venv/bin/activate (mac/linux) or venv\Scripts\activate (Windows) 1. Install: ◦ pip install sounddevice soundfile pyttsx3 requests python-dotenv .env (if using cloud) • If you plan to use Deepgram/OpenAI set in .env: ◦ DEEPGRAM_API_KEY=... ◦ OPENAI_API_KEY=... Script (save as local_agent.py) • This is self-contained. Replace the STT/TTS functions with cloud calls if you prefer higher accuracy.

# local_agent.py
import os
import queue
import threading
import json
import time
import datetime
from pathlib import Path
from dotenv import load_dotenv

import sounddevice as sd
import soundfile as sf
import pyttsx3
import requests  # optional for cloud STT/LLM
# pip install sounddevice soundfile pyttsx3 requests python-dotenv

load_dotenv()

OUTPUT_JSON = Path("conversation_log.json")
SAMPLE_RATE = 16000
CHANNELS = 1
CHUNK_SECONDS = 4  # record 4s chunks; adjust for VAD or push-to-talk

# Conversation structure
conversation = []

# Local TTS engine (fallback)
tts_engine = pyttsx3.init()
tts_engine.setProperty("rate", 150)

def save_conversation():
    with open(OUTPUT_JSON, "w", encoding="utf-8") as f:
        json.dump(conversation, f, ensure_ascii=False, indent=2)
    print(f"Saved conversation to {OUTPUT_JSON}")

def record_chunk(duration=CHUNK_SECONDS):
    print(f"Recording for {duration}s...")
    recording = sd.rec(int(duration * SAMPLE_RATE), samplerate=SAMPLE_RATE, channels=CHANNELS, dtype='int16')
    sd.wait()
    return recording.flatten()

def write_wav(data, filename="chunk.wav"):
    sf.write(filename, data, SAMPLE_RATE, subtype="PCM_16")
    return filename

# STT: try Deepgram (cloud) if API key present, otherwise use a naive local stub
def transcribe_audio(wav_path):
    dg_key = os.getenv("DEEPGRAM_API_KEY")
    if dg_key:
        print("Using Deepgram STT")
        with open(wav_path, "rb") as f:
            r = <http://requests.post|requests.post>(
                "<https://api.deepgram.com/v1/listen?language=auto>",
                headers={"Authorization": f"Token {dg_key}"},
                data=f,
            )
        if r.ok:
            resp = r.json()
            # deepgram returns alternatives; adjust depending on response shape
            transcript = resp.get("results", {}).get("channels", [])[0].get("alternatives", [])[0].get("transcript", "")
            lang = resp.get("metadata", {}).get("language", "unknown")
            return transcript.strip(), lang
        else:
            print("Deepgram error:", r.text)
            return "", "unknown"
    else:
        # Local fallback stub: uses simple whisper api or placeholder
        print("No Deepgram key found — using local placeholder STT")
        # For quick testing, ask user to type the transcript instead of real STT
        typed = input("Type what you said (local test mode): ").strip()
        return typed, "und"

# LLM: simple rule-based or placeholder; swap with OpenAI API if you have a key
def llm_respond(transcript, language="und"):
    openai_key = os.getenv("OPENAI_API_KEY")
    if openai_key:
        print("Calling OpenAI (placeholder) — implement real call")
        # implement OpenAI chat completion function call here
        # For now return an echo
        return f"I heard: {transcript}"
    else:
        # Very simple local logic for testing
        if "order" in transcript.lower():
            return "I can check your order. What is the order id?"
        if "cancel" in transcript.lower():
            return "To cancel, please confirm the order id."
        if transcript.strip() == "":
            return "Sorry, I didn't catch that. Could you repeat?"
        return f"You said: {transcript}"

# TTS: play assistant reply
def speak_text(text, language="en"):
    print("Assistant:", text)
    # For local: pyttsx3
    tts_engine.say(text)
    tts_engine.runAndWait()

def append_turn(role, content, language):
    conversation.append({
        "role": role,
        "content": content,
        "language": language,
        "timestamp": datetime.datetime.utcnow().isoformat() + "Z"
    })

def main_loop():
    try:
        print("Starting local agent. Press Ctrl+C to exit.")
        while True:
            data = record_chunk()
            wav = write_wav(data, "chunk.wav")
            transcript, lang = transcribe_audio(wav)
            append_turn("user", transcript, lang)
            reply = llm_respond(transcript, language=lang)
            append_turn("assistant", reply, lang)
            speak_text(reply, language=lang)
            # short pause
            time.sleep(0.2)
    except KeyboardInterrupt:
        print("Shutting down.")
        save_conversation()

if __name__ == "__main__":
    main_loop()

How to use • Run: python local_agent.py • Speak into your mic (or type in the prompt if no Deepgram key) • Each user/assistant turn is appended to conversation and on exit saved to conversation_log.json • Edit CHUNK_SECONDS or replace the naive record_chunk with VAD or push-to-talk to get better UX Custom JSON formatting • The script appends objects with keys: role, content, language, timestamp. • If you want a different structure (e.g., including turn_id, tool_calls, confidence), change append_turn() to include those fields. • Example: add tool metadata: ◦ append_turn("assistant", reply, lang) -> include {"tools_invoked": [...], "tool_outputs": {...}} Next steps I can help with • Swap the simple placeholders for real Deepgram / OpenAI API calls and provide exact request examples. • Replace audio chunks with VAD so the system captures real conversational turns (better for natural interaction). • Add function/tool calling: show how LLM function-calling results trigger your order/cancel API and are logged in JSON. • Change TTS from pyttsx3 to a higher-quality cloud TTS and show how to stream audio back. • Show how to save richer logs (timestamps per segment, STT confidences, audio file references) in JSON. Tell me: • Do you want cloud STT/LLM/TTS examples (I’ll include exact API request code), or keep everything local for now? • Any specific JSON schema you want (I’ll provide the schema and update the script)? Sources: Build a multilingual voice agent that automatically switches languages | Li...

want to uuse livekit in this

Great — since you want to run this locally with LiveKit, use your system mic, and save conversation data in a JSON file exactly as you define it, here’s a clear, runnable plan and a complete Python example you can copy/paste and run. Quick clarifying question before the code: • Do you want cloud STT/LLM/TTS (Deepgram / OpenAI / Cartesia) or local-only (VOSK/whisper/silero + pyttsx3)? I’ll provide a script that works locally by default and shows where to swap in cloud calls if you want higher accuracy later. What this solution gives you • Uses LiveKit Agents SDK in console mode to capture mic/speaker audio locally • Creates an AgentSession that keeps context for the whole run • Logs each user & assistant turn to a JSON file with fields you choose • Simple place to plug in tool calls (order lookup/cancel) and richer metadata • Works for quick testing and iteration; you can later swap components for production Prep (requirements) 1. Python 3.8+ 2. Create and activate a venv: ◦ linux/mac: python -m venv venv && source venv/bin/activate ◦ windows: python -m venv venv && venv\Scripts\activate 1. Install packages: ◦ pip install "livekit-agents[silero,deepgram,openai,cartesia]" python-dotenv ◦ (If you prefer local-only STT/TTS, you can omit cloud extras and install pyttsx3/sounddevice, but the example below uses LiveKit console mode and the LiveKit plugins.) Environment • Create a .env file with your provider keys if you’ll use cloud: ◦ LIVEKIT_API_KEY, LIVEKIT_API_SECRET, LIVEKIT_URL ◦ DEEPGRAM_API_KEY (optional) ◦ OPENAI_API_KEY (optional) • For a first run you can still use console mode locally without cloud keys — the agent will run and you can type or use fallback flow. How it works (short) 1. Run the agent in LiveKit console mode — it captures mic and plays TTS to speakers. 2. AgentSession uses multilingual STT and TTS (configurable). 3. We hook to session events to log each transcript and assistant reply into a JSON array. 4. On shutdown (Ctrl+C) we write the final JSON to disk. Complete runnable example • Save as agent_console.py

# agent_console.py
import os
import json
import asyncio
import datetime
from pathlib import Path
from dotenv import load_dotenv

from livekit.agents import AgentServer, AgentSession, Agent, JobContext, cli
from livekit.agents.tools import function_tool
from livekit.plugins import silero, deepgram, inference

# Load environment
load_dotenv()

OUTPUT_DIR = Path("console-recordings")
OUTPUT_DIR.mkdir(exist_ok=True)

# Configure what fields you want in the JSON
def make_log_entry(role: str, text: str, language: str = None, meta: dict = None):
    return {
        "timestamp": datetime.datetime.utcnow().isoformat() + "Z",
        "role": role,
        "text": text,
        "language": language or "und",
        "meta": meta or {},
    }

# In-memory conversation log (append-only)
conversation_log = []

# Simple agent with tools (example)
class LocalTestAgent(Agent):
    def __init__(self):
        super().__init__(instructions="You are a helpful multilingual assistant. Reply in the user's language.")

    @function_tool
    async def check_order_status(self, order_id: str) -> str:
        # Replace this with a real API call
        return f"Order {order_id} is out for delivery."

    @function_tool
    async def cancel_order(self, order_id: str) -> str:
        # Replace this with a real API call
        return f"Order {order_id} has been canceled."

# Create server
server = AgentServer()

# Prewarm VAD to reduce cold start (optional)
def prewarm(proc):
    proc.userdata["vad"] = silero.VAD.load()
server.setup_fnc = prewarm

@server.rtc_session(agent_name="local-console-agent")
async def entrypoint(ctx: JobContext):
    # Choose STT/LLM/TTS config. Replace models or providers as desired.
    stt = deepgram.STT(model="deepgram/nova-3-general", language="multi")  # multilingual
    llm = inference.LLM(model="openai/gpt-4o")  # replace if you have a different LLM
    tts = inference.TTS(model="cartesia/sonic-3")  # replace if you want other TTS

    session = AgentSession(
        stt=stt,
        llm=llm,
        tts=tts,
        vad=ctx.proc.userdata.get("vad"),
    )

    # Log incoming transcripts (user)
    @session.on("user_input_transcribed")
    async def on_user_transcript(trans):
        if not hasattr(trans, "transcript"):
            return
        # Add user turn when final
        if getattr(trans, "is_final", True):
            entry = make_log_entry("user", trans.transcript, getattr(trans, "language", None), {
                "confidence": getattr(trans, "confidence", None),
            })
            conversation_log.append(entry)
            print(f"\n[USER] {trans.transcript}")

    # Log assistant responses and tool calls
    @session.on("conversation_item_added")
    async def on_conversation_item(item):
        # item may be assistant text, tool_result etc. Adjust based on item.shape
        role = getattr(item, "role", "assistant")
        text = getattr(item, "text", "") or getattr(item, "content", "")
        entry = make_log_entry(role, text, getattr(item, "language", None), {
            "type": getattr(item, "type", None),
            "meta": getattr(item, "meta", None),
        })
        conversation_log.append(entry)
        print(f"\n[ASSISTANT] {text}")

    # Save JSON whenever a new entry is added (optional)
    def save_json_snapshot():
        fname = OUTPUT_DIR / f"conversation-{datetime.datetime.utcnow().strftime('%Y%m%d-%H%M%S')}.json"
        with open(fname, "w", encoding="utf-8") as f:
            json.dump(conversation_log, f, indent=2, ensure_ascii=False)
        print(f"Saved snapshot to {fname}")

    # Add a shutdown callback to persist final log
    ctx.add_shutdown_callback(lambda: save_json_snapshot())

    # Start session with the agent
    await session.start(room=ctx.room, agent=LocalTestAgent())

    # Optionally greet
    await session.generate_reply(instructions="Greet the user briefly and ask how you can help.")

    # Keep the session alive while connected to console
    await ctx.connect()

if __name__ == "__main__":
    # Run the agent server in console mode:
    # This exposes microphone/speaker I/O in your terminal environment.
    # Use: python agent_console.py console --record
    cli.run_app(server)

How to run (local test) 1. Start: ◦ python agent_console.py console --record ◦ This opens console mode, captures mic input, plays TTS to speakers, and the LiveKit console UI displays transcripts. 1. Interact: ◦ Speak into your mic. LiveKit handles VAD & STT and the agent responds. ◦ You can press Ctrl+T in console mode to toggle between voice and text input. 1. Stop: ◦ Ctrl+C to stop; the script’s shutdown callback will save the final JSON snapshot into console-recordings. Customizing the JSON format • The function make_log_entry controls what fields go into each log row. Add: ◦ turn_id, session_id, audio_file (path), tool_calls (list), stt_confidence, latencies, etc. • If you want a single file that updates in place, change save_json_snapshot to write to a fixed filename (e.g., conversation.json) after each new item. Where to plug in tools (order/cancel) • Implement the actual API calls inside the tool methods (check_order_status, cancel_order). • When tools run, their outputs appear in the conversation stream and will be logged by the conversation_item_added handler. Switching to cloud-only STT/LLM/TTS • Replace deepgram.STT / inference.LLM / inference.TTS with the cloud providers and set keys in .env. • For higher accuracy in Indian languages and code-switching, Deepgram’s multilingual models and Cartesia/Rime TTS are good options — configure language options as needed. Next steps I can help with • Provide a version that uses local STT/TTS if you want zero cloud usage. • Add VAD tuning or push-to-talk to improve capture quality. • Show exact code to wire OpenAI function-calling -> tool invocation -> structured logging. • Provide a JSON schema example you want (e.g., include tool call entries, audio file references, confidence scores). If you want, I can now: • Send a variant that uses local STT/TTS (no API keys). • Or send a version with exact Deepgram/OpenAI API calls wired in. Which would you prefer? Sources: Build a multilingual voice agent that automatically switches languages | Li... | Build Your First AI Voice Agent in Python: Complete Tutorial | LiveKit | Server startup modes | LiveKit Documentation | Build a multilingual voice agent that automatically switches languages | Li... | Agent session | LiveKit Documentation