We’re seeing high latency with Gemini Realtime (2.5 Flash native audio & TTS) in LiveKit SIP calls. Is there a recommended low-latency model/config for real-time phone conversations? Any known best practices to reduce response delay?

Thanks for reaching out, harsha. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

yeah sure

Thanks — I’m responding about reducing latency for Gemini Realtime (2.5 Flash native audio & TTS) on LiveKit SIP calls and recommending a lower-latency setup and best practices. Recommended low-latency config • Use Gemini 2.0 Flash for real-time voice instead of 2.5 Flash native audio — it has lower latency for conversational scenarios. • If you need maximum control, run Gemini in TEXT-only mode and perform TTS separately with a fast low-latency TTS engine (example:

cartesia/sonic-3

). Quick config examples 1) Gemini 2.0 Flash (native realtime):

session = AgentSession(
  llm=google.realtime.RealtimeModel(
    model="gemini-2.0-flash",
    voice="Puck",
    temperature=0.8,
  ),
)

2) Text-only with separate fast TTS:

session = AgentSession(
  llm=google.realtime.RealtimeModel(modalities=["TEXT"]),
  tts="cartesia/sonic-3"
)

Best practices to reduce response delay • Network proximity: Co-locate your LiveKit agents and SIP gateway close to Google services and use LiveKit Cloud’s global mesh to reduce RTT. • Turn-detection: Use Gemini’s built-in turn detection/VAD rather than a separate external detector to avoid extra processing and round trips. • Target latency goals: aim for TTFT < 500 ms, TTFB < 300 ms, and end-to-end < ~1.5 s as practical benchmarks. • Optimize audio pipeline: minimize buffering, use low-latency audio codecs/settings on the SIP gateway, and keep frame sizes small. • Parallelize TTS: when using text-only + TTS, stream TTS audio as soon as tokens are available rather than waiting for full response generation. • Monitor metrics: track TTFT, TTFB, and end-to-end latency in production and correlate spikes with network, codec, or model changes. If you’d like, I can: • Help evaluate whether switching to Gemini 2.0 Flash or text-only + TTS fits your current architecture. • Suggest specific SIP/audio settings to reduce buffering and codec-related delay. Which would you like to do next? Sources: Gemini Live API plugin | LiveKit docs | examples/frontdesk | livekit/livekit_composite | DeepWiki

we Also tested: Gemini 2.5 Flash TTS Despite using Flash / Flash TTS, we’re still seeing noticeable response delays in live phone conversations.

Sorry you’re still seeing delays — I can help troubleshoot. A few quick questions to narrow the cause so I can give targeted recommendations: 1. Are you using Gemini 2.5 Flash in native audio mode (voice-generation inside Realtime) or text-only with separate TTS? 2. Where is your SIP gateway / PBX hosted relative to your LiveKit deployment and Google region? (same region, different region, on-prem?) 3. Which audio codec and packet/frame settings are you using on the SIP side (e.g., opus/20ms frames, PCMU/160-byte frames)? 4. What buffering sizes do you have for capture/playout and for any SIP jitter buffer? 5. Do you see delay primarily in Time To First Token (TTFT) or streaming audio after tokens begin? Any measured numbers for TTFT, TTFB, or end-to-end latency? 6. Is turn detection/VAD handled by Gemini or by an external component? 7. Are you running any intermediate processing (speech-to-text, profanity/ASR post-processing, sentiment analysis) that could add compute time? Send any answers and — if available — a short trace showing timestamps for: SIP audio receipt → STT start/finish (if used) → model request start → first token → TTS start → audio out. With that I’ll give prioritized, concrete steps (config tweaks, buffering/codecs, model choice, and monitoring checks).

Is there a recommended Gemini model or configuration specifically optimized for low-latency, real-time voice / SIP calls? Are there any known best practices (model choice, audio buffering, streaming settings, prompt style, tool usage) to reduce end-to-end latency?

Responding about recommended Gemini model/config for low-latency real-time voice (SIP) and concrete best practices to reduce end-to-end latency. Recommended model/config • Use Gemini 2.0 Flash for lowest latency in real-time voice; it generally outperforms 2.5 Flash for conversational TTFT/TTFB. • If you need full control over latency, run the LLM in TEXT-only mode and perform TTS with a fast low-latency engine (example:

cartesia/sonic-3

). This lets you stream TTS audio as soon as tokens are available. Practical best practices (priority-ordered) 1. Network / deployment ◦ Co-locate LiveKit, SIP gateway/PBX, and the Google realtime endpoint in the same region to minimize RTT. ◦ Use LiveKit Cloud’s global mesh or colocated infrastructure when possible. 1. Model / turn detection ◦ Prefer Gemini’s built-in turn detection/VAD to avoid extra round trips from external detectors. ◦ Use

gemini-2.0-flash

for lower TTFT; switch to text-only + TTS only if you need tighter TTS control. 1. Audio pipeline & codecs ◦ Use low-latency codecs and small frame sizes (e.g., Opus with 20ms frames) and minimize codec transcoding between SIP and LiveKit. ◦ Reduce capture/playout buffering and keep SIP jitter buffer conservative but not excessive. 1. Streaming & parallelism ◦ Stream partial responses and start TTS generation as soon as first tokens arrive; don’t wait for full response. ◦ Parallelize STT → model request → TTS where possible (e.g., start model generation while ASR finalization still in flight if partial ASR is usable). 1. Tuning buffering and timeouts ◦ Reduce internal audio buffering in LiveKit pipeline and SIP gateway. Avoid large batch windows for voice chunking. ◦ Tune VAD thresholds so voice segments are sent promptly without trimming needed speech. 1. Prompt and response design ◦ Keep prompts and system context minimal for latency-sensitive calls; move long context to short retrieval or summarization flows. ◦ Favor concise instructions that let the model emit tokens sooner (avoid heavy, multi-step reasoning inline). 1. Tooling and intermediate processing ◦ Minimize or offload synchronous post-processing (heavy NLP, sentiment, profanity checks) to async tasks unless required before TTS. ◦ If external tools are required, pipeline them asynchronously and stream intermediate outputs when safe. 1. Monitoring & targets ◦ Instrument and monitor TTFT, TTFB, and end-to-end latency. Target TTFT < 500 ms, TTFB < 300 ms, and end-to-end < ~1.5 s where practical. ◦ Correlate spikes with network, codec transcodes, or model selection. If you want, I can: • Recommend specific SIP codec/frame settings for your gateway. • Walk through how to switch to

gemini-2.0-flash

or text-only +

cartesia/sonic-3

TTS in your current AgentSession config. Which of those would help most right now?

• Walk through how to switch to

gemini-2.0-flash

or text-only +

cartesia/sonic-3

TTS in your current AgentSession config. Which of those would help most right now?

You asked for a walkthrough to switch your AgentSession to either

gemini-2.0-flash

or text-only +

cartesia/sonic-3

TTS. Below are concise, ready-to-use examples and the minimal steps to apply each option. Option A — Quick switch to gemini-2.0-flash (lower-latency, simple) • Use when you want easier setup and lower model latency without separate TTS auth. Python

from livekit.agents import AgentSession

session = AgentSession(
    stt="assemblyai/universal-streaming",   # or your STT
    llm="google/gemini-2.0-flash",
    tts="cartesia/sonic-3:9626c31c-bec5-4cca-baa8-f8ba9e84c8bc",
    # optional: vad=True, turn_detection="builtin"
)

Node.js / TypeScript

import { AgentSession } from "@livekit/agents";

const session = new AgentSession({
  stt: "assemblyai/universal-streaming",
  llm: "google/gemini-2.0-flash",
  tts: "cartesia/sonic-3:9626c31c-bec5-4cca-baa8-f8ba9e84c8bc",
});

Option B — Text-only Gemini Live API + cartesia/sonic-3 TTS (best control, lowest perceived latency) • Use when you need maximum latency control and can provide Google API key. 1) Install plugin and set API key: • Install Google plugin in your Agents environment (if not already). • Set env var:

export GOOGLE_API_KEY=your_api_key_here

2) Config examples Python

from livekit.agents import AgentSession
from livekit.plugins import google

session = AgentSession(
    llm=google.realtime.RealtimeModel(
        model="gemini-2.0-flash",
        modalities=["text"],           # disables built-in TTS/audio
        api_key="your_api_key_here",   # or rely on GOOGLE_API_KEY env var
    ),
    tts="cartesia/sonic-3:9626c31c-bec5-4cca-baa8-f8ba9e84c8bc",
    # optional: use_tts_aligned_transcript=True, preemptive_generation=True
)

Node.js / TypeScript

import * as google from "@livekit/agents-plugin-google";
import { AgentSession } from "@livekit/agents";

const session = new AgentSession({
  llm: new google.realtime.RealtimeModel({
    model: "gemini-2.0-flash",
    modalities: ["text"],
    apiKey: process.env.GOOGLE_API_KEY,
  }),
  tts: "cartesia/sonic-3:9626c31c-bec5-4cca-baa8-f8ba9e84c8bc",
});

Practical tips when using text-only + Cartesia TTS • Stream TTS as soon as partial text tokens are available; don’t wait for full response. • Set

use_tts_aligned_transcript=True

if you need better frontend sync of words ↔️ audio. • Consider

preemptive_generation=True

to start generating speculative tokens before turn end (careful with hallucination risk). Which should you try first? • If you want the fastest path with minimal changes: switch to

gemini-2.0-flash

(Option A). • If you need the absolute lowest perceived latency and fine control over voice output: use text-only +

cartesia/sonic-3

(Option B). Tell me which option you want to roll out and whether you want a patch/PR-ready snippet for your repo (Python or Node). Sources: HIPAA Eligible Products and Services | LiveKit | Google Gemini LLM | LiveKit Docs | Gemini Live API plugin | LiveKit docs | OpenAI Realtime API plugin guide | LiveKit docs | Text and transcriptions | LiveKit docs

import { DataSource } from "typeorm"; import { SynthesizerProvider, SynthesizerModel, TtsConfig, ConfigFieldType, } from "../../synthesizer/entities"; export class GeminiSeed { public async run(dataSource: DataSource): Promise<void> { const synthesizerProviderRepository = dataSource.getRepository(SynthesizerProvider); const synthesizerModelRepository = dataSource.getRepository(SynthesizerModel); const ttsConfigRepository = dataSource.getRepository(TtsConfig); // Check if Gemini provider already exists const existingProvider = await synthesizerProviderRepository.findOne({ where: { name: "Gemini" }, }); if (existingProvider) { console.log("Gemini provider already exists, skipping seed..."); return; } // Create Gemini provider const geminiProvider = synthesizerProviderRepository.create({ name: "Gemini", isActive: true, }); await synthesizerProviderRepository.save(geminiProvider); console.log("Gemini provider created successfully!"); // Create Gemini model const geminiModel = synthesizerModelRepository.create({ name: "gemini-2.5-flash-tts", synthesizerProvider: geminiProvider, isActive: true, }); await synthesizerModelRepository.save(geminiModel); console.log("Gemini model 'gemini-2.5-flash-tts' created successfully!"); // Voice options for select type const voiceOptions = [ { displayName: "Zephyr (Bright)", value: "Zephyr" }, { displayName: "Puck (Upbeat)", value: "Puck" }, { displayName: "Charon (Informative)", value: "Charon" }, { displayName: "Kore (Firm)", value: "Kore" }, { displayName: "Fenrir (Excitable)", value: "Fenrir" }, { displayName: "Leda (Youthful)", value: "Leda" }, { displayName: "Orus (Firm)", value: "Orus" }, { displayName: "Aoede (Breezy)", value: "Aoede" }, { displayName: "Callirrhoe (Easy-going)", value: "Callirrhoe" }, { displayName: "Autonoe (Bright)", value: "Autonoe" }, { displayName: "Enceladus (Breathy)", value: "Enceladus" }, { displayName: "Iapetus (Clear)", value: "Iapetus" }, { displayName: "Umbriel (Easy-going)", value: "Umbriel" }, { displayName: "Algieba (Smooth)", value: "Algieba" }, { displayName: "Despina (Smooth)", value: "Despina" }, { displayName: "Erinome (Clear)", value: "Erinome" }, { displayName: "Algenib (Gravelly)", value: "Algenib" }, { displayName: "Rasalgethi (Informative)", value: "Rasalgethi" }, { displayName: "Laomedeia (Upbeat)", value: "Laomedeia" }, { displayName: "Achernar (Soft)", value: "Achernar" }, { displayName: "Alnilam (Firm)", value: "Alnilam" }, { displayName: "Schedar (Even)", value: "Schedar" }, { displayName: "Gacrux (Mature)", value: "Gacrux" }, { displayName: "Pulcherrima (Forward)", value: "Pulcherrima" }, { displayName: "Achird (Friendly)", value: "Achird" }, { displayName: "Zubenelgenubi (Casual)", value: "Zubenelgenubi" }, { displayName: "Vindemiatrix (Gentle)", value: "Vindemiatrix" }, { displayName: "Sadachbia (Lively)", value: "Sadachbia" }, { displayName: "Sadaltager (Knowledgeable)", value: "Sadaltager" }, { displayName: "Sulafat (Warm)", value: "Sulafat" }, ]; // Create Voice Name config const voiceConfig = ttsConfigRepository.create({ label: "Voice Name", key: "voice_name", type: ConfigFieldType.SELECT, list: voiceOptions, defaultValue: "Zephyr", active: true, synthesizerProvider: geminiProvider, }); await ttsConfigRepository.save(voiceConfig); console.log("Voice Name config created successfully!"); // Create Instructions config const instructionsConfig = ttsConfigRepository.create({ label: "Instructions", key: "instructions", type: ConfigFieldType.STRING, list: null, defaultValue: "Speak in a friendly and engaging tone.", active: true, synthesizerProvider: geminiProvider, }); await ttsConfigRepository.save(instructionsConfig); console.log("Instructions config created successfully!"); console.log("Gemini TTS configs have been seeded successfully!"); console.log("Created 1 synthesizer provider (Gemini)"); console.log("Created 1 synthesizer model (gemini-2.5-flash-tts)"); console.log("Created 2 TTS configs (Voice Name, Instructions)"); } } is this corrrect ?

Thanks — I reviewed your seed file. It’s mostly correct and will create the provider, model, and two TTS configs, but a few small improvements and checks will make it safer and clearer. Issues / suggestions • Idempotency: you check the provider exists but not the model or configs. Add existence checks for the model and each config to avoid duplicates on re-run. • Key name escaping: use "voice_name" (no backslash) for the

key

field. In your pasted code it's

voicename

— ensure the actual string is

voice_name

. • Error handling: wrap DB operations in try/catch so failures log and don’t leave partial state. • Relations: you set

synthesizerProvider

on models/configs. Ensure the entity relation is configured to accept the provider object (not just provider id). • Consistent identifiers: include an explicit unique identifier or slug for the model if your app relies on it (e.g.,

modelKey: "gemini-2.5-flash-tts"

). • Active flags & defaults: confirm

isActive

and

active

flags match your app semantics (enabled vs. visible). Optional improvements • Create the model with metadata like

displayName

or

capabilities

if your UI needs it. • Use a transaction to ensure atomic seed execution. • Add created_at/updated_at if your entities use timestamps. If you want, I can: • Provide a patched seed file with idempotent checks, try/catch, and a transaction (Python or TypeScript). Which would you like?

could u pls check my agent code ?

I can — please paste the AgentSession (agent) code you want reviewed, including any setup/initialization and relevant env vars (STT, LLM, TTS, VAD/turn settings, and SIP/gateway hooks). If it’s long, share the key files or a minimal reproducible snippet that shows: • AgentSession creation and plugin imports • STT/LLM/TTS config values • Any VAD/turn_detection, buffering, or preemptive settings • How audio enters/exits the SIP pathway (jitter buffer, codec, frame size) Also tell me: • Which runtime (Python or Node) and LiveKit Agents version you’re running • Whether you want a focus on correctness, latency optimization, or both I’ll review and return a concise, annotated patch with recommended fixes and latency-focused changes.

pls review an dlet me know why there is latency we are facing in gemini model ?

I can help—to diagnose why Gemini is adding latency I need to see your AgentSession code and runtime details. Please paste the AgentSession (or minimal repro) including: • Language/runtime (Node or Python) and LiveKit Agents version • AgentSession creation and imports (LLM/STT/TTS plugin usage) • Full LLM config (model name, modalities, any plugin RealtimeModel usage, api keys) • TTS config (provider string, voice, any streaming/alignment flags) • STT config and whether you use partial vs. final transcripts • VAD/turn_detection settings (builtin or external) • Any preemptive_generation, streaming, or chunking options • SIP gateway / PBX details: hosting region vs LiveKit/Google region, codec (Opus/PCMU), frame size (e.g., 20ms), and jitter buffer settings • Where you observe delay (TTFT, TTFB, TTS start, or audio after first tokens) and any measured numbers or traces/timestamps If you prefer, paste these two things first and I’ll start with targeted checks: 1) your AgentSession code snippet, and 2) a short trace with timestamps for SIP audio received → STT start/finish → model request start → first token → TTS start → audio out. I’ll review and return a concise annotated patch and prioritized fixes focused on latency.

llm=google.realtime.RealtimeModel( model="gemini-2.5-flash-native-audio-preview-09-2025", voice="Leda", proactivity=True, # Enable Gemini to adapt its response style to the input expression and tone. enable_affective_dialog=True, # Enable Gemini to proactively decide not to respond if the content is not relevant. thinking_config=types.ThinkingConfig( include_thoughts=False, # Disable Gemini to include thoughts in the response. ), _gemini_tools=[types.GoogleSearch(), types.GoogleMaps()], ),

Thanks — I reviewed the Gemini config you pasted. I’ll point out what in that snippet likely adds latency and give concise, prioritized changes you can make now to reduce end-to-end delay. What in your snippet increases latency • model="gemini-2.5-flash-native-audio-preview-09-2025": native audio variants (2.5 Flash native audio) are higher-latency than Gemini 2.0 Flash or text-only flows with a fast external TTS. •

proactivity=True

and `enable_affective_dialog=True`: these features add internal processing and decision-making time. • `_gemini_tools=[GoogleSearch(), GoogleMaps()]`: calling tools introduces external API calls and waiting for results. •

thinking_config.include_thoughts

is off (good), but any extended thinking or internal reasoning modes can increase TTFT/TTFB. • Native audio + tool usage makes the pipeline end-to-end synchronous (ASR → LLM audio gen → TTS) which compounds delays. Immediate, high-impact fixes (apply these first) 1. Switch model to

gemini-2.0-flash

for lower model latency: ◦ llm.model="gemini-2.0-flash" 1. Disable proactivity, affective dialog, and tool calls for latency-sensitive calls: ◦

proactivity=False

,

enable_affective_dialog=False

, remove

_gemini_tools

unless essential. 1. If you need absolute lowest perceived latency, use text-only mode + fast external TTS (Cartesia Sonic-3): ◦ run Gemini with

modalities=["text"]

and set

tts="cartesia/sonic-3:..."

. ◦ Stream text tokens to TTS as soon as partial tokens arrive (don’t wait for full response). 1. Prefer built-in VAD/turn detection and small audio frames: ◦ Keep

turn_detection="builtin"

or equivalent so you avoid extra round trips. ◦ Use Opus 20ms frames (or similarly small frames) and minimize capture/playout buffering. Secondary tuning (after immediate fixes) • Disable or avoid synchronous tool usage; if required, call tools asynchronously and stream partial replies. • Set

preemptive_generation=True

only after testing (can help but may increase hallucination risk). • Reduce internal buffering and jitter buffer sizes in SIP gateway; avoid large batching windows. • Monitor TTFT and TTFB and correlate with model/tool calls to confirm improvements. Suggested config example (minimal changes) • Quick low-effort change (lower latency, same pipeline):

llm = google.realtime.RealtimeModel(
  model="gemini-2.0-flash",
  voice="Leda",
  proactivity=False,
  enable_affective_dialog=False,
  # remove _gemini_tools
)

• Best-performance change (text-only + Cartesia TTS):

llm = google.realtime.RealtimeModel(
  model="gemini-2.0-flash",
  modalities=["text"],   # disables built-in audio gen
  api_key="YOUR_GOOGLE_API_KEY",
)
# then set session.tts = "cartesia/sonic-3:..."

If you want, I can: • Produce a patch of your AgentSession with the minimal changes applied, or • Review your full AgentSession + SIP codec/jitter settings and a short timestamped trace to point to the exact bottleneck (TTFT vs TTS streaming). Which would you prefer?

how to enable gemini 3 flash ?

I can help — to enable Gemini 3 Flash I need to confirm your environment first. Quick questions: 1. Are you running Python or Node, and which LiveKit Agents version? 2. Do you plan to use Gemini 3 Flash in native audio mode or text-only (with external TTS)? 3. Do you have a Google API key available (needed for Gemini Live plugin usage)? If you tell me those three details I’ll give exact, copy‑paste code (Python or Node) plus any env vars, plugin install steps, and recommended low‑latency flags.

using python and version is 1.3.6

Thanks — you're on Python v1.3.6. A quick answer: “Gemini 3 Flash” is not listed in the LiveKit sources I can access. The closest supported Flash models are Gemini 2.0 and 2.5 (including

gemini-2.5-flash-native-audio-*

previews). You can enable those via LiveKit Inference or the Google plugin. If you want, I can give exact copy‑paste Python code to enable: • gemini-2.5-flash via LiveKit Inference (no Google key), or • a Gemini realtime model (native audio or text-only) via the Google plugin (requires

GOOGLE_API_KEY

), and include the best low-latency flags and recommended changes for SIP. Which of the two do you want me to provide code for? Sources: Gemini Live API plugin | LiveKit docs | Google Gemini LLM plugin guide | LiveKit Docs | Google Gemini LLM | LiveKit Docs | Agents Framework | livekit/livekit_composite | DeepWiki | HIPAA Eligible Products and Services | LiveKit