Hi eleven lab v3 - doesnt stream but we are using streaming adapter. but there is a latency

Thanks for reaching out, Sathwik. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

yes im using 11labs tts

Thanks — that’s helpful. I’ll guide you through troubleshooting ElevenLabs TTS streaming and ask a few targeted questions so I can give concrete steps to reduce latency. Quick clarifying questions • Which exact ElevenLabs model string are you using (for example

eleven_turbo_v2

,

eleven_flash_v2

, etc.)? • How are you sending TTS audio into LiveKit? ◦ Are you using ElevenLabs’ WebSocket streaming API client-side and piping chunks into LiveKit? ◦ Or are you generating audio server-side and publishing it to the room via LiveKit SDK? • What do you mean by “doesn’t stream”? Do you get a single final audio file after generation (no real-time chunks), or do you get chunks but they arrive slowly/with pauses? • How does the latency manifest? Examples help: ◦ Delay before first audio starts ◦ Frequent pauses during playback ◦ Long tail at the end before final chunk • Do you have any current settings you can share:

streamingLatency

,

chunkLengthSchedule

, sample rate, buffer size, VAD/StreamAdapter settings? • Can you share short logs or network observations (timestamps showing chunk arrival) if available? What I can do next (once you reply) • If you’re using ElevenLabs WebSocket streaming: I’ll suggest tuning

streamingLatency

and

chunkLengthSchedule

and confirm expected audio format and sample rate to avoid rebuffering. • If you’re using a StreamAdapter/VAD that buffers until end-of-speech: I’ll show how to reduce buffer thresholds or disable VAD for true streaming behavior. • If you share a minimal flow (where audio is generated and how it’s published to LiveKit), I’ll provide exact configuration changes and a short checklist to test after each change. Immediate, general checklist you can try now • Ensure the TTS client is using a streaming-enabled model name (if available) so chunks are sent as produced. • Reduce any adapter/VAD buffering so chunks forward to LiveKit sooner. • Match audio format/sample rate between ElevenLabs output and LiveKit input to avoid transcoding delays. • Lower

streamingLatency

and shorten

chunkLengthSchedule

if your client exposes those options. • Test with a very short text to see if latency is proportional to input length (helps isolate buffering vs network issue). Send me the model name and a couple of the config details above and I’ll give step-by-step tuning tailored to your setup.

11labs v3

Thanks — I know this is frustrating. I want to narrow this down so I can give exact tuning steps for ElevenLabs TTS streaming with your adapter. Quick clarifying questions • Can you confirm the exact ElevenLabs model identifier you’re using (for example

eleven_turbo_v3

,

eleven_flash_v3

, or another exact string)? “11labs v3” helps, but the exact model name matters for streaming support. • How is the TTS audio flowing into LiveKit? ◦ Are you connecting to ElevenLabs via WebSocket in the client and piping chunks directly into LiveKit? ◦ Or are you generating audio on a server and then publishing to the LiveKit room? • When you say “doesn’t stream,” which of these matches what you see? ◦ No chunks at all — only a final full audio file returned ◦ Chunks arrive but with a long delay before the first chunk ◦ Chunks arrive intermittently (pauses/jitter) during playback ◦ Other — please describe • Do you have any adapter/VAD buffering enabled (e.g., a StreamAdapter that waits for end-of-speech)? If so, what thresholds/timers are set? • What audio settings are in use (sample rate, PCM vs. Opus, mono/stereo)? Are you transcoding anywhere between ElevenLabs and LiveKit? • Can you share any short logs or timestamps showing when chunks are produced vs when they are published/played in LiveKit? What I’ll do after you reply • If you’re using WebSocket streaming: I’ll suggest exact parameters to lower latency (for example shorter chunk schedules and lower streamingLatency if available) and confirm the audio format to avoid rebuffering. • If a StreamAdapter/VAD is buffering: I’ll recommend which thresholds to reduce or how to bypass it for real-time streaming. • I can give a short checklist of step-by-step tests to isolate network, adapter, or format mismatches. If you can reply with the exact model name and one or two of the items above (flow type and how the latency shows up), I’ll give concrete, prioritized steps to reduce latency.

1. Exact ElevenLabs model: eleven_v3 for Punjabi. It’s defined as PUNJABI_TTS_MODEL = "eleven_v3" in utils/langauge_helper.py, and Punjabi TTS is created with that model in language_handler.py. 2. Audio flow into LiveKit: server-side, not client-side. The app creates an AgentSession in agent.py, and TTS is attached there. On language switch, BaseAgent._update_tts_for_language_context() replaces session._tts with the Punjabi TTS instance from language_handler.py. 3. Direct ElevenLabs WebSocket piping from client: no. I don’t see any client WebSocket code that streams ElevenLabs chunks straight into LiveKit. This is handled inside the server via the LiveKit ElevenLabs plugin. 4. “Doesn’t stream” behavior: the code suggests eleven_v3 is not being used with native streaming in this setup. For Punjabi, it is wrapped in StreamAdapter(tts=tts) because the comment says the LiveKit ElevenLabs plugin’s multi-stream-input WebSocket is not compatible with eleven_v3. So the closest match is: chunks are delayed/buffered rather than true low-latency incremental streaming. 5. Adapter / buffering enabled: yes, StreamAdapter is enabled for Punjabi in language_handler.py. Also, VAD exists, but that is input-side only. Non-English VAD uses min_silence_duration=1.0, min_speech_duration=0.15, activation_threshold=0.5 in base_agent.py and the agent classes. 6. Audio settings: STT is configured at sample_rate=48000 for Google Chirp Punjabi in language_handler.py. For ElevenLabs TTS on this branch, I don’t see an explicit sample rate / PCM / Opus override in app code. I also don’t see explicit manual transcoding; LiveKit/plugin internals appear to handle that. • Logs available: I don’t see chunk-level TTS timing logs in the repo. The closest existing signal is the track_published logging in agent.py, which records the published audio track MIME type in LiveKit, but not per-chunk generation timestamps.

Thanks — this is very helpful and gives me enough to make specific recommendations. Below I’ll address the key causes of the buffered/latency behavior you described (server-side StreamAdapter wrapping

eleven_v3

) and give prioritized, actionable steps you can take to move from buffered output toward lower-latency streaming. I’ll also include ways to gather diagnostic logs so we can verify improvements. Summary of the situation I’m responding to • You’re using

eleven_v3

(Punjabi) server-side and it’s wrapped in

StreamAdapter(tts=tts)

because the LiveKit ElevenLabs multi-stream-input WebSocket is not compatible with

eleven_v3

. • That wrapper plus any VAD/adapter buffering is causing delayed/chunked output rather than incremental low-latency streaming. • You do not currently have chunk-level timing logs to measure where delay occurs. Recommended plan — prioritized 1) Add chunk/timing logs to determine where latency happens (generation vs adapter vs publish) 2) Reduce or bypass adapter buffering for Punjabi where low latency is required 3) Tune audio format/processing to avoid transcoding delays 4) Test with short text and iterate Step 1 — add minimal diagnostics (high priority) • Add logging at three points with timestamps (ISO or epoch milliseconds) so you can see where delay is introduced: ◦ When TTS chunk is produced by ElevenLabs (inside the wrapper that receives a chunk) ◦ When the StreamAdapter forwards/publishes the chunk to LiveKit ◦ When LiveKit confirms track publication or when the server calls the LiveKit publish API for each chunk • If possible, log chunk sizes and a sequence index so you can correlate chunks across logs. • This will let you identify whether ElevenLabs is producing chunks slowly or whether the adapter/buffering/publishing is the bottleneck. Step 2 — reduce adapter buffering / change StreamAdapter behavior • The code comment already says

eleven_v3

is wrapped because the plugin’s multi-stream WebSocket is not compatible. That implies the StreamAdapter buffers to present a consistent stream. Try these changes: ◦ Lower buffered thresholds in StreamAdapter so it forwards partial output sooner. If there’s a parameter controlling minimum buffered duration or chunk count, reduce it aggressively for testing. ◦ If StreamAdapter groups small frames into bigger payloads, change it to forward each chunk as soon as it arrives. ◦ Temporarily disable VAD or any end-of-speech gating for Punjabi TTS output. You noted VAD is input-side only, but confirm no output-side silence-detection is delaying forwarding. • If your StreamAdapter implementation supports modes, add a “low-latency” mode that streams chunks immediately and a “safe” mode that does buffering; toggle it for tests. Step 3 — ensure audio format match to avoid re-transcoding delays • Verify the sample rate and encoding that ElevenLabs produces and what LiveKit expects. Mismatched formats cause server-side transcoding or buffering. ◦ If ElevenLabs TTS delivers PCM at an expected sample rate (e.g., 24k/48k), explicitly set or annotate that sample rate when publishing to LiveKit so the plugin avoids runtime conversions. ◦ If there’s any implicit format negotiation in your LiveKit plugin, make it explicit in the publish request. • If the codebase lacks explicit sample-rate settings for ElevenLabs output, add a field/metadata so LiveKit plugin can skip re-encoding. Step 4 — shorten chunkization parameters on the TTS side (if exposed) • Some ElevenLabs TTS streaming clients expose options like

streamingLatency

or

chunkLengthSchedule

. If your server-side ElevenLabs client supports similar options for

eleven_v3

, reduce: ◦

streamingLatency

(lower gives earlier but possibly more frequent chunks) ◦ chunk length (smaller chunks arrive earlier) • If

eleven_v3

client has no such options, confirm whether the wrapper is artificially chunking; adjust it to forward with smaller chunk sizes. Step 5 — test with minimal inputs and iterate • Use very short prompts (one or two short sentences) to validate first-chunk latency. • Measure time from request to first-chunk produced (TTS log) and time from first-chunk published to LiveKit playback. • If first-chunk latency is large, the issue is in TTS generation or network to ElevenLabs. If first-chunk is quick but playback is delayed, the adapter/publish path is the issue. Additional troubleshooting items • Network latency: If your server talks to ElevenLabs over the network, measure round-trip times. High RTTs increase first-chunk latency. • Concurrency/load: Under load, buffering behavior or queueing might increase delays; reproduce with single-client tests first. • Track the published audio MIME/type logging you mentioned — expand it to include the time you call publish so you can match it to chunk logs. If you want, I can: • Provide a short snippet you can drop into the StreamAdapter to log per-chunk timestamps and sequence numbers. • Suggest exact config names/values to lower buffering if you paste the StreamAdapter implementation or relevant parts of language_handler.py and the adapter code. Next step from you • Paste the StreamAdapter implementation or the functions in language_handler.py where the Punjabi TTS is wrapped and forwarded, or • Tell me whether you want the logging snippet first (I’ll give ready-to-paste code) so you can confirm which component adds the delay. I’ll stay with you through the steps — once we have timing logs we can pick the exact parameter to change to get stable low-latency streaming.