# LiveKit Bug Report - ElevenLabs STT Plugin Hallucination Issue Component: livekit-plugins-elevenlabs STT ## Summary After switching from

inference.STT()

(which has a known bug with ElevenLabs models) to the

elevenlabs.STT()

plugin as recommended by LiveKit team, we are experiencing severe hallucination issues where the STT generates completely fabricated text during silence. The hallucinated content is specific, coherent sentences that were never spoken (e.g., "The first two were the 2009 and 2010 World Cup, and the 2011 World Cup"). ## Context: Why We Switched to elevenlabs.STT() We were using

inference.STT("elevenlabs/...")

but encountered the bug where ElevenLabs models don't work correctly through LiveKit Inference. LiveKit team confirmed this and provided the workaround:

"inference.STT() has a known bug with ElevenLabs models. Use the elevenlabs.STT() plugin directly instead."

We implemented the recommended workaround:

python
_# Before (broken):_
from livekit.agents import inference
stt = inference.STT("elevenlabs/scribe_v2")

_# After (recommended workaround):_
from livekit.plugins import elevenlabs
stt = elevenlabs.STT(
    _use_realtime_=True,
    _language_code_="en",
    _server_vad_={
        "vad_silence_threshold_secs": 0.8,
        "vad_threshold": 0.4,
    },
)

--- ## Environment | Component | Version/Details | |-----------|-----------------| | livekit-agents | >=1.3.0 | | livekit-plugins-elevenlabs | >=1.0.0 | | livekit-plugins-silero | >=1.2.0 | | livekit-plugins-turn-detector | >=1.2.0 | | Python | 3.11 | | Deployment | Railway (Docker) | | ElevenLabs Model | Scribe V2 Realtime | ### Full Agent Configuration

python
from livekit.agents import AgentSession
from livekit.plugins import silero, elevenlabs
from livekit.plugins.turn_detector import MultilingualModel

_# Models_
STT_LANGUAGE = "en"
LLM_MODEL = "openai/gpt-4o-mini"
TTS_MODEL = "cartesia/sonic-3:f786b574-daa5-4673-aa0c-cbe3e8534c02"

_# VAD Configuration_
vad = silero.VAD.load(
    _min_speech_duration_=0.1,
    _min_silence_duration_=0.3,
    _activation_threshold_=0.4,
    _sample_rate_=16000,
)

_# Agent Session_
session = AgentSession(
    _stt_=elevenlabs.STT(
        _use_realtime_=True,
        _language_code_=STT_LANGUAGE,
        _server_vad_={
            "vad_silence_threshold_secs": 0.8,
            "vad_threshold": 0.4,
        },
    ),
    _llm_=LLM_MODEL,
    _tts_=TTS_MODEL,
    _vad_=vad,
    _turn_detection_=MultilingualModel(),
)

## Bug Description ### What Happens 1. User connects to voice room 2. Audio track is subscribed successfully 3. Before user speaks anything, STT outputs:

"The first two were the 2009 and 2010 World Cup, and the 2011 World Cup."

4. User speaks real words - these are transcribed correctly 5. During silence/pauses, STT outputs more hallucinations about World Cups ### Evidence This Is NOT Environmental Audio 1. Timing: First hallucination at 2.5s after connect, before any user speech 2. Content: Impossibly specific text about "2009 World Cup held in Brazil" - no such audio existed 3. Environment: Test PC had NO media playing, NO TV, NO YouTube, NO audio sources 4. Pattern: Same "World Cup" hallucinations across multiple test sessions 5. Reproducibility: Consistent hallucination pattern on silence --- ## Logs ### Session Timeline

14:31:33.282 - Session started, audio track subscribed
14:31:33.284 - GAIN_BOOST: +24dB gain applied
14:31:35.971 - STT: "The first two were the 2009 and 2010 World Cup..."  <-- HALLUCINATION
14:31:36.938 - STT: "Hello, can you hear me?"  <-- REAL (correct)
14:31:48.131 - STT: "uh how are you doing"  <-- REAL (correct)
14:31:57.488 - STT: "The first was the 2009 World Cup, which was held in Brazil."  <-- HALLUCINATION
14:32:00.463 - STT: "The first two were the 2009 and 2010 World Cup, and the 2011 and 2012 World Cup."  <-- HALLUCINATION

### Full Logs

14:31:33.281 [inf] Creating AgentSession with STT=elevenlabs.STT(language=en)
14:31:33.281 [inf] DEBUG track_flow: participant_connected - identity=web-tester-c70b1793
14:31:33.282 [inf] Starting agent session for room: voice-test-c70b1793
14:31:33.284 [inf] DEBUG track_flow: SUCCESS - Subscribed to AUDIO track from web-tester-c70b1793
14:31:33.284 [inf] GAIN_BOOST: frame 1 AFTER +24.0dB - min=0, max=0, rmsdb=-96.0

# HALLUCINATION - User hadn't spoken yet!
14:31:35.971 [inf] DEBUG pipeline: user_input_transcribed - STT result: 'transcript='The first two were the 2009 and 2010 World Cup, and the 2011 World Cup.' is_final=True'
14:31:36.352 [inf] User said: The first two were the 2009 and 2010 World Cup, and the 2011 World Cup.

# REAL USER SPEECH - Correctly transcribed
14:31:36.938 [inf] STT result: 'transcript='Hello, can you hear me?' is_final=True'
14:31:37.476 [inf] User said: Hello, can you hear me?

# QUOTA ERRORS
14:31:40.909 [err] ElevenLabs STT error [quota_exceeded]: Unknown error
14:31:40.989 [err] failed to process ElevenLabs STT message
14:31:40.995 [wrn] failed to recognize speech, retrying in 0.1s

# REAL USER SPEECH - Correctly transcribed
14:31:48.131 [inf] STT result: 'transcript='uh how are you doing' is_final=True'
14:31:48.189 [inf] User said: uh how are you doing

# MORE HALLUCINATIONS during silence
14:31:57.488 [inf] STT result: 'transcript='The first was the 2009 World Cup, which was held in Brazil.' is_final=False'
14:32:00.463 [inf] STT result: 'transcript='The first two were the 2009 and 2010 World Cup, and the 2011 and 2012 World Cup.' is_final=False'

## Additional Issue: quota_exceeded Errors During testing, we also received multiple

quota_exceeded

errors: ``` ElevenLabs STT error [quota_exceeded]: Unknown error livekit.agents._exceptions.APIConnectionError: quota_exceeded: Unknown error (body=None, retryable=True) ElevenLabs STT connection closed unexpectedly IMPORTANT: This

quota_exceeded

error CANNOT be from ElevenLabs side because: - This is a fresh creator account with 0 usage - We literally just created the ElevenLabs account for testing - There is no way we hit any quota limit with zero prior usage This suggests the error is either: 1. A bug in the

livekit-plugins-elevenlabs

plugin 2. Incorrect error mapping from ElevenLabs API responses 3. Some other issue being incorrectly reported as "quota_exceeded" The STT reconnects after these errors but continues hallucinating. ## Questions for LiveKit Team 1. Is this a known issue with the elevenlabs plugin? - Does the plugin have any built-in hallucination filtering? 2. Should server_vad prevent this? - We're using

server_vad

configuration but hallucinations still occur - Is ElevenLabs' server VAD not filtering silence properly? 3. Is there client-side filtering we should add? - Should we add our own silence detection before forwarding to STT? - Is there a minimum audio energy threshold we can check? 4. Recommended configuration? - Are there better VAD settings to prevent hallucinations? - Should we increase

vad_threshold

or

vad_silence_threshold_secs

? 5. inference.STT() timeline? - When will the

inference.STT()

bug with ElevenLabs be fixed? - Would

inference.STT()

have better hallucination handling? 6. quota_exceeded errors - Is this an ElevenLabs API issue or plugin issue? - We have a paid ElevenLabs account ## Impact 1. Agent responds to hallucinated text - AI thinks user said World Cup stuff 2. Conversation completely breaks - User asks "how are you", agent responds about World Cups 3. Production deployment blocked - Cannot ship voice agent with this behavior 4. Business impact - We're building B2B voice agent for Lithuanian market, blocked on STT issues ## Requested Guidance 1. Immediate workaround - Any configuration to reduce/eliminate hallucinations? 2. Alternative? - We can only use ElevenLabs STT, as LiveKit framework was exactly chosen because of ElevenLabs support 3. Plugin update - Is there a newer version of livekit-plugins-elevenlabs that addresses this? 4. Timeline - When will inference.STT() work with ElevenLabs models? ## Reproduction Steps 1. Create LiveKit room 2. Connect web client (browser) to room 3. Start agent with

elevenlabs.STT(use_realtime=True, language_code="en")

4. Do NOT speak - wait in silence 5. Observe STT output in logs - hallucinated text appears

How does the whole AgentSession setup look like?

yeah… looks like the STT decided to invent its own World Cup history. 😂 Sorry about that, definitely some hallucinations.

@quaint-ambulance-32873 No agent observability enabled - just Sentry for errors. Here's our AgentSession setup:

from livekit.agents import AgentSession
from livekit.plugins import silero, elevenlabs
from livekit.plugins.turn_detector.multilingual import MultilingualModel

# VAD Configuration (Silero - runs locally)
vad = silero.VAD.load(
    min_speech_duration=0.1,
    min_silence_duration=0.3,
    activation_threshold=0.4,
    sample_rate=16000,
)

# Agent Session
session = AgentSession(
    # STT: ElevenLabs plugin directly (inference.STT has bug - confirmed by LiveKit)
    # Plugin connects to <wss://api.elevenlabs.io/v1/speech-to-text/realtime>
    stt=elevenlabs.STT(
        use_realtime=True,
        language_code="en",
        server_vad={
            "vad_silence_threshold_secs": 0.8,
            "vad_threshold": 0.4,
        },
    ),
    
    # LLM: OpenAI via LiveKit Inference (string format)
    llm="openai/gpt-4o-mini",
    
    # TTS: Cartesia via LiveKit Inference (string format)
    tts="cartesia/sonic-3:f786b574-daa5-4673-aa0c-cbe3e8534c02",
    
    # Local VAD
    vad=vad,
    
    # Turn detection - ML model for end-of-turn detection
    turn_detection=MultilingualModel(),
    
    # Interruption handling
    allow_interruptions=True,
    min_interruption_duration=0.5,
)

# Session started with:
await session.start(
    room=ctx.room,
    participant=participant,
)
Environment:
livekit-agents>=1.3.0
livekit-plugins-elevenlabs>=1.0.0
livekit-plugins-silero>=1.2.0
livekit-plugins-turn-detector>=1.2.0
Python 3.11
Deployed on Railway (Docker)

I have full Railway logs showing the quota_exceeded errors with timestamps. What else do you need?

I think quota_exceeded mite be a problem on 11lab side, can you check on their dashboard?

@quaint-ambulance-32873 You are correct indeed. wanted to share a bug discovery that might affect LiveKit Inference with ElevenLabs STT. The Problem • When using

elevenlabs.STT()

plugin with an unrestricted ElevenLabs API key, we experienced:`quota_exceeded` errors on a brand new account with zero usage • STT hallucinations (fabricated text like "2009 World Cup held in Brazil" during silence) • Constant reconnection loops The Fix • Switching from an unrestricted API key to an STT-only restricted API key in ElevenLabs dashboard immediately fixed all issues:Zero

quota_exceeded

errors • No hallucinations • Clean transcription of real speech Suggestion for LiveKit Inference If LiveKit Inference uses ElevenLabs API keys internally for the

inference.STT("elevenlabs/scribe_v2")

endpoint, you may want to verify those keys are properly scoped/restricted to STT only. The unrestricted key seems to trigger some kind of rate limit or internal error that ElevenLabs incorrectly reports as

quota_exceeded

.Our Setup •

livekit-agents>=1.3.0

•

livekit-plugins-elevenlabs>=1.0.0

• Using

elevenlabs.STT()

plugin directly (not inference.STT, since that has a known bug) Logs Before (unrestricted key)

ElevenLabs STT error [quota_exceeded]: Unknown error
livekit.agents._exceptions.APIConnectionError: quota_exceeded: Unknown error (body=None, retryable=True)

Logs After (STT-restricted key)

STT result: transcript='hello can you hear me' is_final=True
STT result: transcript='do you hear me clearly' is_final=True

Zero errors, clean transcription. Hope this helps! https://livekit-users.slack.com/archives/C09J2AYGN0M/p1765814105590239

stt inference 11lab is also fixed

FWIW, I've seen ElevenLabs Scribe V1 used directly via their API also hallucinate entire sentences. Give it an audio track of one side of a conversation, and it'll just make it up the other side of the conversation. Unusable IMO.