The user voice input messages are breaking up very quickly. This is not because of partial transcriptions. I checked and matched the ID of the transcriptions coming in. The logic for transcriptions with same IDs is working fine. Each of these is given a different ID causing speech to be broken up. I further tried implementing the logic to keep updating the same message unless isFinal is true but it seems isFinal is true very quickly, even for momentary silences.

endpointing_ms

does not help cause i think it is for complete turn end so agent can respond and not for within a user’s speech.

stt=stt.FallbackAdapter([
    deepgram.STT(model=config.survey_data.get('deepgram_stt_model', ''),endpointing_ms=50),
]),

Any other options to create longer user voice messages? (edited)

Thanks for reaching out, vnandan. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

didn’t work.

def create_agent_session(config):
    """Create the agent session with plugins"""
    return AgentSession(
        turn_detection=MultilingualModel(),
        min_endpointing_delay=5.0,
        allow_interruptions=True,
        vad=silero.VAD.load(),
        stt=stt.FallbackAdapter([
            deepgram.STT(model=config.survey_data.get('deepgram_stt_model', ''),endpointing_ms=5000),
        ]),
        llm=llm.FallbackAdapter([
            anthropic.LLM(model='claude-sonnet-4-20250514'),
            openai.LLM(model=config.survey_data.get('openai_llm_model', '')),
        ]),
        tts=tts.FallbackAdapter([
            elevenlabs.TTS(voice_id="UEKYgullGqaF0keqT8Bu", model="eleven_multilingual_v2"),
        ]),
    )

Hey there! If you’re noticing that user voice input is breaking up, especially with

endpointing_ms

set to 5000, no worries — here are some tweaks you can try out: 1. Lower Deepgram's

endpointing_ms

Value • Try setting

endpointing_ms

to a smaller number, like 100ms, so it recognizes when speech is done a bit faster:

stt = deepgram.STT(
    endpointing_ms=100  # Lower value for quicker response
)

2. Adjust LiveKit's Turn Detection Settings • Instead of depending only on Deepgram’s endpointing, you can use LiveKit’s turn detection with some thoughtful delay settings:

session = AgentSession(
    min_endpointing_delay=0.5,  # 500ms minimum delay
    max_endpointing_delay=2.0   # Shorter than the default 5.0s
)

3. Turn Off Deepgram Endpointing • If you want, you can completely disable Deepgram’s endpointing and just rely on LiveKit’s system:

stt = deepgram.STT(
    endpointing_ms=0  # Disable Deepgram endpointing
)

4. Try a Multi-Layered Approach • Mix a reasonable

endpointing_ms

setting with LiveKit’s turn detection for smoother control:

stt = deepgram.STT(
    endpointing_ms=25  # Default value
)

agent = VoicePipelineAgent(
    stt=stt,
    min_endpointing_delay=0.5,
    max_endpointing_delay=2.0
)

Hopefully, these ideas help cut down on those annoying breaks in voice input and keep your conversations feeling natural. If you want to tweak things more or have questions, just let me know — I’m here to help! Sources: livekit-agents/livekit/agents/voice/audio_recognition.py | Deepgram STT integration guide | LiveKit Docs | VoicePipelineAgent | LiveKit Docs

give the logic behind each suggestion you made.

Here’s a quick rundown of the reasoning behind each suggestion to help with the fragmented user voice input issue: 1. Reduce Deepgram's

endpointing_ms

Value •

endpointing_ms

sets how long (in milliseconds) the system waits during silence before deciding the speech has ended. If this is set too high (like 5000ms), it causes long delays and can fragment the input because the system waits too long to wrap up speech segments. Bringing this down (say, to 100ms) helps the system finalize speech faster, reducing fragmentation. 2. Configure LiveKit's Turn Detection Parameters • LiveKit’s

min_endpointing_delay

and

max_endpointing_delay

control how long it waits before recognizing the end of a turn. Setting

min_endpointing_delay

to 0.5 seconds and

max_endpointing_delay

to 2.0 seconds gives room for natural pauses without cutting the turn off too soon. It’s a nice balance between being responsive and accurate. 3. Disable Deepgram Endpointing • Turning off Deepgram’s endpointing (

endpointing_ms=0

) means LiveKit takes over detecting when speech ends. This works well when LiveKit’s system is better suited for your app’s needs, especially if you’re dealing with longer or more complex speech patterns. 4. Use a Multi-Layer Approach • By combining a moderate

endpointing_ms

value (like 25ms) with LiveKit’s turn detection settings, you get the best of both worlds. Deepgram handles the fine details of speech completion, while LiveKit manages the bigger picture of turn-taking. This helps speech get finalized quickly without unnecessary breaks or delays. Each of these ideas targets different parts of the problem—cutting down delays, handling pauses smoothly, and syncing Deepgram with LiveKit just right. If you want me to go over anything in more detail or tweak these suggestions, just let me know—I’m here to help!