hi guys, we have seen an issue from the livekit ag...
# agents
b
hi guys, we have seen an issue from the livekit agents side that we dont think is the expected behaviour so wanted to check here, basically alot of times, the LLM when responds with the stream and then the user speaks immediately, the agent gets interrupted and the llm stream stops. but the tts has not started speaking yet. so the agent has not started speaking yet but in the chat context there is still a chunk of text even though it was not spoken by the agent. it seems like there is some discrepency between what the TTS speaks and what text goes into the chat context, any way to make sure only the text which was spoken goes into the chat context? has anyone done any implementation to solve this case and make sure whatever was said by the tts only gets appended to the chat context
👀 1
l
There's probably a few ways to do this, so others may chime in, but you can use
conversation_item_added
to modify the context and remove items where the LLM was interrupted, like this:
Copy code
import asyncio
import logging
import os
from pathlib import Path
from dotenv import load_dotenv
from livekit.agents import JobContext, WorkerOptions, cli, ConversationItemAddedEvent
from livekit.agents.voice import Agent, AgentSession
from livekit.agents.llm import ChatContext, ChatMessage
from livekit.plugins import silero

load_dotenv(dotenv_path=Path(__file__).parent.parent / '.env')

logger = logging.getLogger("listen-and-respond")
logger.setLevel(<http://logging.INFO|logging.INFO>)

class ListenAndRespondAgent(Agent):
    def __init__(self) -> None:
        super().__init__(
            instructions="""
                You are a helpful agent. When the user speaks, you listen and respond.
            """,
            stt="assemblyai/universal-streaming",
            llm="openai/gpt-4.1-mini",
            tts="cartesia/sonic-2:6f84f4b8-58a2-430c-8c79-688dad597532",
            vad=silero.VAD.load()
        )

    async def on_enter(self):
        self.session.generate_reply()

async def entrypoint(ctx: JobContext):
    await ctx.connect()

    session = AgentSession()
    agent = ListenAndRespondAgent()

    @session.on("conversation_item_added")  
    def on_conversation_item_added(item: ConversationItemAddedEvent):  
        # Check if this is an agent message that might have been interrupted  
        if (isinstance(item.item, ChatMessage) and   
            item.item.role == "assistant"):  
              
            if hasattr(item.item, 'interrupted') and item.item.interrupted:  
                # Remove the interrupted item from context  
                current_ctx = agent.chat_ctx  
                filtered_ctx = ChatContext()  
                  
                for ctx_item in current_ctx.items:  
                    # Keep all items except the interrupted one  
                    if ctx_item.id != item.item.id:  
                        filtered_ctx.items.append(ctx_item)  
                  
                # Update the agent's chat context (async operation in sync callback)
                async def update_context():
                    await agent.update_chat_ctx(filtered_ctx)  
                    <http://logging.info|logging.info>(f"Removed interrupted item {item.item.id} from context")
                
                asyncio.create_task(update_context())  
        else:
            # Print out the current chat context if not an interrupted agent message
            current_ctx = agent.chat_ctx
            print("Current chat context:")
            for ctx_item in current_ctx.items:
                print(f"- ID: {getattr(ctx_item, 'id', None)} | Role: {getattr(ctx_item, 'role', None)} | Content: {getattr(ctx_item, 'content', None)}")

    await session.start(
        agent=agent,
        room=ctx.room
    )

if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))
e
AFAIK there's a
TranscriptSynchronizer
component that estimates how much of the text has played out, so that when an interruption occurs only the text that was heard by the user is added to the chat history. What I've seen: • The estimation is good but not perfect, so it's usually off by a word or two. • If the agent gets interrupted before it even had a chance to start playing (as OP described), it always seems to add the first word to the chat history... This seems like a bug to me... • It doesn't work that well for languages that are very different from English, because of the word/syllable tokenization being used.