https://livekit.io logo
Join Slack
Powered by
is there any recommended way to truncate the conve...
# ask-ai
a
is there any recommended way to truncate the conversation history before we generate the llm response?
t
Thanks for reaching out, Adian. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
To help you manage your conversation history before generating an LLM response, here are a few friendly methods you can try: • Using `before_llm_cb`: You can create a callback function that trims the message list according to what works best for you, like keeping just the last 15 messages. Here’s an example to guide you:
Copy code
async def truncate_context(assistant: VoicePipelineAgent, chat_ctx: llm.ChatContext):
    if len(chat_ctx.messages) > 15:
        chat_ctx.messages = chat_ctx.messages[-15:]

agent = VoicePipelineAgent(
    ...
    before_llm_cb=truncate_context,
)
• Using `ChatContext.truncate`: This handy method automatically removes older messages while keeping the first system message intact, so your conversation stays clear and coherent. Here’s how you can use it:
Copy code
chat_ctx.truncate(max_items=15)
• Custom Truncation Logic: If you have more specific needs, you might want to create a helper function like 
_truncate_chat_ctx
that filters messages based on criteria important to you, such as keeping system messages or function calls. These approaches are designed to help you manage token usage effectively and make sure the LLM gets a clear, relevant context to work with. If you have any questions or want to explore these options further, I’m here to help! Sources: VoicePipelineAgent | LiveKit Docs | livekit-agents/livekit/agents/llm/chat_context.py | complex-agents/personal_shopper/README.md
a
Copy code
agent = VoicePipelineAgent(
    ...
    before_llm_cb=truncate_context,
)
can I apply this on the Session, i.e. I currently have: session = AgentSession[SessionInfo]( stt=stt.FallbackAdapter( [ cartesia.STT(model="ink-whisper"), stt.StreamAdapter( stt=openai.STT(model="gpt-4o-mini-transcribe"), vad=ctx.proc.userdata["vad"], ), ] ), llm=llm.FallbackAdapter( [openai.LLM(model="gpt-4.1"), openai.LLM(model="gpt-4o")] ), tts=tts.FallbackAdapter( [ cartesia.TTS(model="sonic-2", voice=voice), openai.TTS( model="tts-1", voice="ash", ), ] ), _turn_detection_=EnglishModel(), vad=ctx.proc.userdata["vad"], userdata=session_data, )
t
Hi there! I wanted to share some helpful info about the 
before_llm_cb
callback. It can’t be directly applied to 
AgentSession
anymore since it was part of the older Agents v0.x framework. But don’t worry! In the newer Agents 1.0+ setup, you can achieve similar results using the 
llm_node
mechanism. For example, you might consider overriding the 
llm_node
method in a custom agent class. This way, you can tweak the chat context or clean up the LLM output before it moves on to the TTS stage. Here’s a simple example to get you started:
Copy code
class CustomAgent(Agent):
    async def llm_node(self, chat_ctx, tools, model_settings=None):
        async def process_stream():
            async with self.llm.chat(chat_ctx=chat_ctx, tools=tools) as stream:
                async for chunk in stream:
                    # Modify the chunk content if needed
                    yield chunk
        return process_stream()
Then, you can use this custom agent with your 
AgentSession
like this:
Copy code
session = AgentSession()
await session.start(agent=CustomAgent(), room=ctx.room)
This approach gives you the flexibility to manage the chat context or LLM output while keeping everything compatible with the streaming pipeline. If you’d like, I’m here to help you with the implementation or answer any questions you have! Sources: Modifying LLM output before TTS | LiveKit Docs | Agents 0.x migration guide | LiveKit Docs
a
2025-09-18 141008,480 - ERROR livekit.agents - Error in _llm_inference_task Traceback (most recent call last): File "/Users/adianliusie/Developer/APTA/PrepPartner/prep_partner_backend/livekit/.venv/lib/python3.12/site-packages/livekit/agents/utils/log.py", line 16, in async_fn_logs return await fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/adianliusie/Developer/APTA/PrepPartner/prep_partner_backend/livekit/.venv/lib/python3.12/site-packages/opentelemetry/util/_decorator.py", line 71, in async_wrapper return await func(*args, **kwargs) # type: ignore ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/adianliusie/Developer/APTA/PrepPartner/prep_partner_backend/livekit/.venv/lib/python3.12/site-packages/livekit/agents/voice/generation.py", line 124, in _llm_inference_task async for chunk in llm_node: File "/Users/adianliusie/Developer/APTA/PrepPartner/prep_partner_backend/livekit/src/agents/base_agent.py", line 44, in process_stream async with self.llm.chat(chat_ctx=chat_ctx, tools=tools) as stream: ^^^^^^^^^^^^^ AttributeError: 'NotGiven' object has no attribute 'chat' {"room": "test_room_a7b85828-e197-4bd4-b255-81a63432aac4", "pid": 13212, "job_id": "AJ_tT32pQuuz8wj"}
t
It looks like you're encountering the error 
AttributeError: 'NotGiven' object has no attribute 'chat'
because the LLM instance (
self.llm
) wasn't properly initialized and is still set to the sentinel value 
NOT_GIVEN
. This usually happens when the LLM isn’t provided during the agent's initialization. Here’s how you can fix this: 1. Make sure you explicitly pass the LLM when you create your agent instance. For example:
Copy code
agent = VoiceAgent(
    llm=openai.LLM(model="gpt-4o", api_key="your-api-key"),
    # ... other parameters
)
1. Add a check in your overridden 
llm_node
method to confirm that 
self.llm
is valid before calling `chat()`:
Copy code
def llm_node(self, chat_ctx, tools, model_settings):
    if self.llm == NOT_GIVEN:
        raise ValueError("LLM must be provided to use custom llm_node")
    # Proceed with chat logic
1. Double-check that your LLM plugin (like OpenAI) is properly initialized with all necessary parameters, including your API keys. 2. Also, ensure your agent is started correctly within a session so it has the right activity context. If you have any questions or need a hand with any of these steps, please don’t hesitate to reach out—I’m here to help! Sources: livekit-agents/livekit/agents/voice/generation.py
Open in Slack
PreviousNext
Powered by