To minimize the prompt size sent to the LLM, you can use the

before_llm_cb

callback to truncate or adjust the chat context before it is sent to the LLM. This can help reduce the number of tokens processed, thereby decreasing latency. According to the LiveKit documentation, you can implement a function to truncate the context if it exceeds a certain length:

async def truncate_context(assistant: VoicePipelineAgent, chat_ctx: llm.ChatContext):
    if len(chat_ctx.messages) > 15:
        chat_ctx.messages = chat_ctx.messages[-15:]

agent = VoicePipelineAgent(
    ...
    before_llm_cb=truncate_context,
)

Would you like more detailed guidance on implementing this or any other aspect? Sources: - LiveKit VoicePipelineAgent Documentation 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

If you're already using

before_llm_cb

to manage the prompt size, consider further optimizing the context by focusing on the most relevant parts of the conversation. You can prioritize recent messages or those containing key information. Additionally, ensure that your callback efficiently processes and modifies the context to avoid adding unnecessary latency. Another approach is to use a more compact representation of the conversation context, such as summarizing previous exchanges or removing redundant information. This can help reduce the token count without losing essential context. Would you like more detailed guidance on any specific optimization technique? Sources: - LiveKit VoicePipelineAgent Documentation 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.