https://livekit.io logo
Join Slack
Powered by
how livekit handles context of ongoing agent sessi...
# ask-ai
a
how livekit handles context of ongoing agent sessions?
✅ 1
t
Thanks for reaching out, Aditya. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — I’ll explain how LiveKit manages the context of ongoing agent sessions and what you can do with it. Overview • LiveKit uses a ChatContext mechanism to store ordered messages and events for an agent/task; each agent has 
self.chat_ctx
. • The full conversation across agents is available on 
session.history
. How context is updated during a session • Lifecycle hooks (e.g., 
on_user_turn_completed
) let you inject data (RAG results, tool outputs) into the chat context before the model responds. • Changes made in 
on_user_turn_completed
are temporary for that turn unless you call 
update_chat_ctx
to persist them. Context operations you can use • Create and modify contexts with 
ChatContext()
and 
add_message()
. • Copy contexts with 
copy()
and use filters (for example, exclude system instructions or function calls). • Truncate history with 
truncate()
while preserving system instructions. • Merge contexts from parallel tasks with 
merge()
. Agent handoffs and continuity • For agent-to-agent handoffs, copy the chat context (commonly 
copy(exclude_instructions=True)
) to preserve user/assistant messages but avoid carrying over the prior agent’s system prompts. • Summarization can be used before handoffs to reduce token usage. Per-reply and override behavior • 
generate_reply()
accepts a 
chat_ctx
parameter to override the session context for a single response (useful for conditional instructions or injected context). External data & session state • You can initialize context with external user or job metadata before a session starts and enrich context during the session via tools or RAG lookups. • Typed session state (e.g., 
userdata
) can store persistent session values accessible across tools and agents. If you’d like, I can: • Show code snippets for creating/updating 
ChatContext
. • Walk through a typical handoff flow with example calls. • Suggest strategies for long-running sessions (summarization, state vs. history). Which would you prefer? Sources: Chat context | LiveKit Documentation | Pipeline nodes and hooks | LiveKit Documentation | External data and RAG | LiveKit Documentation | Agents and handoffs | LiveKit Documentation | Agent speech and audio | LiveKit Documentation, with interactive elements
a
during long ocnverastions where the model context window might hit, how will agents manage this?
t
Thanks — I’ll explain how LiveKit agents handle long conversations when the model’s context window is at risk of being exceeded. How agents manage context-window limits • Truncation: Use 
ChatContext.truncate()
to remove older, less relevant items while preserving system instructions. This keeps the most recent, relevant turns in the context. • Summarization: Summarize older parts of the conversation (either automatically or via a pipeline node) and store the summary as a compact message in the chat context so you preserve meaning with far fewer tokens. • Persistent typed state: Store structured session data in 
userdata
(or a similar typed state) for things like user profile, preferences, or task state—these values are available to agents/tools without consuming prompt tokens. • Selective copying/filters: Use 
copy()
with filters (e.g., exclude instructions, function calls) to create a slimmer context for downstream agents or tools. • Merge & replace strategies: Merge parallel-task contexts when needed, or replace the per-reply context via 
generate_reply(chat_ctx=...)
to control exactly what the model sees for a single response. • Handoff best practice: Before agent handoffs, summarize or copy with 
exclude_instructions=True
so the next agent receives conversation-relevant content without redundant system prompts. When to apply each approach • Use truncation for straightforward sliding-window behavior (keep recent turns). • Use summarization when conversation history contains important long-term info you need to retain but don’t need verbatim. • Use 
userdata
for persistent, structured state that must survive many turns without token cost. • Combine strategies (summarize older history, keep recent turns verbatim, store user state in 
userdata
) for long-running sessions. If you want, I can show a short code example for truncation + summarization or outline a handoff flow—which would be most useful? , with interactive elements
Open in Slack
PreviousNext
Powered by