https://livekit.io logo
Join Slack
Powered by
Is it normal for Agent to start performing worse o...
# ask-ai
d
Is it normal for Agent to start performing worse over longer calls due to LLM context getting full? Is there any built in context compaction from Livekit? should we handle this ourselves? is there any best practices ?
t
Thanks for reaching out, Stewart. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Thanks — great questions. I’m responding to your three points: whether agent performance normally degrades on long calls, whether LiveKit does context compaction automatically, and what best practices you should follow. Short answers • Yes — agent performance can degrade as a call grows because the LLM chat context grows toward the model’s context window. • LiveKit does not provide automatic context compaction. • You should manage context yourself using the patterns LiveKit supports (structured session storage + summarized context) to keep agents responsive and predictable. Why this happens • LLMs have a finite context window. As the conversation history grows, you either hit the model’s limit (forcing truncation) or you send larger payloads which increase latency and cost. Both can reduce responsiveness and make the agent appear to perform worse. What to do (recommended patterns / best practices) 1. Use 
session.userdata
for primary memory ◦ Keep deterministic, structured items (user identity, account details, collected form data, ongoing task state) in 
session.userdata
. This data persists across agent exchanges and is compact and reliable. 1. Maintain a full canonical history in 
session.history
◦ Store the complete raw transcript there so you never lose information. This lets you reconstruct or analyze conversations later even when you compact active context. 1. Implement a custom context manager that combines summaries + recent messages ◦ Build a compact payload for the model that contains: ▪︎ A short summary of the session state (key facts, intents, decisions) ▪︎ A small window of the most recent N messages (e.g., last 3–10 turns) ◦ Use the summary to preserve long-term context and the recent messages for short-term continuity. 1. Design agents assuming context truncation can occur ◦ Make agents rely on `userdata`/summaries for important state rather than assuming the full chat history will always be present. ◦ Re-surface important facts explicitly when handing off between agents or when resuming after a long session. 1. Periodic compaction / rollup strategies ◦ Periodically run an automatic summarization step (e.g., every X minutes or after Y turns) that folds recent history into a concise state summary and moves older messages to 
session.history
. ◦ Keep summaries short, deterministic, and machine-parseable if downstream logic needs to read them. 1. Cost/latency tradeoffs ◦ Limit how much history you send per inference to reduce token cost and latency. ◦ Only include highly relevant context; offload everything else to 
userdata
or 
session.history
. Questions I can help with next • Do you want a concrete example of a summary schema to store in 
session.userdata
? • Would you like sample code for a context compaction workflow (summarize -> store -> trim active history)? • How long are your typical calls and what model/context window are you using? That helps tune the compaction frequency and window size. If you tell me which of the above you want (schema, code, or tuning guidance), I’ll provide a focused example you can adopt. Sources: Building multi-agent architectures with LiveKit agents | LiveKit
d
what size should input tokens start compacting at ? we're using Gemini 3.0 flash
t
Thanks — I’m responding to your question about when to start compacting input tokens for Gemini 3.0 Flash. Short answer • The LiveKit docs do not specify an exact token threshold for Gemini 3.0 Flash. You should pick a practical threshold based on the model’s context window, latency/cost targets, and your app’s tolerance for losing recent turns — then validate it with experiments. Practical recommendation to get started 1. Determine the model’s usable context window ◦ Check Gemini’s documentation for the maximum context window for the 3.0 Flash variant. Use a safety margin (e.g., 10–20% below the max) so you don’t hit hard truncation. 1. Choose an initial compaction threshold (two common approaches) ◦ Conservative: start compacting when active input reaches ~50–60% of the usable window. This prioritizes low latency and a big safety margin. ◦ Balanced: start compacting at ~70–80% of the usable window. This keeps more history while still avoiding last-minute truncation. 1. Tune based on measurements ◦ Measure latency, token cost, and any drop in agent quality as the session grows. ◦ If latency or cost rises too much, lower the threshold. If agent quality degrades after compaction, raise it. Recommended operational flow 1. Keep full transcript in 
session.history
. 2. Maintain important structured state in 
session.userdata
. 3. When active context reaches your threshold: ◦ Summarize older turns into a compact state summary. ◦ Move detailed turns to 
session.history
and only include the summary + recent N turns in the active prompt. Suggested starting numeric example (adjust after testing) • If Gemini 3.0 Flash has a 128k token window (example only — verify actual value): ◦ Usable window with safety margin: ~115k tokens ◦ Conservative compact at 57k–70k tokens (50–60%) ◦ Balanced compact at 80k–92k tokens (70–80%) Questions that will help me give a tighter recommendation • What is the actual context window size you’re using for Gemini 3.0 Flash (from Google docs)? • What’s an acceptable latency target and token-cost budget for your calls? • Typical call length in minutes or expected number of turns/tokens? If you share the model’s context window and a bit about latency/cost goals, I’ll propose a specific numeric threshold and a compacting cadence (e.g., every X turns or Y minutes) plus an example summary schema you can store in 
session.userdata
. Sources: Google Gemini LLM | LiveKit Documentation | Google Gemini LLM plugin guide | LiveKit Documentation | Google AI and LiveKit | LiveKit Documentation | Gemini Live API plugin | LiveKit Documentation
d
https://deepwiki.com/search/what-chat-context-compaction-a_66302dce-3e87-4449-aff6-7d92e6cebe40 Interesting, it seems like livekit implements their own chat compaction techniques already.
t
Thanks — good find, and I’ll address that observation directly. Acknowledgement • You found a source suggesting LiveKit implements chat compaction techniques. You're asking whether that’s accurate and how it changes the recommendations about managing context for long calls with Gemini 3.0 Flash. Direct answer • LiveKit does not provide an automatic, opaque “context compaction” feature that will autonomously summarize and trim chat context for you. Instead, LiveKit provides the primitives and recommended patterns (for example, 
session.userdata
for structured memory and 
session.history
for a canonical transcript) so you can implement context compaction in a controlled way. Why you might see references to compaction • Documentation, examples, or community posts may show reference implementations, patterns, or sample utilities that demonstrate compaction workflows (summarize -> store -> trim). These are guidance and sample code to help you implement compaction, not a built-in automatic service that LiveKit runs for every session by default. What this means for your app • You should still plan to manage compaction yourself (or adopt an implementation from examples/community): ◦ Store important structured state in 
session.userdata
. ◦ Keep the full transcript in 
session.history
. ◦ Build a summarization/compaction step that produces a compact session summary and trims active prompt history. ◦ Control when compaction runs (token threshold, time interval, or number of turns). Next steps I can help with • If you’d like, tell me the exact reference you saw (or paste the snippet). I can confirm whether it’s a sample implementation or something else and map it to the recommended patterns above. • I can also provide a concrete compaction plan (numeric token thresholds, summarization cadence, and a summary schema) tailored to your Gemini 3.0 Flash context window and latency/cost targets. Which would you prefer?
3 Views
Open in Slack
PreviousNext
Powered by