Current I am facing LLM TTFT of 1200ms I want to reduce the latency by reducing prompt caching or stateful agent according to current setup for each query I have to pass whole prompt along with what user said and then what is previous take, how I can implement it? "@observe(_name_override_="general_llm_query") async def get_llm_response(_prompt_: str) -> str: """General-purpose LLM function that takes a prompt and returns response""" # VoiceEval: Log LLM query observation start if voiceeval_client: logger.info(f"🔬 VoiceEval: Starting LLM query observation - prompt length: {len(prompt)} chars") llm_instance = openai.LLM.with_azure( _azure_deployment_=AZURE_OPENAI_DEPLOYMENT, _azure_endpoint_=AZURE_OPENAI_ENDPOINT, _api_key_=AZURE_OPENAI_API_KEY, _api_version_=OPENAI_API_VERSION, model=AZURE_OPENAI_MODEL, temperature=0.1, timeout=60.0 ) ctx = ChatContext() ctx.add_message(role="user", content=prompt) stream = llm_instance.chat(_chat_ctx_=ctx) response_content = "" async for chunk in stream: if chunk.delta and chunk.delta.content: response_content += chunk.delta.content # VoiceEval: Log LLM response completion if voiceeval_client: logger.info(f"VoiceEval: LLM query observation completed - response length: {len(response_content)} chars") return response_content"

Thanks for reaching out, Anjani. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

Explain me how to implement stateful agents so that I reduce passing whole prompts

Thanks — I’ll explain how to implement stateful agents so you stop re-sending the entire prompt every query. I’ll keep this practical and focused on what you can change in your current async streaming flow. What “stateful agent” means here • A stateful agent keeps conversation state (chat history, system instructions, summaries, session metadata) in memory per user/session. • Each request only sends the new user message (and any small required context) to the LLM; the LLM call uses the session’s maintained chat context rather than rebuilding a full prompt string each time. • This reduces request payload size and model input processing, cutting TTFT and overall latency. High-level design • Create one AgentSession per conversation (user, room, or call). • Store chat history in the session as structured messages (role + content). • On each turn: ◦ Append the incoming user message to the session chat context. ◦ Optionally summarize or truncate older history to keep the context small. ◦ Call the LLM using the session’s chat context (not a reconstructed giant prompt). ◦ Append the LLM response to the session context. • Use preemptive generation (if available) to start LLM inference earlier for streaming scenarios. Concrete steps + example flow (async streaming style) 1) Initialize per-session state • When a conversation starts, create an AgentSession object that contains: ◦ session_id (user or room id) ◦

chat_ctx

— a list of messages or a ChatContext object ◦ configuration: LLM instance, preemptive_generation flag, truncation/summarization policy 2) Add messages incrementally (only send new content) • On new user input: ◦ session.chat_ctx.append({role: "user", content: new_user_message}) ◦ (Do NOT rebuild full prompt string with all prior messages.) 3) Call the LLM with the session chat context and stream tokens • Use the LLM chat API that accepts a chat context object (messages list) rather than a single prompt string. • Stream tokens back directly to your client as they arrive. 4) Persist or mutate session context with the response • When generation finishes (or during streaming), append the assistant reply to session.chat_ctx. • Keep the session in memory while the conversation is active. 5) Manage history size • Periodically trim or summarize old turns: ◦ If chat_ctx grows, either remove oldest messages, or replace chunks with a short summary message (role: system or assistant_summary). ◦ Summaries preserve context while keeping model input short. Code outline (pseudo/structured) — adapt to your stack:

# 1. Create or fetch a session
session = get_or_create_session(session_id)
# session.chat_ctx is a list like: [{"role":"system","content":"..."},
#                                   {"role":"user","content":"..."}]

# 2. On new user message
session.chat_ctx.append({"role": "user", "content": new_user_message})

# 3. Call LLM with the session chat context (streamed)
llm_instance = openai.LLM.with_azure(..., model=AZURE_OPENAI_MODEL, timeout=60)
stream = llm_instance.chat(chat_ctx=session.chat_ctx)  # chat_ctx not a single prompt

response_content = ""
async for chunk in stream:
    if chunk.delta and chunk.delta.content:
        response_content += chunk.delta.content
        # optionally forward tokens to the client

# 4. Append assistant reply to session context
session.chat_ctx.append({"role": "assistant", "content": response_content})

# 5. Trim or summarize if needed
if len(session.chat_ctx) > MAX_MESSAGES:
    summary = summarize_old_messages(session.chat_ctx[:N_OLD])
    # replace old messages with a short summary entry
    session.chat_ctx = [summary_message] + session.chat_ctx[N_OLD:]

Key implementation details to reduce TTFT • Use a chat API that accepts a messages/chat_ctx structure rather than a single combined prompt string. The model can process structured contexts more efficiently. • Keep most of the session data in memory — avoid re-serializing and re-sending everything over the network each turn. • Only include the minimal system/instruction messages needed for the model to behave correctly each turn. • Trim or summarize history proactively; older details usually can be represented as a short summary message. • Enable preemptive_generation if you have partial transcripts (reduces perceived latency by starting generation earlier). • Co-locate your service and the Azure OpenAI endpoint in the same region to reduce network RTT. When to persist session state vs. in-memory • In-memory (fastest): good while the conversation is active. • Persist to Redis or DB if you need fault tolerance, horizontal scaling, or recovery. When fetching persisted state, load into memory for the LLM call to avoid repeated DB reads per token. Metrics and validation • Measure TTFT before and after changes. Also track: ◦ average request payload size ◦ LLM TTFB and throughput ◦ token counts per request • Iteratively tune the summary/truncation strategy until latency and quality are balanced. Questions so I can give a tailored code sample 1) Do you already have a per-user or per-call session store (in memory, Redis, etc.)? 2) Is your input streaming partial ASR, or one full text message per turn? 3) Are you able to enable preemptive_generation in your environment? Tell me answers to those and I’ll provide a concrete code example that matches your current async streaming pattern and how to integrate summarization/truncation.