How can I have my multimodal agent generate a text summary of the interaction to push to front-end via an RPC call?

Thanks for reaching out, Matthew. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

I want to sanity-check one detail to avoid implementing based on pseudo-code. In your example you mention:

context.session.generate_reply(...)

and then “read the assistant message from chat context”. In LiveKit Agents (Python) with a RealtimeModel, is there an actual supported API on

RunContext

(or another object) that lets the agent programmatically request an additional LLM turn from the same session (i.e., outside the model deciding to call a tool), and then retrieve that generated text? If yes, could you point me to: The exact object/type name (e.g.,

AgentSession

,

RealtimeSession

, etc.) The exact method signature(s) 1. A minimal working snippet that runs inside a

@function_tool()

like

evaluation_completed

, generating a summary string and returning it (or sending via RPC) 2. Any caveats about calling the model from within a tool callback (reentrancy / deadlocks / ordering) If there is no supported “generate_reply” style API in the current Agents SDK, please confirm and recommend the intended pattern (e.g., issuing a separate OpenAI chat completion request for summarization, or capturing transcript events and summarizing server-side). For context: I already have RPC delivery working reliably; the remaining question is “what’s the supported way to generate the summary text at completion time?”

Thanks — I’m addressing your question about whether there’s a supported API in the LiveKit Agents (Python) runtime to programmatically request an extra LLM turn from inside a

@function_tool()

(i.e., call the model yourself at completion time), and how to do it safely. Short answer • Yes. Use the

AgentSession

accessible as

context.session

inside a

RunContext

and call

generate_reply(...)

to produce an additional LLM reply. You can await the returned handle and then read the generated text from the session chat context (

session.chat_ctx.messages

). This is the supported pattern for generating a summary from within a tool callback. Exact object / method • Object type: AgentSession (accessible as

context.session

in tool callbacks where

context

is a

RunContext

). • Method:

generate_reply(...)

Typical signature (conceptual):

# async
generate_reply(
    *,
    user_input: Optional[str] = None,
    instructions: Optional[str] = None,
    allow_interruptions: bool = True
) -> SpeechHandle

• `user_input`: if you want the generated content to be added as a user message in the chat log. • `instructions`: system-level guidance for the model (does not appear in the chat history). • Returns a handle you can await (often a

SpeechHandle

) which resolves after generation and any TTS playout (if applicable). Minimal working snippet This snippet runs inside a

@function_tool()

and returns the generated summary. It also shows how you would send it via RPC (you already said RPC delivery works).

from livekit.agents import function_tool, RunContext, get_job_context
import json

@function_tool()
async def evaluation_completed(self, context: RunContext, final_score: float, feedback_points: list[str]) -> str:
    # 1) Ask the session to generate a concise summary
    prompt_instructions = (
        f"Summarize the evaluation with score {final_score}/10. "
        f"Highlight these points: {', '.join(feedback_points)}. "
        "Keep it concise (2-3 sentences), professional, and user-facing."
    )

    handle = context.session.generate_reply(instructions=prompt_instructions)

    # 2) Wait for the generation (this also waits for any TTS playout if configured)
    await handle

    # 3) Read the last assistant message from the chat context
    chat_ctx = context.session.chat_ctx
    summary_text = "Summary could not be generated."
    if chat_ctx.messages and chat_ctx.messages[-1].role == "assistant":
        summary_text = chat_ctx.messages[-1].content

    # 4) Optionally: send via RPC to the front-end (your RPC path)
    job_ctx = get_job_context()
    if job_ctx is not None:
        remote = next(iter(job_ctx.room.remote_participants.values()), None)
        if remote is not None:
            try:
                await job_ctx.room.local_participant.perform_rpc(
                    destination_identity=remote.identity,
                    method="receive_summary",
                    payload=json.dumps({"summary": summary_text}),
                    response_timeout=10.0,
                )
            except Exception:
                # fall back to returning the summary even if RPC fails
                pass

    # Return the summary text (tool return values may be visible to the LLM)
    return summary_text

Caveats & best practices • Reentrancy / deadlock: ◦ Calling

generate_reply()

from a tool is supported and safe. ◦ Awaiting the returned handle will block the tool until generation (and any TTS) completes. That can be OK for short operations; avoid long blocking if you need concurrency. ◦ Avoid infinite recursion: don’t create flows where generate_reply triggers the same tool repeatedly (e.g., LLM calls tool → tool calls generate_reply → LLM triggers tool again). • Ordering: ◦ The newly generated assistant message will appear as the latest assistant entry in

session.chat_ctx.messages

. Read that entry to get the text. ◦ If you need to guarantee message ordering relative to other asynchronous events, consider tracking message counts or timestamps before/after calling

generate_reply

. • Text-only vs audio: ◦ Realtime/ multimodal setups often produce spoken output. If you want only text, retrieve the text from

chat_ctx.messages[-1]

. To suppress TTS playout, configure the pipeline accordingly or use modalities that produce text-only. • Tool return values: ◦ The return value of the tool is visible to the agent as the tool result. If you don’t want the LLM to react to the tool return, include instructions in your agent prompt to ignore administrative/tool-return text, or return a minimal acknowledgement. • Non-blocking alternatives: ◦ If you don’t want to block the tool, you can start generation without awaiting playback completion (e.g., create a background task) and later check the

SpeechHandle

or session chat context to retrieve the text asynchronously. • If you need fine-grained prompt control or separate billing/metrics, you can always do an external LLM call (OpenAI / other) inside the tool instead—but that duplicates model management and will be outside the LiveKit agent’s session/state. When to choose an external call instead • If you require a different model, separate rate-limiting, or tighter control over prompt engineering and you don’t care about keeping the summary in the agent’s session voice/persona, issuing a separate chat completion request is reasonable. • If you want pure server-side summarization of recorded transcripts without tying it into the live agent session, summarizing transcripts externally is also a valid pattern. Next steps I can provide • A copy-paste-ready, fully annotated example that suppresses TTS and only returns text. • A suggested RPC payload schema (timestamps, conversation id, summary_type). • An example that performs non-blocking generation and later polls for the result. Which of these would you like next? Sources: Agent speech and audio | LiveKit Documentation | Amazon Nova Sonic integration guide | LiveKit Documentation | Agents and handoffs | LiveKit Documentation | Collect recording consent with tasks | LiveKit Documentation | External data and RAG | LiveKit Documentation

To make sure I implement against the actual SDK behavior, can you please link to the official Python docs / source for

AgentSession.generate_reply()

and confirm the exact signature for LiveKit Agents v1.3.2 (or whatever version you’re assuming)? Specifically: Please provide the doc link or GitHub permalink to the method definition for

AgentSession.generate_reply()

(Python). Confirm parameter names and defaults (e.g.,

user_input

vs

user_question

,

instructions

,

allow_interruptions

, and whether there is an

add_to_chat_ctx

flag). I’m seeing discussion about chat context append controls, so I want to match the current API. In a tool callback (e.g., inside

@function_tool evaluation_completed

), does

await handle

wait for LLM generation only or also for TTS playout? If it includes playout, what’s the supported way to get the text summary without blocking on TTS? (I’m trying to keep completion reliable and fast.) Is the generated text always guaranteed to appear as the last assistant message in

session.chat_ctx.messages

, or should I track message count/IDs before/after generation to avoid race conditions? 1. Are there known issues/timeouts with

generate_reply()

+ RealtimeModel and the recommended mitigation (timeouts/retries/fallback)? Once I have the exact signature + guarantees, I can implement this cleanly.

I don't have the answer you're looking for. You could also try asking your question: • in one of the other Slack channels or • to https://deepwiki.com/livekit/livekit_composite which is trained on all LiveKit source code If you find the answer, please post it here to help others!