How can I instruct my multimodal realtime agent to trigger a defined rpc event when the user's interaction is complete?

Thanks for reaching out, Matthew. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

I need this to trigger only when the full interaction is complete. I have the function and I have instructed the AI to invoke it when the interaction is over but it is not triggering properly.

I'd love to help you get this sorted out! Could you share a bit more about your setup? Specifically: • How are you defining the "full interaction" completion? Are you using a particular event or condition for this? • Which method or function are you using to trigger the RPC event? • Have you noticed any errors or logs that might explain why the RPC isn't triggering? With these details, we'll be in a great position to figure out what's going on and find the best solution together.

A full interaction condition is when the user successfully completes all questions in the bot's knowledge bank The current function tool is: @function_tool() async def evaluation_completed(self, context: RunContext) -> str: with sentry_sdk.start_span(op="rpc.call", description="evaluationCompleted"): try: job_ctx = get_job_context() room = job_ctx.room if job_ctx else None if room is None: raise RuntimeError("No room on job context") rpc_target = None try: room_meta = extract_metadata_as_json(room) or {} rpc_target = room_meta.get("rpcTargetIdentity") except Exception: pass if not rpc_target: remotes = list(room.remote_participants) if room.remote_participants else [] if not remotes: raise RuntimeError("No remote participants to deliver RPC to") rpc_target = remotes[0].identity payload = json.dumps({"success": True}) resp = await room.local_participant.perform_rpc( destination_identity=rpc_target, method="evaluationCompleted", payload=payload, response_timeout=5.0, ) logger.info(f"[tool-RPC] evaluationCompleted -> {rpc_target} ; response={resp}") return "OK" except Exception as e: sentry_sdk.capture_exception(e) return f"RPC failed: {e}" No errors or logs have been noticed. I have validated that rpc fires successfully when explicitly told to in the script itself.

Let's walk through some steps together to troubleshoot why the

evaluation_completed

function tool might not be triggering the RPC event properly. I’m here to help you get this sorted out! 1. Function Tool Definition: First, please double-check that the function is correctly defined with the

@function_tool

decorator and that it accepts

RunContext

as an argument. This setup is essential for the function to run smoothly. 2. Room and Participant Initialization: It’s important to make sure that both the

room

and

local_participant

are properly initialized. If either is missing, the RPC call won’t work as expected. 3. RPC Call Logic: Let’s confirm that the function’s logic actually reaches the RPC call without exiting early. Sometimes, certain conditions might prevent the RPC from triggering, so reviewing those could help. 4. Error Handling: Using try-except blocks to catch and log any exceptions during the RPC call can be really helpful. This way, we can spot any transient issues or invalid states that might be causing trouble. 5. Tool Execution: Check if the tool is being selected and executed by the LLM. You can verify this by looking at

function_tools_executed

events or running test assertions. 6. Custom Implementation: Just a heads-up that

evaluation_completed

isn’t a built-in function in LiveKit. You’ll want to make sure it’s explicitly defined and registered as a custom tool. If you try these steps and still face issues, please don’t hesitate to reach out. I’m more than happy to help you dig deeper and find a solution together. Sources: Tool definition and use | LiveKit Docs | complex-agents/drive-thru/tools/management_tools.py | Events and error handling | LiveKit Docs | Testing and evaluation | LiveKit Docs | tests/test_evals.py

For function_tools_executed, how do I view this?

I'm happy to help you with viewing the

function_tools_executed

event! You can listen for this event on the

AgentSession

object. It gets emitted right after all the function tools have run for a specific user input, which is pretty handy. Here’s a simple example in Python to get you started:

from livekit.agents import FunctionToolsExecutedEvent

@session.on("function_tools_executed")
def on_function_tools_executed(event: FunctionToolsExecutedEvent):
    # Access the function calls and their outputs
    for function_call, function_output in event.zipped():
        print(f"Function {function_call.name} was executed")
        if function_output:
            print(f"Output: {function_output.output}")

And if you’re working with JavaScript/Node.js, here’s how you can do it:

import { AgentSessionEventTypes } from '@livekit/agents';

session.on(AgentSessionEventTypes.FunctionToolsExecuted, (event) => {
    console.log('Function tools executed:', event.functionCalls);
    event.functionCalls.forEach((call, index) => {
        console.log(`Function ${call.name} executed with arguments:`, call.arguments);
        if (event.functionCallOutputs[index]) {
            console.log('Output:', event.functionCallOutputs[index].output);
        }
    });
});

This event gives you all the details about the executed function calls and their outputs. If you have any questions or need a hand with anything else, just let me know—I’m here to help! Sources: Events and error handling | LiveKit Docs | livekit-agents/livekit/agents/voice/agent_activity.py

You mentioned that my evaluation completed custom tool is not a built-in function. Are there built-in functions for ai-initiated rpc calls that would be better suited to my use case?

Thanks for reaching out! While LiveKit doesn’t come with built-in AI-specific RPC functions, it does offer a really flexible RPC mechanism that lets you create and register your own custom methods for AI agents to call. This way, you can smoothly connect AI-driven logic with actions on the client side. Here’s a simple way to get started: 1. First, define and register an RPC method on a participant using

registerRpcMethod

along with a handler function. 2. Then, from the AI agent’s side, you can make the call using

performRpc

, where you specify the destination participant, method name, and any payload you need. If you’d like, I’m more than happy to help you create a custom RPC method that fits your specific needs—just let me know! Sources: Remote method calls (RPC) | LiveKit Docs | realtime/openai-realtime-tools.py | Function calling with Voice Agents | LiveKit Docs

I need to be able to signal the front end evaluationcompleted with success: true, which would be initiated when the user has correctly answered a number of questions given by the ai agent. What is the correct approach?

To let the frontend know with

evaluationCompleted

and

success: true

when the user answers questions correctly, you can use LiveKit's RPC mechanism. Here’s a friendly way to approach this: 1. Agent-Side: You’ll want to use

perform_rpc

to send a structured JSON payload to the frontend right after evaluating the user’s answers. For example:

evaluation_payload = {
    "action": "evaluationCompleted",
    "success": True,
    "message": "All answers correct!"
}

json_payload = json.dumps(evaluation_payload)
await room.local_participant.perform_rpc(
    destination_identity=participant.identity,
    method="client.evaluation",
    payload=json_payload
)

1. Frontend-Side: On the frontend, you can register an RPC handler to handle the evaluation result like this: Sources: avatars/tavus/README.md

I already have an RPC handler. Once the payload is constructed, how do I make the agent invoke it at the correct time?

Let's make sure your agent calls the RPC at just the right moment, like after it’s finished checking the user’s answers. Here’s a simple way to do that: 1. Use Function Tools: Create a function tool that handles the user’s input and evaluates their answers. This helps keep everything tidy and makes sure the RPC only runs when the input is fully ready. 2. Invoke RPC After Evaluation: After the evaluation is done, you can use

perform_rpc

to send the results to the frontend. Here’s a quick example:

await ctx.room.local_participant.perform_rpc(
    destination_identity=participant.identity,
    method="client.evaluation",
    payload=json.dumps({
        "action": "evaluationCompleted",
        "success": True,
        "message": "All answers correct!"
    })
)

1. Control Timing with LLM Instructions: Guide the agent’s prompts so it waits until all user input is in before running the function tool. For instance, you can tell the agent to only evaluate after receiving every answer. This way, the RPC call happens right after the evaluation finishes. If you’d like, I’m here to help with more details or examples—just let me know! Sources: complex-agents/nova-sonic/form_agent.py

How should the LLM instructions be structured to make it consistently successful, though?

I'm here to help you ensure consistent invocation of function tools and RPC calls by guiding you through structuring the LLM instructions effectively: 1. Define Tools with Clear Schemas: It’s important to use precise descriptions and parameter types for each tool. For example:

@llm.ai_callable(description="Evaluate user answers and trigger RPC")
async def evaluate_answers(
    answers: Annotated[list, llm.TypeInfo(description="List of user answers")]
) -> dict:
    # Implementation
    return {"success": True, "message": "All answers correct!"}

1. Use Model-Specific Templates: Make sure the LLM outputs tool calls in strict JSON format. Here’s an example:

{
  "name": "evaluate_answers",
  "parameters": {
    "answers": [
      "A",
      "B",
      "C"
    ]
  }
}

1. Control Tool Invocation: You can use the

tool_choice

parameter in

ModelSettings

to specify when tools should be called. For instance:

session = AgentSession(
    model_settings=ModelSettings(tool_choice="required")
)

1. Suppress Unwanted Replies: To avoid unnecessary responses, use

ev.cancel_tool_reply()

within the

function_tools_executed

event. 2. Forward Results via RPC: Once evaluation is complete, you can use

perform_rpc

to send the results to the frontend seamlessly. If you’d like, I’m more than happy to provide further clarification or tailor examples to your specific needs—just let me know! Sources: Tool definition and use | LiveKit Docs | models/templates/meetkai-functionary-medium-v3.1.jinja | Function calling with Voice Agents | LiveKit Docs | examples/voice_agents/silent_function_call.py | agents/src/voice/generation.ts

I am asking how, in the exact instruction block for the realtime agent, I should instruct it to use the function

To help the realtime agent consistently use the function, it’s best to structure the instruction block clearly. Here’s a friendly example you can follow:

instructions="""
You are a helpful assistant. When the user completes all questions successfully, you must call the `evaluation_completed` function with the following payload:

{
    "success": true,
    "message": "All questions answered correctly."
}

Please don’t call this function until all questions are answered.
"""

In this block: • Be sure to clearly mention the function name (

evaluation_completed

). • Specify the exact condition for calling the function (like “when all questions are answered”). • Include the expected payload structure so it’s called correctly. If you’d like, I’m here to help with any further clarification or tweaks to fit your needs perfectly! Sources: livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/real... | Tool definition and use | LiveKit Docs

Are there any example scripts available where an agent calls an rpc tool in this manner?

I'm happy to share that there are example scripts available where an agent calls an RPC tool! One great example is the RPC + State Agent, which beautifully shows how a voice agent can use RPC to update a state database and interact with the frontend. You can check out the implementation here: RPC Agent Example There are also other helpful examples, such as: • Rust SDK: An agent requests a square root calculation via RPC. • Node.js SDK: A participant calls an RPC method like "square-root" and handles the response. • Python SDK: A caller requests a square root operation via RPC from a "math-genius" participant. • Go SDK: Demonstrates performing an RPC to compute the square root of 16. • JavaScript SDK: A browser-based demo where participants invoke RPC methods. If you'd like, I’m here to help you implement something similar tailored to your needs—just let me know! Sources: LiveKit Docs | examples/rpc/src/main.rs | packages/livekit-rtc/README.md | examples/rpc.py | examples/rpc/main.go

The instructions in the example you gave me seem to shed some light: instructions=""" You are an agent that manages state through RPC calls and also through function calls. You can create, read, update, and delete data objects. Available functions: - create_note: Create a new note - update_note: Update an existing note - read_note: Read a note by ID - list_notes: List all available notes - delete_note: Delete a note by ID """, Should I be creating similar instructions for the function call I need at the beginning of my own AI instructions?

Absolutely, it’s a great idea to create similar instructions at the start of your AI agent’s configuration. This really helps the agent understand what functions are available and when it should use them. For your situation, you might want to set up the instructions like this:

instructions="""
You are an agent that evaluates user answers and signals the frontend when the evaluation is complete.

Available functions:
- evaluation_completed: Notify the frontend that the user has successfully completed all questions. This function must be called with the following payload:
  {
      "success": true,
      "message": "All questions answered correctly."
  }

Do not call this function until all questions are answered correctly.
"""

By clearly outlining the function and its purpose, your agent will have a solid understanding of how and when to use it. If you’d like, I’m here to help you fine-tune this further—just let me know!