How do I use custom tool calling with the Gemini R...
# agents
a
How do I use custom tool calling with the Gemini Realtime model?
l
I found this: https://github.com/livekit-examples/gemini-playground/blob/main/agent/main.py#L224, which will generate a picture of a banana for you: https://gemini.livekit.io/ The docs also have a code snippet for built-in Google tools with Gemini: https://docs.livekit.io/agents/models/realtime/plugins/gemini/#gemini-tools
f
I spent hours on this yesterday, eventually concluding that Google RealTime does not support agent tools at all. In testing, it knows about them, can read descriptions/parameters - but fails to execute them.
f
@adamant-gigabyte-17689 Can you please elaborate on what you're trying to accomplish. Are you creating a livekit agent using Gemini Realtime as the modal and want your lk agent to have custom tools. If yes, this should be OK. Or are you trying to use built-in Google tools with the Gemini model? Please clarify
a
@fresh-hospital-14455 yes, I'm trying to using Gemini realtime with custom tool use not built-in google tools.
@lively-winter-88668, I found the same issue as @flaky-scooter-85113 . I’ve declared the tools and passed them to the Gemini Realtime model. When I ask the model what tools it has, it correctly lists the
lookup_weather
tool, so the tool definitions are clearly being loaded. However, when I then ask the model to use the tool, it keeps responding that it needs user permission before calling it. Is there a parameter or setting that allows the model to call tools directly without asking for additional user approval?
Copy code
@function_tool(raw_schema=raw_schema)
async def lookup_weather(raw_arguments: dict):
    """Use this tool to look up current weather information in the given location."""
    location = raw_arguments["location"]
    <http://logger.info|logger.info>(f"Looking up weather for {location}")
    return f"The weather in {location} is sunny with a temperature of 32 degrees Celsius."


class Assistant(Agent):
    def __init__(self) -> None:
        # --- 2. Pass the tool list to the parent Agent class ---
        super().__init__(
            tools=[lookup_weather], # <--- Tools go here
            instructions="""
# Role & Objective
You are a **CUSTOMER** with persona: 
Your goal is to role-play this CUSTOMER realistically.

# Context (INTERNAL ONLY)
- **YOUR Profile:** 

# Information Disclosure Policy
- **KEEP IT SECRET:** Do NOT voluntarily share personal details.
- **WAIT FOR QUESTIONS:** Only reveal details if asked.

# Handling Unclear Audio
- **Language Rule:** Always respond in **Thai**.

# Voice & Audio Guidelines
- **Speaking Style:** Speak naturally.
- **Fillers:** Use natural Thai fillers ("อ่า...", "เอ่อ...", "ครับ").

You can tell user what tools do you have and how to use them when appropriate.
"""
        )


server = AgentServer()

@server.rtc_session()
async def my_agent(ctx: JobContext):
    ctx.log_context_fields = {
        "room": ctx.room.name,
    }

    # Note: We do NOT pass tools to the model config here.
    # The Agent class handles passing them from the 'tools' list in __init__
    model = google.realtime.RealtimeModel(
        model="gemini-2.5-flash-native-audio-preview-09-2025",
        thinking_config=types.ThinkingConfig(
            include_thoughts=False,
        ),
        # modalities=["TEXT", "AUDIO"],
        proactivity=True,
        realtime_input_config=types.RealtimeInputConfig(
            automatic_activity_detection=types.AutomaticActivityDetection(),
        ),
    )

    session = AgentSession(
        llm=model,
    )
f
@adamant-gigabyte-17689 Can you try the model_id
gemini-live-2.5-flash-preview
. I have found this gemini model to be better with tool calling. This should work, let me know how it goes.
a
@fresh-hospital-14455 Thanks. The tool calling works better now, but the latency and support for my specific language (Thai) are still not very good.