Hey everyone! I've been dealing with some problems...
# agents
e
Hey everyone! I've been dealing with some problems related to tool calling recently and could use some help... For a bit of context, we create tools programmatically schemas as described here 1. I'm fairly certain I used to be able to talk to my agent while it waited for a tool to respond, going so far as even calling other tools before some particularly long tool calls responded. This does not work for me anymore and the documentation seems to indicate it really isn't possible. Has this changed in some recent version? Is it possible to do what I described? 2. I can't make the generate_reply work from inside a tool call anymore. I copied this example almost 2 months ago and it worked fine if a tool call took over 2 seconds to end. It doesn't work anymore and I have no idea why. 3. My tool handler function is being called twice in a row every time... I think the agent generates a message when it gets the first response, but still, that's not something that happened in the past. I don't know when or why it started... Anyone else with a similar problem?
Answering myself for points 2 and 3. After calling generate_reply the agent was deciding the best way to reply would be calling the tool again without saying anything...
l
For 3, there was a similar report recently for gpt-realtime that I was unable to reproduce: https://livekit-users.slack.com/archives/C07FY8WHGPM/p1761729135359359. Do you have a minimal reproducible example? I was trying to reproduce previously by using this: https://github.com/livekit-examples/python-agents-examples/blob/main/realtime-agents/openai-realtime-tools.py.
e
Hi @lively-winter-88668, sorry it took me over a week to answer this... In the case I was talking about I'm pretty sure it was just the llm hallucinating because of unclear instructions. But I just got another case where the LLM calls the same tool over and over and it takes a few minutes for the first response to come through, after that I get a 'tools executions completed' event for every time the agent used said tool Any ideas? Have you seen this before?
l
I haven't seen that before, that's a LOT of function tool calls. It's strange they all have exactly the same timestamp - I thought maybe something to do with this, then I realised they all finished at the same time, not necessarily started. Make sure the tool choice is not set to
required
and you have clear definitions for your tools - but I assume you are already doing that.
e
Yes, the tool calls didn't start at the same time and only ended because the room was closed. This was an outbound call that went to voicemail. I think the tools only completed because the user ended the call. I'll double check my code but that's very weird...
Also, it doesn't happen every time. Most times it works just fine
And the 'llm_node' attributes are always the same
@lively-winter-88668 updating this. I have a 'generate_reply' in this tool and I was actually into the same problem as before. Whenever I use generate_reply inside a tool call it seems that the agent has no context that it already used the tool once, so it just tries to use the tool again
I'm not sure if that was always the case but I'm getting this behavior even by just copying examples from Livekit's documentation
l
Hey Lucas, which examples are you using to show this? I would like to try and reproduce it.
Whenever I use generate_reply inside a tool call it seems that the agent has no context that it already used the tool once, so it just tries to use the tool again
e
Copy code
import asyncio
from livekit.agents import function_tool, RunContext

@function_tool()
async def search_knowledge_base(
    self,
    context: RunContext,
    query: str,
) -> str:
    # Send a verbal status update to the user after a short delay
    async def _speak_status_update(delay: float = 0.5):
        await asyncio.sleep(delay)
        await context.session.generate_reply(instructions=f"""
            You are searching the knowledge base for \"{query}\" but it is taking a little while.
            Update the user on your progress, but be very brief.
        """)
    
    status_update_task = asyncio.create_task(_speak_status_update(0.5))

    # Perform search (function definition omitted for brevity)
    result = await _perform_search(query)
    
    # Cancel status update if search completed before timeout
    status_update_task.cancel()
    
    return result
We also have a tool to leave a voice message if the agent detects an answering machine, which is basically a generate_reply followed by a function to end the call
l
Ah, yes, I was able to trigger this with https://docs.livekit.io/agents/build/external-data/#verbal-status-updates, and seeing multiple tool executions with
generate_reply
. Seems like it was kicking off the pipeline again and deciding a function tool call was necessary. Is it feasible to add
tool_choice="none"
to your
generate_reply
call?
e
Absolutely! Thanks for your help!
Hey @lively-winter-88668, sorry to tag and bother you again... Using this thread because it's the tool. Since it's a voicemail tool I must make sure the message is spoken by the agent. Sometimes the agent has enough time to trigger the voicemail tool between sentences of a recorded message, so this would trigger the tool and immediately interrupt the agent making it not leave the message and ultimately not end the call. To avoid that, instead of messing with turn detection and stuff, I decided to make a loop in the tool that should only stop if the agent speaks the message. For that I followed the example from here. And it looks like this:
Copy code
...
        async def handle_voicemail():
            await self.session.generate_reply(
                instructions="""- Generate ONE short, natural voicemail message telling the person you will call back later.
                - Speak the message once, then STOP.
                - Do NOT repeat the message.
                - Do NOT add anything else.
                - Do NOT call or invoke any tool again after producing the voicemail.
                - After the voicemail is spoken, the call ENDS immediately.""",
                tool_choice=None,
            )

        while not voicemail_spoken:
            await context.speech_handle.wait_if_not_interrupted([handle_voicemail()])
            if context.speech_handle.interrupted:
                await asyncio.sleep(1.0)
            else:
                await context.wait_for_playout()  
                await asyncio.sleep(
                    0.5
                )  # Add a natural gap to the end of the voicemail message
                voicemail_spoken = True
                await hangup_call()
...
I implemented this last week and I'm sure it worked... it doesn't now. context.speech_handle.interrupted is always False. Am I missing something?