I am running into an issue where my agent becomes irresponsive, I am using the VoicePipelineAgent. Here is what happens, I set a context where I tell the agent it can use a function to GetProducts, but actually the function has not been added to context, normally what should happen is that the agent tells me that it cannot find the products or it doesn't have a function like GetProducts in context but instead of that I face an error and my agent becomes completely irresponsive. Why could this be? Here is the error log;

2025-02-28 10:59:34,511 - livekit.agents.pipeline - ERROR - Error in _stream_synthesis_task
Traceback (most recent call last):
  File "/Users/xx/Library/Caches/pypoetry/virtualenvs/xxxx/lib/python3.11/site-packages/livekit/agents/utils/log.py", line 16, in async_fn_logs
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xx/Library/Caches/pypoetry/virtualenvs/xxxx/lib/python3.11/site-packages/livekit/agents/pipeline/agent_output.py", line 273, in _stream_synthesis_task
    async for seg in tts_source:
  File "/Users/xx/Library/Caches/pypoetry/virtualenvs/xxxx/lib/python3.11/site-packages/livekit/agents/utils/aio/itertools.py", line 47, in tee_peer
    item = await iterator.__anext__()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xx/Library/Caches/pypoetry/virtualenvs/xxxx/lib/python3.11/site-packages/livekit/agents/pipeline/pipeline_agent.py", line 1102, in _llm_stream_to_str_generator
    async for chunk in stream:
  File "/Users/xx/Library/Caches/pypoetry/virtualenvs/xxxx/lib/python3.11/site-packages/livekit/agents/llm/llm.py", line 246, in __anext__
    raise exc from None
  File "/Users/xx/Library/Caches/pypoetry/virtualenvs/xxxx/lib/python3.11/site-packages/livekit/agents/llm/llm.py", line 156, in _main_task
    return await self._run()
           ^^^^^^^^^^^^^^^^^
  File "/Users/xx/Library/Caches/pypoetry/virtualenvs/xxxx/lib/python3.11/site-packages/livekit/plugins/openai/llm.py", line 736, in _run
    raise APIConnectionError(retryable=retryable) from e
livekit.agents._exceptions.APIConnectionError: Connection error.
2025-02-28 10:59:34,511 - livekit.agents.pipeline - ERROR - Error in _stream_synthesis_task
Traceback (most recent call last):
  File "/Users/xx/Library/Caches/pypoetry/virtualenvs/xxxx/lib/python3.11/site-packages/livekit/agents/utils/log.py", line 16, in async_fn_logs
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xx/Library/Caches/pypoetry/virtualenvs/xxxx/lib/python3.11/site-packages/livekit/agents/pipeline/agent_output.py", line 273, in _stream_synthesis_task
    async for seg in tts_source:
  File "/Users/xx/Library/Caches/pypoetry/virtualenvs/xxxx/lib/python3.11/site-packages/livekit/agents/utils/aio/itertools.py", line 47, in tee_peer
    item = await iterator.__anext__()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xx/Library/Caches/pypoetry/virtualenvs/xxxx/lib/python3.11/site-packages/livekit/agents/pipeline/pipeline_agent.py", line 1102, in _llm_stream_to_str_generator
    async for chunk in stream:
  File "/Users/xx/Library/Caches/pypoetry/virtualenvs/xxxx/lib/python3.11/site-packages/livekit/agents/llm/llm.py", line 246, in __anext__
    raise exc from None
  File "/Users/xx/Library/Caches/pypoetry/virtualenvs/xxxx/lib/python3.11/site-packages/livekit/agents/llm/llm.py", line 156, in _main_task
    return await self._run()
           ^^^^^^^^^^^^^^^^^
  File "/Users/xx/Library/Caches/pypoetry/virtualenvs/xxxx/lib/python3.11/site-packages/livekit/plugins/openai/llm.py", line 736, in _run
    raise APIConnectionError(retryable=retryable) from e
livekit.agents._exceptions.APIConnectionError: Connection error.
2025-02-28 10:59:34,518 - livekit.agents.pipeline - DEBUG - speech playout finished

It seems likely that the issue you're encountering is related to an

APIConnectionError

when the agent attempts to execute a function that isn't properly defined in the context. This error suggests a connection problem with the API, possibly due to the function call not being handled correctly or the function not being available in the context, leading to an unhandled exception. My understanding is that the

VoicePipelineAgent

should handle such cases gracefully by either skipping the function call or providing a default response indicating the function is unavailable. However, if the function context isn't set up correctly, it might lead to the agent becoming unresponsive. You might want to ensure that the function context is correctly initialized and that any API calls are non-blocking to prevent the agent from stalling. Additionally, checking the network connectivity and API endpoint configurations could help resolve the connection error. Would you like more detailed guidance on setting up the function context or handling API errors? Sources: - https://livekit-users.slack.com/archives/C07FY8WHGPM/p1732345136.594869 - https://livekit-users.slack.com/archives/C07FY8WHGPM/p1731900104.974219 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.