i randomly started getting this log and then my ag...
# agents
m
i randomly started getting this log and then my agent crashes:
Copy code
{"level": "ERROR", "name": "asyncio", "message": "Task exception was never retrieved\nfuture: <Task finished name='agent_job_termination' coro=<AgentServer._handle_termination() done, defined at /usr/local/lib/python3.13/site-packages/livekit/agents/worker.py:1224> exception=ValueError('process object is closed')>", "exc_info": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.13/asyncio/tasks.py\", line 507, in wait_for\n    return await fut\n           ^^^^^^^^^\nasyncio.exceptions.CancelledError\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.13/site-packages/livekit/agents/ipc/supervised_proc.py\", line 245, in aclose\n    await asyncio.wait_for(\n        asyncio.shield(self._supervise_atask), timeout=self._opts.close_timeout\n    )\n  File \"/usr/local/lib/python3.13/asyncio/tasks.py\", line 506, in wait_for\n    async with timeouts.timeout(timeout):\n               ~~~~~~~~~~~~~~~~^^^^^^^^^\n  File \"/usr/local/lib/python3.13/asyncio/timeouts.py\", line 116, in __aexit__\n    raise TimeoutError from exc_val\nTimeoutError\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.13/site-packages/livekit/agents/worker.py\", line 1229, in _handle_termination\n    await proc.aclose()\n  File \"/usr/local/lib/python3.13/site-packages/livekit/agents/ipc/supervised_proc.py\", line 254, in aclose\n    await self._send_kill_signal()\n  File \"/usr/local/lib/python3.13/site-packages/livekit/agents/ipc/supervised_proc.py\", line 310, in _send_kill_signal\n    self._proc.kill()\n    ~~~~~~~~~~~~~~~^^\n  File \"/usr/local/lib/python3.13/multiprocessing/process.py\", line 139, in kill\n    self._check_closed()\n    ~~~~~~~~~~~~~~~~~~^^\n  File \"/usr/local/lib/python3.13/multiprocessing/process.py\", line 101, in _check_closed\n    raise ValueError(\"process object is closed\")\nValueError: process object is closed", "timestamp": "2025-12-05T18:56:40.618918+00:00"}

...couple of other logs

worker process is not responding.. worker crashed?
worker process is not responding.. worker crashed?
worker process is not responding.. worker crashed?
worker process is not responding.. worker crashed?
worker process is not responding.. worker crashed?
worker process is not responding.. worker crashed?
worker process is not responding.. worker crashed?
worker process is not responding.. worker crashed?
Any idea why? How can i go about diagnosing what's causing this?
It's on Render right now. I can restart the service, and once I do that, it works fine for a while, but this has happened 3 times in past 24 hours
Just got it again:
Copy code
{"level": "ERROR", "name": "asyncio", "message": "Task exception was never retrieved\nfuture: <Task finished name='agent_job_termination' coro=<AgentServer._handle_termination() done, defined at /usr/local/lib/python3.13/site-packages/livekit/agents/worker.py:1224> exception=ValueError('process object is closed')>", "exc_info": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.13/asyncio/tasks.py\", line 507, in wait_for\n    return await fut\n           ^^^^^^^^^\nasyncio.exceptions.CancelledError\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.13/site-packages/livekit/agents/ipc/supervised_proc.py\", line 245, in aclose\n    await asyncio.wait_for(\n        asyncio.shield(self._supervise_atask), timeout=self._opts.close_timeout\n    )\n  File \"/usr/local/lib/python3.13/asyncio/tasks.py\", line 506, in wait_for\n    async with timeouts.timeout(timeout):\n               ~~~~~~~~~~~~~~~~^^^^^^^^^\n  File \"/usr/local/lib/python3.13/asyncio/timeouts.py\", line 116, in __aexit__\n    raise TimeoutError from exc_val\nTimeoutError\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.13/site-packages/livekit/agents/worker.py\", line 1229, in _handle_termination\n    await proc.aclose()\n  File \"/usr/local/lib/python3.13/site-packages/livekit/agents/ipc/supervised_proc.py\", line 254, in aclose\n    await self._send_kill_signal()\n  File \"/usr/local/lib/python3.13/site-packages/livekit/agents/ipc/supervised_proc.py\", line 310, in _send_kill_signal\n    self._proc.kill()\n    ~~~~~~~~~~~~~~~^^\n  File \"/usr/local/lib/python3.13/multiprocessing/process.py\", line 139, in kill\n    self._check_closed()\n    ~~~~~~~~~~~~~~~~~~^^\n  File \"/usr/local/lib/python3.13/multiprocessing/process.py\", line 101, in _check_closed\n    raise ValueError(\"process object is closed\")\nValueError: process object is closed", "timestamp": "2025-12-05T20:01:23.456287+00:00"}

worker process is not responding.. worker crashed?worker process is not responding.. worker crashed?
worker process is not responding.. worker crashed?
worker process is not responding.. worker crashed?
worker process is not responding.. worker crashed?
I'm chatting with the livekit deepwiki and it seems like it's something internal to livekit: https://deepwiki.com/search/any-idea-what-might-be-going-o_eeea2e9c-31cc-4b1f-8e81-cd996de2ab18?mode=fast
l
How can I reproduce this? Do you only see it with your agent hosted in Render? I don't see any similar open issues right now (maybe this one is the closest, but I don't think it's identical: https://github.com/livekit/agents/issues/3841)
m
That gh issue to be very close to what I'm experiencing I can't figure out a way to reproduce this using a minimal viable example, mainly because for 95% of my calls, the agent works totally fine and doesn't throw this issue (which is why it's hard to reproduce). Then the issue is randomly thrown, and i have to restart my service to get the worker to accept anything again this did start happening soon after I upgraded my package versions to start using the AI Insights in the livekit dashboard. Probably coincidence but may be related Also, I have an agent currently deployed to livekit cloud. I can try to disable my render agent to see if that has something to do with it. The reason I have two deployments is because I've tried to only use the livekit one in the past and it has randomly crashed, and not scaled up, leaving me with 0 instances to accept new session. • side note - It also seems that livekit sends ~80% of new sessions to the render agent, which is strange to me. i thought it would split them evenly
I'll disable my render agent for a while to see if that changes anything
l
If it is the same issue as I referenced in GitHub, I recommend adding onto that thread, though the engineer is asking for a reproducible example, at least it lets the team know this issue is more widespread. I do also note that that issue OP said they are using agents
1.2.14
, so you upgrading may be a coincidence.
• side note - It also seems that livekit sends ~80% of new sessions to the render agent, which is strange to me. i thought it would split them evenly
Me too, and those agent dispatch errors happen when the balancer can't find an available agent worker to accept the job.
m
I added to the gh issue. Today is the first full day I'll have it only on LK cloud, so i'll let you know if i still get the error
šŸ‘€ 1
d
@melodic-arm-17634 did you ever find a solution here? Experiencing this every now and then with an agent hosted via Livekit Agents. Curious about what your experience with the agent on render has been as well
m
@dry-accountant-99441 Render has been great. scales up and down nicely, and not having any concurrency cap is super nice the only solution I had was to downgrade all of my livekit versions to what they were like 1-2 weeks before I started having issues. then, a few weeks later I upgraded them and haven't had that same issue I do still occasionally have specific instances where my code reports that the agent doesn't join the call, but the issue I was describing here was like 1x per day, no agent would join any call, so that's fixed
d
thanks for the reply! yeah was just experiencing the same thing – the worker process crashed and then no agents could join anything. Will try out render! Have you diverged much from the autoscaling settings here? and like any sense at how many concurrent calls per instance you're getting