https://livekit.io logo
Join Slack
Powered by
When running a stress test on lk cloud for my self...
# cloud
b
When running a stress test on lk cloud for my self-hosted agent running GKE. i run into this issue:
Copy code
Traceback (most recent call last):
  File "/app/.venv/lib/python3.13/site-packages/livekit/agents/worker.py", line 1243, in _job_request_task
    await self._request_fnc(job_req)
  File "/app/src/main.py", line 918, in request_fnc
    await req.accept()
  File "/app/.venv/lib/python3.13/site-packages/livekit/agents/job.py", line 798, in accept
    await self._on_accept(accept_arguments)
  File "/app/.venv/lib/python3.13/site-packages/livekit/agents/worker.py", line 1210, in _on_accept
    raise AssignmentTimeoutError() from None
livekit.agents._exceptions.AssignmentTimeoutError
What trigger this error? How to prevent and handle it ? Can i just add a retry in
Copy code
<http://log.info|log.info>(
        "✅ Accepting job request",
        room=room_name,
        load=_JobAwareLoadCalc.current_load,
        cpu_load=_JobAwareLoadCalc.cpu_load,
        active_job_count=_JobAwareLoadCalc.active_job_count,
        job_id=req.id,
        dispatch_id=dispatch_id,
    )
    await req.accept()
l
When running a stress test on lk cloud
My guess would be you are hitting plan limits during your test. Also bear in mind that the infrastructure on ship / scale is designed for realistic usage scenarios, so if your test is trying to start lots of simultaneous calls, you will experience issues.
b
I dispatched 200 rooms with an lk agent. I found this fails 2-10 times,
Will retrying accept make it work ? I noticed that in some cases after 
AssignmentTimeoutError
lk will send another dispatch job and it will be accepted. But 2-10 times, after 
AssignmentTimeoutError
lk stop dispatching. I check the rooms, and they dont have any participants or any errors.
Copy code
await req.accept()
@lively-winter-88668 Thank you for looking into this issue
l
I dispatched 200 rooms with an lk agent
Are you dispatching rooms simultaneously? Please add a second or two between each dispatch to make it more realistic. There was a similar question in the community recently: https://community.livekit.io/t/agent-join-latency-spikes-at-50-concurrent-users-on-scale-plan/261/8?u=darryncampbell
b
Copy code
async def request_fnc(req: JobRequest) -> None:
    """Reject job if duplicate dispatch or room already has an agent."""
    ...
    try:
        await req.accept()
        return 
    except AssignmentTimeoutError:
        # Clean up dedup cache — the job was never actually assigned
        if dispatch_id:
            _seen_dispatch_ids.pop(dispatch_id, None)
        log.error(
            "🚨 Assignment timed out — call might be dropped",
            room=room_name,
            job_id=req.id,
            dispatch_id=dispatch_id,
        )
        return 
        # Shoud i re-raise the error ? or just return is ok ?
l
Copy code
# Shoud i re-raise the error ? or just return is ok ?
Just return
2 Views
Open in Slack
PreviousNext
Powered by