Hi team, were are experiencing a very strange and critical issue today where roughly 1/3 of our sessions get "phantom" agent joins but our self-hosted agent never receives a job request. Details in thread
Solved
microscopic-salesmen-11790
11/10/2025, 3:36 PM
Project ID: p_4886vaza587
Example bad sessions:
⢠RM_or5K6ULsm93q
⢠RM_F9vdg8jmonrD
⢠RM_ipSzL4k8hBvo
microscopic-salesmen-11790
11/10/2025, 3:38 PM
So far this morning we have had 11 sessions, but only 8 job requests to the worker.
For each of the "bad" sessions, LiveKit shows the agent having joined the room, but it is clear from our logs that our agent never received a job request.
microscopic-salesmen-11790
11/10/2025, 3:47 PM
The only link/clue that I can find is that LiveKit shows the agent as having two workers, and one of the two has a created timestamp from before our last deployment (which took place at Nov 9 11:08pm PT):
microscopic-salesmen-11790
11/10/2025, 3:51 PM
I found the issue. There was an another instance of our agent incorrectly deployed with the same LiveKit connection string in another cloud project that was "stealing work" from our prod agent š