<@U08CKJH7C12> Is Livekit deploymens are down?
# agents
c
@refined-appointment-81829 Is Livekit deploymens are down?
r
What do you mean?
c
I have not changed anything and I am getting this in my Datadog logs Dec 16 142511.676 CAW_krmhzJoJu26h cloud.livekit.io self._context.run(self._callback, *self._args) Dec 16 142511.654 CAW_krmhzJoJu26h cloud.livekit.io File "/usr/local/lib/python3.11/asyncio/events.py", line 80, in _run Dec 16 142511.634 CAW_krmhzJoJu26h cloud.livekit.io handle._run() Dec 16 142511.607 CAW_krmhzJoJu26h cloud.livekit.io File "/usr/local/lib/python3.11/asyncio/base_events.py", line 1922, in _run_once Dec 16 142511.582 CAW_krmhzJoJu26h cloud.livekit.io self._run_once() Dec 16 142511.559 CAW_krmhzJoJu26h cloud.livekit.io File "/usr/local/lib/python3.11/asyncio/base_events.py", line 607, in run_forever Dec 16 142511.542 CAW_krmhzJoJu26h cloud.livekit.io self.run_forever() Dec 16 142511.494 CAW_krmhzJoJu26h cloud.livekit.io File "/usr/local/lib/python3.11/asyncio/base_events.py", line 640, in run_until_complete Dec 16 142511.474 CAW_krmhzJoJu26h cloud.livekit.io loop.run_until_complete(self._task) Dec 16 142511.454 CAW_krmhzJoJu26h cloud.livekit.io File "/home/appuser/.local/lib/python3.11/site-packages/livekit/agents/ipc/proc_client.py", line 71, in run Dec 16 142511.426 CAW_krmhzJoJu26h cloud.livekit.io client.run() Dec 16 142511.400 CAW_krmhzJoJu26h cloud.livekit.io File "/home/appuser/.local/lib/python3.11/site-packages/livekit/agents/ipc/job_proc_lazy_main.py", line 85, in proc_main Dec 16 142511.381 CAW_krmhzJoJu26h cloud.livekit.io self._target(*self._args, **self._kwargs) Dec 16 142511.361 CAW_krmhzJoJu26h cloud.livekit.io File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run Dec 16 142511.335 CAW_krmhzJoJu26h cloud.livekit.io self.run() Dec 16 142511.314 CAW_krmhzJoJu26h cloud.livekit.io File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap Dec 16 142511.287 CAW_krmhzJoJu26h cloud.livekit.io return self._bootstrap(parent_sentinel) Dec 16 142511.266 CAW_krmhzJoJu26h cloud.livekit.io File "/usr/local/lib/python3.11/multiprocessing/spawn.py", line 135, in _main Dec 16 142511.244 CAW_krmhzJoJu26h cloud.livekit.io code = spawn._main(child_r, parent_sentinel) Dec 16 142511.225 CAW_krmhzJoJu26h cloud.livekit.io File "/usr/local/lib/python3.11/multiprocessing/forkserver.py", line 313, in _serve_one Dec 16 142511.193 CAW_krmhzJoJu26h cloud.livekit.io code = _serve_one(child_r, fds, Dec 16 142511.169 CAW_krmhzJoJu26h cloud.livekit.io File "/usr/local/lib/python3.11/multiprocessing/forkserver.py", line 274, in main Dec 16 142511.150 CAW_krmhzJoJu26h cloud.livekit.io File "<string>", line 1, in <module> Dec 16 142511.099 CAW_krmhzJoJu26h cloud.livekit.io Call stack: Dec 16 142511.071 CAW_krmhzJoJu26h cloud.livekit.io TypeError: 'bool' object is not subscriptable Dec 16 142511.050 CAW_krmhzJoJu26h cloud.livekit.io ~~~~~~~~~~~~~~~^^^ Dec 16 142511.020 CAW_krmhzJoJu26h cloud.livekit.io record.exc_info = (record.exc_info[0], record.exc_info[1], None) # type: ignore Dec 16 142510.993 CAW_krmhzJoJu26h cloud.livekit.io File "/home/appuser/.local/lib/python3.11/site-packages/livekit/agents/ipc/log_queue.py", line 97, in emit
r
What LiveKit versions are you using?
c
livekit-agents==1.3.6
r
How about the rest?
And definitely not running 1.3.7?
c
I re-pushed another version and it seems to be fine now. No, am not running 1.3.7
everything is 1.3.6
r
ok
glad it is working. Not sure why you would have an issue. if you are not running latest version then not sure what could have changed.
c
Umm nevermind.. the issue is happening again
File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap Dec 16 143639.780 CAW_krmhzJoJu26h cloud.livekit.io return self._bootstrap(parent_sentinel) Dec 16 143639.756 CAW_krmhzJoJu26h cloud.livekit.io File "/usr/local/lib/python3.11/multiprocessing/spawn.py", line 135, in _main Dec 16 143639.731 CAW_krmhzJoJu26h cloud.livekit.io code = spawn._main(child_r, parent_sentinel) Dec 16 143639.706 CAW_krmhzJoJu26h cloud.livekit.io File "/usr/local/lib/python3.11/multiprocessing/forkserver.py", line 313, in _serve_one Dec 16 143639.687 CAW_krmhzJoJu26h cloud.livekit.io code = _serve_one(child_r, fds, Dec 16 143639.672 CAW_UDsL5QoqJX7M cloud.livekit.io process exiting Dec 16 143639.666 CAW_krmhzJoJu26h cloud.livekit.io File "/usr/local/lib/python3.11/multiprocessing/forkserver.py", line 274, in main Dec 16 143639.647 CAW_krmhzJoJu26h cloud.livekit.io File "<string>", line 1, in <module> Dec 16 143639.629 CAW_krmhzJoJu26h cloud.livekit.io Call stack: Dec 16 143639.611 CAW_krmhzJoJu26h cloud.livekit.io TypeError: 'bool' object is not subscriptable
All my calls are failing now.
r
What versions are you using?
c
it was an infra issue it seemed. Dan helped fix it - he sent a SIGTERM
r
ok
e
this was an issue with signal handling. there was an error which caused the shut down signal to be ignored. i forced killed it. can check in with the sdk team to see if we can ensure those signals always get honored
j
@cuddly-cartoon-47334 are you using
python agent.py dev
in the cloud hosting?
e
no he’s not using dev. The other versions of the agent shutdown correctly.
although I guess the bad build could have had dev enabled, not sure 🤷‍♂️
c
I used lk agent deploy