I'm encountering an issue with an VoicePiplineAgen...
# server-api
s
I'm encountering an issue with an VoicePiplineAgent Process deployed on AWS (1GB RAM and 0.5 vCPU). For the past two days, CPU usage has consistently spiked to 100%. The logs indicate that the system is trying to create new agent processes, but these attempts are failing due to CPU overload, ultimately leading to a crash. Here’s a snippet of the logs:
Copy code
March 22, 2025 at 15:16 (UTC+5:30)
2025-03-22 09:46:50,851 - INFO - killing process {"pid": 3394}

March 22, 2025 at 15:16 (UTC+5:30)
2025-03-22 09:46:50,657 - INFO - killing process {"pid": 3391}

March 22, 2025 at 15:16 (UTC+5:30)
2025-03-22 09:46:50,657 - ERROR - initialization timed out, killing process {"pid": 3391}

March 22, 2025 at 15:16 (UTC+5:30)
2025-03-22 09:46:49,066 - ERROR - initialization timed out, killing process {"pid": 3388}

March 22, 2025 at 15:16 (UTC+5:30)
2025-03-22 09:46:49,067 - INFO - killing process {"pid": 3388}
Notably, once the CPU maxes out to 100%, no new sessions are being made; the system does not recover or cool down, remaining at maximum usage consistently. I understand that one simple fix would be to switch to an instance with more vCPU, but I’m looking for alternative solutions.
r
Yes, if CPU (by default) is greater than 80% no new jobs will be routed to the worker. You can control that in the worker options. Why is your half of a virtual CPU at 100%? And..... why are you only using 0.5 vCPU? Have you read this? https://docs.livekit.io/agents/deployment/ And this? https://docs.livekit.io/agents/build/anatomy/
s
Yes i have read the above docs. The reason behind setting the test server on 0.5vCPU is because i only need a single agent for testing (hence i also turned off auto scaling for test server). It's runs with no issues almost 90% of the time and with an average utilization of 60%, and when it goes above limit as you said it does not spin up the agent and again goes to minimal usage. However, I encountered this strange issue where the server went to 100% utilization and didn't cooled down until it i redeployed the container. Here's a usage graph of past week for reference:
r
Did you happen to run
top
or find what thread/processes were consuming the CPU? See anything interesting in the log at that time what it was doing that led up to that state? It would be good to try and isolate the root cause of that issue if you can so we can validate it is not something in the Agent code (which I suspect it is not)
s
Good idea. I'll recreate the issue and see which process is consuming the most resources.
r
Also I assume you are not running it with
dev
right?
s
no
n
@strong-xylophone-89556 Are you using the new Krisp background voice removal?
s
Nope, i am not even using the EOU model.
From then i have increased the server configuration to 1vCPU and 2GB ram, i haven't been able to recreate the issue since.
n
oh nice, ya we're running on 4 vCPU + 16gb (render) - we started hitting it during spikes of traffic.
s
Yeah for production we have a 4vCPU + 8GB variant. This issue was highlighted to me by the QA team during their testing.
on test server (.5vCPU and 1GB ram)