Hi all! We have been doing load testing with a basic bot, built with agents-js 1.0.25 (latest). Our goal was to measure how many calls we can make in a CPU optimized server with 4 CPUs 8 GB RAM. We have our agent deployed in a sole node inside a k8s cluster We have tried with and without Turn Detector on, and the results are that we can have 10-12 calls per server before the calls start to add latency and not answer properly. What's more worrying for us is that the memory doens't seem to be freed properly... We are adding an extra 1.3GB after each set of 15 calls that is never released...until finally the pod restarts because of Memory Pressure We have made tests in the same environment with an old bot, with Silero and turn detection on, built with 0.x version, and we can reach 30 calls easily, plus the memory is released without any problem... Have anybody go throught something like that? Any advice? Anything we can potentially doing wrong? Many thanks in advance

@bitter-ghost-46918 any advice here?

Hi Edu, are you using the inference gateway by any chance?

Sorry, forgot to explain the whole picture... we work on a self-hosted livekit instance, so no, we are not using inference

do you have the agent set up code and trying to see if we can replicate

well, actually we tried with several agents, one being the basic_agent from the agents-js repo, and the results have been pretty much the same...

Which STT/TTS are you using?

Deepgram STT Elevenlabs TTS

Currently the elevenlabs TTS is implemented in a way that it starts a new websocket connection every time, which may leads to high memory usage.. I'm working on a fix for that issue

There is a PR that looks quite promising for fixing this on the 11lab plugin

Actually we don't see big RAM consumption, the problem is that is not freed properly.... What is not allowing us to have more than 10-12 concurrent calls with this basic bot build with 1.x Node SDK seems to be CPU. So, we have 2 different problems: 1. With a brand new pod / node we can't have more than 12 concurrent calls because CPU goes high... This can be solved by adding more pods, right. I't a matter of costs... but what suprises us is that with 0.x version we could have more than double capacity with similar functionality 2. What's worst, is that memory is not freed properly, some processes seem to be never freed (as sais we have dive deep into the pod), so 3-4 sets of 10 concurrent calls finally end up killing the pod

Your test seems to be in the ballpark of what might be expected for a 4CPU with 8GB of ram. I believe our tests (with Python) was ~10-20 concurrent sessions. https://docs.livekit.io/agents/ops/deployment/custom/#memory-and-cpu > This agent server can handle 10-25 concurrent jobs, depending on the components in use. Team is looking into the memory retention issue. Some retention is expected as we keep some number of processes warm as the agent scales. https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/worker.py#L177 I expect Node agent has a similar facility but I am less familiar with that. Also, you might consider using the Python Agents SDK.

Hey @cool-baker-11020, those two PR would most likely resolves the current issue: https://github.com/livekit/agents-js/pull/910 https://github.com/livekit/agents-js/pull/912

Hi again. @refined-appointment-81829 we have tried with Python also, but when adding some imports we need for our agents, we just can have 4 or 5 calls per server. @bitter-ghost-46918 We have made a "workaround" to solve the problem with "zombie" child processes not being killed and not freeing memory by catching SIGTERM and exiting process manually in our agent. But we have seen on your agent-js code that for some reason you are not treating SIGTERM on purpose.

// don't do anything on SIGTERM

// Render uses SIGTERM in autoscale, this ensures the processes are properly drained if needed

process.on('SIGTERM', () => {

logger.debug('SIGTERM received in job proc');

});

However we are not sure about the consequences of this workaraound because from time to time we see an ESRCH that kills the whole pod and all the calls running at this time. On the other hand, we are pretty sure that there is a race condition when the call ends...We have added a "workaround" in our agent, adding a 5 second sleep before closing room and session shutdown. If not doing this, when we hang up the call, we receive some errors about writing in closed streams