This message was deleted.
# helpdesks
steep-balloon-41261
11/20/2022, 6:32 PMThis message was deleted.
d
dry-elephant-14928
11/21/2022, 5:01 AMcpuload is probably the better metric to use. sysload is slower reacting.
what are you setting the threshold to? how quickly are these rooms being spun up?
p
polite-printer-17791
11/21/2022, 10:54 AMHi, I've this:
polite-printer-17791
11/21/2022, 10:54 AMnode_selector:
kind: cpuload
sort_by: cpuload
cpu_load_limit: 0.5
polite-printer-17791
11/21/2022, 11:04 AMI've some initial pods in eks and some placeholders to speed up new pods when necessary. When I start stress testing around 30% avg cpu, the hpa launches more livekit pods. I was expecting to see the initial pods(nodes) going up only to 50-60% due to nodeselector but they go all way up to 100% cpu!
d
dry-elephant-14928
11/22/2022, 5:24 PMhow many users are joining per second? and how is your HPA set up?
If the pods are spinning up slower than demand, you'd need to spin up more capacity ahead of the time. there are probably more tuning necessary in the node selector stack too. Can you log the CPU load over time (i.e. one log entry every 2 seconds) ? It'd be good to see how they are responding to the traffic increases
p
polite-printer-17791
11/22/2022, 6:17 PMHi, please take a look in the pic attached. It shows the initial and subsequent created nodes. The initial nodes keep receiving new rooms and respective participants all the way even when the cpu is above the defined limit.
d
dry-elephant-14928
11/22/2022, 6:20 PMthis is showing me number of rooms per node, without information about
• CPU load on these nodes
• amount of traffic each room is doing
If rooms are allocated, and then they keep getting larger, that would also push the CPU utilization above the limit previously set.
p
polite-printer-17791
11/22/2022, 6:41 PMHi, The number is room allocated per node at that minute in the initial ~10m(Numbers are not cumulative). After the room is created it handles 5 subs/pubs for 25m, then all participants disconnect. I'l work on getting the cpu load and bandwith data integrated in logs for better insight.
d
dry-elephant-14928
11/22/2022, 6:53 PMcan you share your config as well?
2 Views