LiveKit Community

Migrating_from_Twilio_to_Livekit.md

I read this entire thread.. this is SUPER helpful to hear your experiences. Thank you for sharing it.

You've brought up a lot of good points, especially with the annoyingness of type casting. we'll improve the API in the next version so it's a bit easier to work with.

Also.. 100% agreed with the poor state of deployment docs.. My goal is to have a helm chart for Kube in the next couple of weeks so it'd be much easier to deploy something on kube this way.

Yes a helm chart would be fantastic, I'm sure my boss would be very happy (He likes helm and tiller a lot).

Personally, Gitlab Pipelines K8s integration can create and deploy workloads from Dockerfile's just fine, so it _ain't no thang_ to me, but I'm sure there are folks that would really like a public helm chart too :joy:

yeah I think it gets tricky with the port configurations.. and autoscaling

it'll need quite a bit of ports forwarded due to the nature of UDP

Speaking of autoscaling, I need write up something so we can start a conversation about global load balancing

Oh interesting! Do post back if you end up implement that change, I would love to test that out

Full react-native support would do wonders for us, we’re working on building the call view in native iOS and bridging the view but it’s a pita 

this is the response in the websocket to the join message <https://pastebin.com/Scs1RmaN>

But I am 100% confident that there are not that many `otherParticipants` in the call

livekit is returning zombie participants, they're long long gone

Bye the last 3 messages in this thread were intended for a different thread :X

<@U01KZ61GV5Z> any chance you were able to mess with this?

not yet, probably won't get to it until middle of next week

some good news to share around RN. I managed to get subscribe working on this stack last night. It ended up being a big change, as the RN WebRTC stack only supports an older "streams" API. It'll take some more effort for publishing, but I'm feeling confident this path will work out

That’s great, sounds like good progress. Very excited to see how it goes for publishing. Do you want me to add publishing to the react native livekit example app I pushed?

 Can do! I’ll tackle that this weekend for you

btw <@U01KZ61GV5Z> here's a session that's currently live using livekit, thought you might be interested in seeing it work: <https://livespot.co/session/4f63911c-9545-4923-932f-ae863ee72ca8>

I'm afk and couldn't see it while it's going. I think the issue with "zombie connections" is because of running out of ports

we're making the box bigger and bringing it back up

somethign happened and it maxed out an 8core box

the interesting bit is the websocket response to the `join` message

those zombie participants are filling up the `otherParticipants` array

and I'm unsure if the server actually does more work as that number gets higher

we also had issues on other ends of our stack. we got rate limited by our auth

we've had a 15 person call before and it was way lower in usage than this

<@U01KZ0SCDKL> yeah you're likely hitting us as we're redeploying stuff

you haven't ran the debug build that I'd just cut right?

I think it's a combination of "zombie participants" causing the slowdown.. Let's try `0.7.4d` for now and see if that improves things.

I was very tempted to try that version yesterday during all those call issues but decided not to. I’ll be testing it out tonight

sounds good, my suspicion is port issues being a major cause of these problems. I'm adding single port support into pion itself. when that's out.. we should be in much better shape

Its hard to say that’s it because of the pileup. You would think running out of ports would simply fail any new clients to join a room

Question for you is, does a pileup of zombies result in the server doing more work? That would explain our massive CPU run up 

the server does have to lock and go through all the dead participants in a room. It's difficult to say how much overhead there are; but it's definitely there

zombies didn't show up with the latest server version

but when I roll back to 0.7.4, I get zombies consistently

great, that's good to know.. are the participants connecting though?

the failed connections caused the pile up

I don't think livekit was causing the failed connection

we'll upgrade to 0.7.5 after we run some load tests, want to make sure we test livekit for scale before upgrading going forward

do you guys have any plans to write load tests?

awesome, let us know if there's anything we can help with on that front

definitely. I'll def hit you up.. since the JS client simulates pretty decent browser load

quick update, I think this is peering up again in 0.7.5
i was running a load test of 100 users, 99 subscribers and 1 publisher, on an 8 core cpu 4gb ram box.
some of the bots were able to join, reaching a max of around 62 bots actually streaming video.
a lot of other bots ran into errors connecting to livekit, which caused them to trigger a reconnect, but that caused the `participants` list to explode in size, potentially from the dead connections pilling up

every 5-10 seconds or though the `participants` list would shrink in size cleaning out some of the dead connections

I plan to add a retry backoff to the connect call to make sure the browsers don't ddos the servers when they're all trying to join at once

I should try a load test that staggers the join times of the users a bit, we're running on <http://zeet.co|zeet.co> which runs pods for us in k8s so they spin up instantly

this sounds like a port issue. just as last time

right, now I'll try to expand that and limit the user count

200 ports isn't enough if you want to handle 100 users.. even though it's theoretically possible.

would you mind linking me to that code if you see it again?

basically it tries 5 times at random to find an available port, gives up after it fails

sweet, we should see a good bump in stream quality

checkout the sample config for updates to the format: <https://github.com/livekit/livekit-server/blob/master/config-sample.yaml>

upon testing, we are discovering some connectivity issues when the user is on cellular networks. please don't upgrade just yet

we found the source of the issues to be performance related to single port mode. working on it tonight/tomorrow and will release 0.8.3 then

:wave: feeps. <@U01UEMQ8C3F> is the person I mentioned that's also looking to use LiveKit in C#

he's started to work on a LiveKit client that I figured it might be an chance for us to collaborate

howdy - yeah I'm working on a Unity integration. Hoping to have something working this week and something productionized by next

<@U022KB72NFJ> repo is bootstrapped with working code and I added a super basic README: <https://github.com/livekit/client-sdk-unity>

I think this is a good point where we can begin to collaborate on improvements. For that we can use github issues

Yes, that sounds like a good point to start. I'll start looking at the repository in detail next week. Maybe I'll start by implementing the "Participant" class like in the other sdk clients