LiveKit Community

I see an improvement in the single user refresh use case. Basically it works for me now! Abe has a more load test focused test i'm sure he'll be able to test next day or 2

yah, works pretty well. adjustable video res is in too

<@U01KZ61GV5Z> any chance the `AudioConfig` could be passed in via env variables?

yeah, you can pass any of the config attributes as yaml via `LIVEKIT_CONFIG`

yeah.. you can skip fields if you'd like.. it'll use default

we already have a field for this in livekit

Can you throw up another one for the js server api?

will do, next on my list :slightly_smiling_face:

hi folks, just wanted to quickly summarize where were at with our top issues:
1. <https://github.com/livekit/livekit-server/issues/26>
    a. Not so sure this one is livekit so we're doing everything possible to test different network setups, server setups, client side testing, testing of the livekit sample app (and more) to determine if this is a livespot issue, livekit issue, or a little of both
2. <https://github.com/livekit/livekit-server/issues/24>
    b. This is also a tricky one. Its difficult to reproduce but I think we're getting close.
3. Info on client side disconnect event
    c. I started a thread in <#C01KVTJH6BX|> about this to see how we should handle socket closures (we see the  `RoomEvent.Disconnected` event and will hook into it asap

overall I'd say this last week has been awesome in terms of stability improvements and feature additions so great job livekit folks :clap::skin-tone-4:

Yeah to add on to this, 3 / 4+ publishers we start getting issues where we get PLIs and “freezing”. We’ve tested our client vs sample (happens on both), aws vs vultr vs zeet, docker vs no docker and really not sure where else this could be happening.

I think this is currently our last P0 blocker before the first few customers are on so anything we can do to help debug / identify this issue let us know!

yeah, I'd be happy to. can you send me the url and the API key/secret that you are using?

that'll make it a bit easier for me to ping the server directly

also.. are you using v0.5.3? I pushed a fix merged upstream from ion-sfu, that was dropping Nack responses

I actually just deployed 0.5.3 this morning, have yet to test it

<http://livekit-server.zeet.app|livekit-server.zeet.app>

Key: test
Secret: test123

thinking out loud if it had anything to do with it.. FWIW.. PLIs are normal, everytime a new participant joins, all publishers would receive a PLI, which tells them to publish a new key frame

but the inability for clients to recover from the "freeze" is more worrisome

yeah that makes sense, also dropping nack responses would be a massive issue that causes PLI

<@U01KZ61GV5Z> I tried `0.5.3` and still seeing the video freezes

yeah I'm able to reproduce this, on a noisy-ish wifi and can definitely see the nack and pli going pretty high up...

what kind of machine is this running on? and what's the network configuration like?

I'm doing more tests on EC2 this week.. this seems fixable.. but def needs some thinking

Our main deployment is on <http://zeet.co|zeet.co> but that runs on AWS. We also tried it directly on AWS (through docker and outside) and also on vultr (this is where my dev box is) both docker and not.

The reason we tried docker vs docker (network host) vs direct is b/c docker network proxying even on the box does some weird things. It didn’t seem to effect anything.

the problem gets worse as more people are on, anecdotally it feels like more publishers causes it but we also tired 5 subscribers 2 publishers and had issues.

CPU / memory are bored on the boxes we’re working with (they’re not huge but there isn’t much load). It could be some thread contention which wouldn’t show up in CPU % but potentially delayed frames or something where NACKs aren’t processed / returned.

We can help you setup a test case on <http://livespot.co|livespot.co> where we get these bots to enter the session if it’s helpful, might be good for us to pair for a little bit tomorrow so we can show you what’s happening / do some live debugging

I'm able to repro on your server, so it's fairly straight forward to see what's happening. it seems to be a bandwidth constraint coupled with congestion control. I'll investigate

Is the bandwidth constraint the bit rate we’re setting? 

Should we be expanding that number as more tracks are added to the pc?

I'm not sure.. have to dig in to be certain

Do you see issues on the subscriber side or publisher? I think from what i could see it looked like the publisher was functioning correctly and subscribers were struggling.

I’m curious about the bitrate + CC now, it feels somewhat in line with the behavior we’re seeing. Especially if NACKs aren’t getting processed properly

getting my own environment so I can isolate this, stay tuned

curious if you were able to reproduce on your environment? I can also invite you to our deployment so you can access the logs / instance

yah I can repro it. there isn't much server logs at the moment.

just FYI, this is the highest priority thing for us right now. tho it's a bit involved so it might take a bit before we can resolve.

it has to do with congestion control and nack/pli interactions.

awesome yeah understand is pretty deep down and thanks for making it #1 on the list. We'll keep playing with it and start digging deeper to see what else we can figure out.

It's super weird b/c yesterday i was able to get it 4-5 person calls working really well (but my network is very good). I want to introduce some packet loss to see if that can trigger some of the issue (or use a hotspot)

when you are testing.. are the clients uploading from the same wifi? I'm seeing issues that indicate that it's upstream contention. i.e. congestion on the publisher's network..

hmm we certainly had an issue when james and i were together at my house the other day and his experience was worse than typical, but we also have the freezing problem in general even when we’re all not in the same place

I see, when it freezes for you, does it recover after a few seconds? or does the frames get stuck for minutes?

it recovers after a few seconds usually. i feel like it’s exactly the experience that this guy from an old google group thread is describing: <https://groups.google.com/g/discuss-webrtc/c/omveQB6Jq5A>

tho i think our freezing is sometimes just like 1 or 2 seconds long. it always goes back to normal after. i don’t think we’ve had it stuck for minutes before

the screenshot Abe shared with me, showed very high NACK/PLI counts. this happens because we are pushing more bits than the network can handle. Now the fun part is to track down if it's publisher side, subscriber side, or SFU side (less likely)

FWIW.. this is why Zoom is valued at 100B.. their shit just works regardless of the network. and where we aspire to be at.

ok, so you’ll need to determine what the network can handle and push through just under the limit amount of bits?

basically.. publish lower resolution streams when congested

and on the subscriber side, if downstream connection is congested, receive lower res versions. with simulcast this is very possible

I think that generally makes sense but i think its currently very regular for us, for example it will freeze for me on 2 devices (1 wifi, 1 hotspot) going to my server (in sf)

you can try this: turn off simulcast.. and publish a very low res stream

in my tests its ~5Mbit in / 15Mbit out (on the server)

my bandwith / network are pretty damn good other services dont' really cause this, e.g. i can do like 10Mbit 1080p webrtc streams

i'll try to turn downthe bandwith to 1Mbit per publisher and 480p res

to see if it helps but currently i believe its congestion / NACKs but i think the source isn't b/c of network congestion

there wouldn't be any nacks needed if there isn't congestion

I trust your wifi's great.. but sometimes speedtest is deceiving. I'm using this not-so-great-wifi from an airbnb right now.. speedtest shows me 30mbps down and 8mbps up.. tho for real time purposes, it was doing at most 1-2mbps upstream

jitter matters too.. if the packet arrives out of order and/or too late, it won't be used.. this gives you a good idea for that: <https://packetlosstest.com/>

yeah i get that, i've dealt with the webrtc problems / quality / packet loss issue before on a pretty big scale so i understand the nuances here.

we had this working on ion-cluster and didn't have the same issue. Granted this was on pion/webrtc v2 (at the time). same with janus/mediasoup/pion previously, not saying its livekit specific but i think there's something happening here that isn't the normal webrtc issues

Capture.PNG

we'll do a test call tomorrow and have everyoen run this test to isolate some networks

i'll also try no simulcast, h264, low bitrate, etc

I'm not saying this is definitely the problem you are seeing.. but I'm seeing issues with even just a couple of publishers with a shitty wifi

i'll check which setting we're currently using but i think its 2.5Mbit

but with webcam only it doesn't really reach that

k, I'll see what else might be happening too

<https://livespot.co/session/8f8e147e-41c0-4c77-b5ce-2514773f8599>

Running a test stream here 4 publishers (using our headless chrome publishing). when i do 2 publishers its smooth but when i do 4 the video starts to struggle. No PLIs / NACKs on the subscribers, the publisher box is ~50% loaded, livekit server is bored &lt; 5% cpu

it’s possible the publishers are struggling so i’ll try to create this test with 4 individual servers publishing instead of one but the test bots don’t seem stressed when i look at that system

ok i messed around with it 3 pulishers seems to work well 4 doesn’t. tried vp8/h264, simulcast true/false similar behavior on all

“work well” == no frame drop, 4 publishers we start seeing this freezing / lagging (frame drop) behavior

what resolution/bitrate are you using for the publishers?

we dont set it so whatever the `client-sdk-js` uses by default for the `createVideoTrack`

by default it's quite high. but I'll try to repro with the same settings and see

Spent a lot of time today digging into how we're capturing media streams, the format, bitrate, etc. Tried to set everything down pretty low and we're still experiencing similar issues. We did isolate some more details (1 person has a lossy network) and trying to filter out for "good cases" first. A bit stuck on where to dig into next but will keep updating here

hey man, yeah, I spent a bunch of time digging into this. I think it has to do with the publishing side.

russ and I were on an hour call and observed freezes ourselves.

I'm going to put in some stats debugging into the sdk so we can get more info from client logs.

yeah I also think its publisher side, the things you mentioned previously about NACKs not getting a response might actually be what's happening. I see that same behavior difference. I simulated minimal packet loss and tested with google meet. I see NACKs but no PLIs, the retransmissions are working. with LK i think re-transmissions get confused b/c i see re-transmitted bytes but PLIs too

Meets is very good about handling congestion.. LK is not

yeah, also did a test on janus: <https://janus.conf.meetecho.com/videoroomtest.html> worked a lot like meet

we've seen today where I wasn't receiving any packets from russ's side, but he could continue to receive my packets

the way meets work.. is if it detects any congestion at all, it'll quickly downgrade stream quality

what bitrate/resolution did you test with today?

cool, that's pretty reasonable.. with simulcast? or without?

<@U01KZ61GV5Z> when you guys do test calls do you regularly see what we're seeing? Or is it only with more publishers?

we see it even in 1 on 1 scenarios if one of the participants is on a poor connection. I’m releasing a client build later today that will log when it detects packet loss on either side, so we can get more info about what might be happening

Awesome, feel free to tag me when you release it 

ok, just pushed this. I'm def seeing quite a bit of nacks on the subscriber side.

this gives me a decent idea of where to look.

<https://ai.googleblog.com/2021/02/lyra-new-very-low-bitrate-codec-for.html?m=1|https://ai.googleblog.com/2021/02/lyra-new-very-low-bitrate-codec-for.html?m=1>

oh man incredible, going to test it out now

what was the issue / fix if you don’t mind me asking?

i’ll take a look at the commits shortly

<https://github.com/pion/ion-sfu/pull/447>

ah makes perfect sense missing header info

so NACKs were happening but being thrown away b/c they couldn’t be associated

initial test looked good! will do more extensive testing tommorrow!

Did a longer test, looks good! Appreciate the hard work on the fix this should unlock us to get some customers on!

awesome dude. still room for improvement for handling poor wifi though.. I’ll be working on this next

<@U01KZ61GV5Z> hey can you add one of our team members thats working on mobile to the livekit github org? `siddarthkay` please and thx

Zoom profit for 2020: $1b, up 1000% from the prior year. This space is only going to get bigger and bigger.

More niche video applications will continue to pop up and they’ll all need a platform like LiveKit to provide a plug-and-play video stack. I’m very very bullish on LiveKit 

1B profit a year.. damn. these guys raking it in

Yeah, absolutely unprecedented growth. I’ve never seen growth like this before for any product ever. Their Q4 2020 profit is up almost 9000%