This message was deleted LiveKit Community #helpdesk

Join Slack

This message was deleted.

# helpdesk

steep-balloon-41261

02/03/2021, 11:50 PM

This message was deleted.

dry-elephant-14928

02/04/2021, 1:00 AM

just tried to repro locally, have up to 4 participants and all receiving streams.

dry-elephant-14928

02/04/2021, 1:01 AM

will dive into the logs and see what I can spot.

dry-elephant-14928

02/04/2021, 1:01 AM

question for you.. what do you use to view the JSON in the CSV logs? it's a bit hard to parse

wonderful-manchester-12533

02/04/2021, 1:04 AM

its a dump from kibana 😕 if you run it locally/remotely you can import the logs

dry-elephant-14928

02/04/2021, 1:05 AM

is it possible to dump a text-only version?

wonderful-manchester-12533

02/04/2021, 1:17 AM

we can invite you to the kibana if you want

wonderful-manchester-12533

02/04/2021, 1:17 AM

then you'll have direct log access

dry-elephant-14928

02/04/2021, 1:17 AM

sure, that works

wonderful-manchester-12533

02/04/2021, 1:17 AM

and I can just give you links to queries

wonderful-manchester-12533

02/04/2021, 1:17 AM

cool, email?

dry-elephant-14928

02/04/2021, 1:18 AM

david@davidzhao.com

wonderful-manchester-12533

02/04/2021, 1:21 AM

https://app.logz.io/#/goto/ce675d96ddad86c863805557fa45d805?switchToAccountId=156134

wonderful-manchester-12533

02/04/2021, 1:21 AM

cool invited

wonderful-manchester-12533

02/04/2021, 1:21 AM

here's a query you can click at anytime that will show: • All livekit server logs • All client side logs from me, you, and furqan

dry-elephant-14928

02/04/2021, 1:21 AM

nice

wonderful-manchester-12533

02/04/2021, 1:21 AM

hope it helps, lmk if you have any q's

dry-elephant-14928

02/04/2021, 1:22 AM

kk, will have a look

dry-elephant-14928

02/04/2021, 1:26 AM

the logs you sent me has a timestamp of

"2021-02-03T22:13:44.216Z"

dry-elephant-14928

02/04/2021, 1:26 AM

I'm not seeing those on kibana

wonderful-manchester-12533

02/04/2021, 1:27 AM

which logs are you looking at rn

wonderful-manchester-12533

02/04/2021, 1:27 AM

the ones on https://github.com/livekit/livekit-server/issues/7?

wonderful-manchester-12533

02/04/2021, 1:28 AM

here they are

wonderful-manchester-12533

02/04/2021, 1:28 AM

https://app.logz.io/#/dashboard/kibana/discover?_a=(columns:!(message),index:'logzi[…]03T23:00:00.000Z'))&accountIds=156134&switchToAccountId=156134

dry-elephant-14928

02/04/2021, 1:29 AM

ahhh ok timezone diff then

wonderful-manchester-12533

02/04/2021, 1:29 AM

I just manually set the time range to 5pm est -> 6pm est which is around when I tested that

dry-elephant-14928

02/04/2021, 1:29 AM

the exported ones are GMT

wonderful-manchester-12533

02/04/2021, 1:29 AM

ah yes sorry kibana default

dry-elephant-14928

02/04/2021, 1:29 AM

cool no biggie.. but this is good will take a look

wonderful-manchester-12533

02/04/2021, 1:31 AM

sweet

dry-elephant-14928

02/04/2021, 1:33 AM

am I right to guess that each user has a diff email?

dry-elephant-14928

02/04/2021, 1:33 AM

each livekit participant has a diff email

wonderful-manchester-12533

02/04/2021, 1:35 AM

yes

wonderful-manchester-12533

02/04/2021, 1:35 AM

we never pass the email to livekit tho, we pass our internal IDs

dry-elephant-14928

02/04/2021, 1:38 AM

yup, that's fine.. just trying to understand the order of operations

dry-elephant-14928

02/04/2021, 1:43 AM

is this consistently reproducible? I see one bug that's in the pion stack.. been meaning to submit a patch to that

dry-elephant-14928

02/04/2021, 1:52 AM

we seem to be missing some logs here.. at least for the server

dry-elephant-14928

02/04/2021, 1:52 AM

for example.. I see that you have

error getting buffered packet

printed out..

dry-elephant-14928

02/04/2021, 1:52 AM

but missing all of the debug logs..

dry-elephant-14928

02/04/2021, 1:53 AM

other than the crash.. I do see the error that we all dread NACK causing new tracks not to be added

wonderful-manchester-12533

02/04/2021, 1:55 AM

I can enable the debug logs

wonderful-manchester-12533

02/04/2021, 1:55 AM

is that configurable through the env variable?

dry-elephant-14928

02/04/2021, 1:56 AM

nope.. it's part of the config file

wonderful-manchester-12533

02/04/2021, 1:57 AM

😕 well we started deploying the docker image now

wonderful-manchester-12533

02/04/2021, 1:58 AM

any chance you can make that configurable through env var?

dry-elephant-14928

02/04/2021, 1:58 AM

k. also if you are building from source

wonderful-manchester-12533

02/04/2021, 1:58 AM

I can add a PR too if you want

dry-elephant-14928

02/04/2021, 1:58 AM

try the dump-sdp branch

wonderful-manchester-12533

02/04/2021, 1:58 AM

using the dockerhub image

wonderful-manchester-12533

02/04/2021, 1:58 AM

https://hub.docker.com/r/livekit/livekit-server

dry-elephant-14928

02/04/2021, 1:58 AM

dry-elephant-14928

02/04/2021, 1:59 AM

I do have a good clue tho.. but this is the same issue we saw before

dry-elephant-14928

02/04/2021, 1:59 AM

has little to do with # of participants.. more with nack causing issues

wonderful-manchester-12533

02/04/2021, 1:59 AM

gotcha

dry-elephant-14928

02/04/2021, 2:00 AM

you can pass --dev to enable debug

wonderful-manchester-12533

02/04/2021, 2:13 AM

cool just added that. I'll need to reproduce the bugs for those logs to surface, old debug logs are gone now

wonderful-manchester-12533

02/04/2021, 2:13 AM

also having redis problems so I can't reproduce for maybe an hour or so

wonderful-manchester-12533

02/04/2021, 2:14 AM

I'll send you a timestamp after I reproduce again

wonderful-manchester-12533

02/04/2021, 2:37 AM

just had a call with 3 people @dry-elephant-14928 https://app.logz.io/#/goto/080359faa21373bd1acc7f6c8b9cc94a?switchToAccountId=156134

dry-elephant-14928

02/04/2021, 2:38 AM

and which users stream went out?

wonderful-manchester-12533

02/04/2021, 2:38 AM

its super odd

wonderful-manchester-12533

02/04/2021, 2:38 AM

I think you should join a call with us and observe it

wonderful-manchester-12533

02/04/2021, 2:38 AM

easier than trying to describe it in text, too lossy

wonderful-manchester-12533

02/04/2021, 2:39 AM

can jump on meet.google.com then jump on livespot call together and observe behavior

wonderful-manchester-12533

02/04/2021, 2:39 AM

lmk if you have a few min for that

dry-elephant-14928

02/04/2021, 2:40 AM

I'm afk right now

wonderful-manchester-12533

02/04/2021, 2:40 AM

no worries

dry-elephant-14928

02/04/2021, 2:40 AM

will jump on later.. but a description would help regardless.. if it's not completely out.. but lossy.. that's good to know tpo

dry-elephant-14928

02/04/2021, 2:40 AM

too

wonderful-manchester-12533

02/04/2021, 2:40 AM

if thre's a time that works for you lmk

wonderful-manchester-12533

02/04/2021, 2:40 AM

total loss for some users

wonderful-manchester-12533

02/04/2021, 2:40 AM

they'll get a

TrackUnsubscribed

wonderful-manchester-12533

02/04/2021, 2:41 AM

even tho publisher never unpublished

wonderful-manchester-12533

02/04/2021, 2:41 AM

Copy code

"room event" "trackUnsubscribed" {"_events":{},"_eventsCount":2,"kind":"audio","trackSid":"TR_Pi5iAhEsAPMwd7cC5MxY7H","trackName":"","isMuted":false,"track":{"_events":{},"_eventsCount":0,"kind":"audio","name":"","isStarted":false,"isMuted":false,"attachedElements":[],"mediaStreamTrack":{},"sid":"TR_Pi5iAhEsAPMwd7cC5MxY7H"}} {"_events":{},"_eventsCount":5,"sid":"PA_s93t6vmiUJPvjcachJwhPh","identity":"0x73F8957f89AD035a880d2DacE78937cee9F01502","audioTracks":{},"videoTracks":{},"dataTracks":{},"tracks":{},"participantInfo":{"sid":"PA_s93t6vmiUJPvjcachJwhPh","identity":"0x73F8957f89AD035a880d2DacE78937cee9F01502","state":2,"tracks":[{"sid":"TR_Pi5iAhEsAPMwd7cC5MxY7H","type":0,"name":"","muted":false},{"sid":"TR_TsmF6RxucqEcyHLFxBXeP5","type":1,"name":"c922 Pro Stream Webcam (046d:085c)","muted":false}]}}

wonderful-manchester-12533

02/04/2021, 2:41 AM

for example

wonderful-manchester-12533

02/04/2021, 2:42 AM

different users are getting different track events, its very odd, hard to describe procedurally

wonderful-manchester-12533

02/04/2021, 2:45 AM

I'll have everyone on this test call record their screen and send them to you stitched together

dry-elephant-14928

02/04/2021, 2:47 AM

yeah I'll hop on later tonight. hanging with my kids for a bit

wonderful-manchester-12533

02/04/2021, 2:56 AM

sounds good, have fun!

wonderful-manchester-12533

02/04/2021, 4:06 AM

okay I recorded a video of the problem so we can keep debugging async

wonderful-manchester-12533

02/04/2021, 4:06 AM

top left screen is my computer

wonderful-manchester-12533

02/04/2021, 4:06 AM

bottom right screen is a friends computer

wonderful-manchester-12533

02/04/2021, 4:06 AM

I simulated a third user with a private tab

wonderful-manchester-12533

02/04/2021, 4:07 AM

notice the events that are fired on each window and what happens when publishers start to call in

wonderful-manchester-12533

02/04/2021, 4:09 AM

logs during that test span https://app.logz.io/#/goto/a00593087e45a5d374daab09252825ac?switchToAccountId=156134

wonderful-manchester-12533

02/04/2021, 4:09 AM

hope it helps!

wonderful-manchester-12533

02/04/2021, 4:10 AM

wrong video link here is the right one https://www.dropbox.com/s/aqtgprak1c3g38g/livekit-two-person-demo.mp4?dl=0

dry-elephant-14928

02/04/2021, 4:12 AM

ok, just getting on, thanks for putting the video together.. free to try it live too

wonderful-manchester-12533

02/04/2021, 4:15 AM

ok cool, if you'd like to I'm available now!

dry-elephant-14928

02/04/2021, 4:16 AM

sure, let's do it

dry-elephant-14928

02/04/2021, 4:17 AM

which session?

wonderful-manchester-12533

02/04/2021, 4:17 AM

https://meet.google.com/mmk-ozzn-xpt

dry-elephant-14928

02/04/2021, 4:17 AM

wait google meets or livespot?

4 Views

Open in Slack

Previous Next