LiveKit Community

you'd get the participant object in the second parameter. <https://docs.livekit.io/client-sdk-js/enums/roomevent.html#participantmetadatachanged>

we'll need to update the docs on participant event.

<https://docs.livekit.io/client-sdk-js/classes/participant.html#metadata|Participant.metadata> contains the new metadata

We were getting second argument as null though...

on RoomEvent it was not firing.. We got the callback for ParticipantEvent.metadataChanged <https://docs.livekit.io/client-sdk-js/enums/participantevent.html#metadatachanged>

ok, the participant isn't passed back on the ParticipantEvent, since you should have the participant that is being listened to

due to a bug it wasn't returned on the Room level callbacks, I'll submit a PR

I've been unable to get it to write incrementally - we wanted to do this to use multipart s3 uploads, and tried doing it with the filesink parameters. If you do find a way, I'd be happy to put it in

I'm getting some promising results by playing with the mp4mux plugin in my own pipeline.  Will definitely share what I end up with.

So, it's a minor change, but you can look at it <https://github.com/ct7t/livekit-recorder/commit/885dcbb10dbea08cd90a1c24c50c1470515938e5|here> .  I can't help but suspect you've tried it before - so let me know if you had any experiences with it.  I tried a number of other settings but can't see much of an effect on the result.

I'm playing with the resulting file a little more to see what it's meant for the metadata.

Ahh nice! I hadn't seen that parameter. I _think_ (this is all unconfirmed since I never got it flushing) the metadata won't be written until the pipeline finishes, but the metadata is at the beginning of the file since it's a fast start file. The plan for s3 multipart uploads was to reupload the first chunk once it finished

I'll pick this up tomorrow.  Our usual tools can't seem to get the length and frame rate from the "fragmented" file (they can with the regular recording), but I've got another tool that _can_ get that info.  Will take it up with my team in the morning.

If we can get it figured (and if you'd like) I'll put up a PR that has a recording option for it.

So, an update...

Using the `fragment-duration` worked in that I was getting the mp4 file built up incrementally, and it was playable (even if there was a crash) afterwards...but the audio/video was now hopelessly out of sync...in fact they seemed to have been interleaved between each other.  Setting fragments to 10s would get me about 10s of audio with frozen video, then 10s of video with no audio, and so on.  I tried a few other combinations of settings, but didn't really get anywhere.  (gstreamer is new for me - so will happily take advice)

There are some other plugins that exist to create sets of files (a few, in fact - making me wonder if those who know have already accepted that what I'm trying to do doesn't really work).  I'm looking at a couple of them: `splitmuxsync` (which uses `mp4mux` under the hood) and `hlssink2`.

My _preference_ is for a meeting recording to be a single file that can survive a crash of the system - but will happily settle for a set of files that represents a playlist that can replay the meeting.  I'm optimizing for retaining records.

<@U01UB0LHTTQ>, if you remain interested, I'll update again when I have more (may be a few days for this one).  Feedback welcome, as ever.

Ah, that’s a bummer. I’m not sure that a single file surviving a crash is possible though - it’ll be unplayable if it isn’t closed out properly. Have you run into crashes?

Sorry - didn't see this message last night.  No - haven't had an issue with crashes, but I accept they happen, and for this solution good record keeping is supreme.

In the mean time, had a productive couple of days.  <https://github.com/ct7t/livekit-recorder/compare/f/hls-recording|Here> is a branch where I've added the capability to support the saving of an HLS playlist of files.  It more or less fits the bill for what I need.

I'd call the code POCish rather than "done"; apologies for that, but there are a few reasons:
1. The original code has some assumptions of commonality between the "local" or "streaming" options that this solution does not share... ie. filepath and streaming bins both have the idea of an audio pipeline and a video pipeline sinking into a mux that produces a single src for the output, whereas my output element (hlssink2) takes in two sinks.  So I've tried to work around this with some extra conditional branches (not ideal, but other options seemed worse for now).
2. I initially wanted to give this option equal "status" to the File and RTMP, but I quickly realized it would require `proto` updates, and I thought best that I not try to change too much at once.
3. Not knowing whether you'd like to port this at all (and surely not in this form), I'm trying (perhaps failing) to keep this mode mostly "mergeable" with recorder evolution until this fork is no longer needed.
So, the compromise is thus: no changes to RTMP path, and if you configure a Filepath with an ".mp4" file, nothing changes.  However, if you configure a Filepath with an ".m3u8" target - it will use my new code; switching on `isPlaylist` just as it does `inStream` elsewhere (a bit awkward since HLS is also for streaming, but again - trying to make myself at home without rearranging all your furniture).  ;&gt;

Would love to know what you think, <@U01UB0LHTTQ>.  Have a great weekend!

I also experienced mixed results on the empty_timeout.

ah I think there are some confusion regarding what empty_timeout does. and maybe worth revisiting.

currently it's designed to hold the room open *before* anyone joins the room. If someone joins the room and leaves, a separate, non-configurable, timer gets fired. (~20s)

the logic is here: <https://github.com/livekit/livekit-server/blob/master/pkg/rtc/room.go#L377>

what's the use case that you guys have in mind? is recorder causing the room to close before a real participant has?

Excellent clarification, <@U01KZ61GV5Z>.  Thank you very much!

In my case - there are going to be a limited set of conceptual rooms and they are well known in advance, but not already running on the livekit server.

Let's suppose one is `boardroom`.  When people (Jack and Jill) join the room and it's created as a side effect; I get a `room-started` webhook call, and so there's a recording at play.  When Jack and Jill leave the room, the recorder detects that there are no visible participants left, and the recording ends.

But there appears to be a delay before the room is really finished.  And if some new people (Tom and Jerry) join `boardroom`, there is no `room-created` webhook call.

So, I wind up with a problem.  The recording that was running has stopped, and (without tracking participants on my side) there isn't a clean indicator to start a new one.

My expectation is that with the room created as a side-effect of the first person joining, it would be finished along with the last person leaving.  This also appears to be the assumption of the recorder, and the ultimate behaviour - but there's a significant delay between the last `participant-left` call and `room-finished` call.

in my case: once I receive the room-finished event, I mark the room as "finished" in my backend db, and don't let any users re-join that room anymore (I use unique random generated room names). So a configurable timer for the room-finished event would be much appreciated, that way I can control the time we let a user rejoin the room if he accidentally disconnected and happened to be the only one in the room waiting for someone else to join.

Sounds like <@U02HN8GPUBY> would like a grace period, and I'd like to be able to rule one out - but a deterministic "this is how long the room stays open after the last user leaves" would do both jobs.

<@U01KZ61GV5Z>, would you mind giving a pointer to the non-configurable timer you mentioned?

Thanks for sharing the use cases. It makes total sense. I've filed an issue <https://github.com/livekit/livekit-server/issues/241|here> and will think on this a bit.

There are some tricky edge cases to this, for example, when the only participant in the room refreshes their browser. We don't want to close the room immediately after the last person leaves in that case.

this is likely a bug in the latest SDK release. we'll take a look

fixed. livekit-cli `v0.6.3` resolves this issue

Awesome! it works now. thank you, <@U01KZ61GV5Z>

can you print out the TURN url it's trying to connect to? how is livekit server configured?

what do you mean? there are options to createLocalAudioTrack and createLocalVideoTrack to set the features and constraints that you may need, is that what you want?

I already have application which uses WebRTC video call capabilities and tensor flow for analysing video. For this I create local audio/video tracks and share them between TF and Webrtc. Also can be scenario when application can be simultaneously in a few rooms.

For this we had our own solution (only Google WebRTC library on Android Side) and for media server was used pion.  Now we want to try livekit

<@U02HN8GPUBY> Can you please show the sample to setting these for video and audio (constraints, resolution,. codes, bitrate and such...) Thanks.

<@U02Q369F4BF> check this url, it has an example on how to use them

<https://docs.livekit.io/guides/room/publish#advanced-track-management>

here's the docs for the audio track options:
<https://docs.livekit.io/client-sdk-js/interfaces/createaudiotrackoptions.html>

and the video track options:
<https://docs.livekit.io/client-sdk-js/interfaces/createvideotrackoptions.html>

<@U02Q369F4BF> oh, forgot to mention, there are also options when you publish the track to the room

```await room.localParticipant.publishTrack(track, trackPublishOptions)```
and the docs for those options can be found here
<https://docs.livekit.io/client-sdk-js/interfaces/trackpublishoptions.html>

Thanks <@U02HN8GPUBY> I do not see audio codec anywhere ? Is it default to OPUS ? Thanks.

<@U02Q369F4BF> I think it uses opus, I'm kinda new to livekit, maybe <@U01KZ61GV5Z> can tell?

<@U02Q369F4BF> I replied on the other thread:
<https://livekit-users.slack.com/archives/C01KVTJH6BX/p1639074958448900?thread_ts=1638983374.441900&amp;cid=C01KVTJH6BX>

<@U02Q369F4BF> this is managed automatically if you pass in `autoManageVideos: true`

client will observe video elements you are rendering to and automatically request the layer with the right resolution from the server

<@U01KZ61GV5Z> Hi thanks. How would "client/server" do that actually ? What "automatic" really mean in scenarios ? Said if team decide to put one on-stage , I would want his resolution/bitrate as high as possible (1920, simulcast of course) while the rest on grid (how would client/server know ?) should be switched to much less bitrate and resolution (480 maybe) ?

as long as you are using `LocalVideoTrack.attach`, it would monitor the size and visibility of your video elements and automatically ask the server to send down the right layer that matches the dimensions.

so if the video component representing a user was sized to a small thumbnail (say 300x200), the resize observer will be triggered and request for a stream that's appropriate for 300x200.

we'll publish a blog post about this soon, along with details of how it works.

I use a couple of recorders on the same VM without problems, but I record in very low resolution (640x480@10 fps, 750 kb/s)

this is how I configure them in docker-compose.yml

```services:
  recorder1:
    restart: always
    image: livekit-recorder:custom
    build:
     context: livekit-recorder
    cpus:  "3"
    expose:
      - 8989
    environment:
      SERVICE_MODE: 1
      LIVEKIT_RECORDER_CONFIG: |
        api_key: "${LIVEKIT_API_KEY}"
        api_secret: "${LIVEKIT_API_SECRET}"
        ws_url: "${LIVEKIT_URL}"
        health_port: 8989
        log_level: debug
        template_address: "${REC_URL}"
        redis:
          address: redis:6379
        file_output:
          azblob:
            account_name: "${AZBLOB_ACCOUNT_NAME}"
            account_key: "${AZBLOB_ACCOUNT_KEY}"
            container_name: "${AZBLOB_CONTAINER_NAME}"
        defaults:
          width: 640
          height: 480
          depth: 24
          framerate: 10
          audio_bitrate: 96
          audio_frequency: 22050
          video_bitrate: 750```
then create an `.env` file with all the variables being used in the docker-compose.yml

btw, I use a custom image of the recorder (to disable the dct8x8 feature I mentioned in the other thead) so I just clone the livekit-recorder repository in the same directory where the docker-compose.yml file is at, make the changes I need, and run `docker-compose up -d --build` to build and run my custom image

<@U01UB0LHTTQ> and <@U02HN8GPUBY> My yaml and json are dynamic and call docker run,,,from Python daemon. And yes I only get 16 FPS :disappointed: ...single docker run, Xeon 54 cores, 1T RAM and all SSD, no load.

You probably want to use the data channel <https://docs.livekit.io/guides/room/data/#data-messages>

also if you run the client-sdk-js example (instructions at bottom of readme) there is a chat/messaging example: <https://github.com/livekit/client-sdk-js>

Thanks! I'm checking that out now. That said, we use the data channel for sharing files such as photos and videos, which could be arbitrarily large. Those would occupy the data channel, requiring chat messages to wait an arbitrary amount of time to be sent, which is obviously not optimal for a realtime application like chat. It seems to me that the "right" way to do this would be to use the same signaling method that's used for e.g. ICE negotiation. I'm guessing LiveKit is using some flavor of websockets? Any thoughts on that? Would the team consider some PRs if we took that on ourselves?

reading thru, looks like you have access currently in livekit to both a lossy and reliable data channel. For now you could split your data between those if one is less important? With webrtc though you can create as many data channels as you like and livekit looks like they're building on top of pions webrtc.DataChannel so you'd have to do some work there if you wanted more data channels, but def do-able

I am a noob here on livekit though so someone else might have a better answer :smiley:

Oh interesting, so the reliable vs lossy parameter passed to the sendData method actually routes the data through two different data channels. That makes sense given reliability is set when the data channel is first established. So yes we could potentially send media via the unreliable data channel and text signals via the reliable. Thanks Dan!

(btw for any core LiveKit devs reading this, I'd still be keen on eventually routing plain text signals and chat messages via the existing but private signaling mechanism, so I'd appreciate thoughts on that too)

I think DataChannel in livekit maintain sequential data transfer. In this you won't get next data until you finished to load previous data. Sending larger file using DataChannel seems a problem. You can easily use websocket or other file uploading system for file transferring. Later can share download links/file link as chat message. This way it will be faster.

Make sure your ports are defined and opened properly... Other than that not sure

it's likely a server configuration/port/ip issue. Check it out with <https://livekit.io/connection-test>

That connection test seems fine:
```Connecting to signal connection via WebSocket✓
Connected to server, version 0.14.2.
warning: Server is insecure, clients may block connections to it
Establishing WebRTC connection✓
Can connect via TURNskipped
warning: No TURN servers configured.
Can publish audio✓
published 23 audio packets
Can publish video✓
published 179 video packets
Resuming connection after interruption✓
warning: reconnection is only possible in Redis-based configurations
warning: Not able to reconnect
TEST COMPLETE```

it seems like locally the example in the js sdk will work some/most times but not always

are you running the server via docker by any chance?

```  docker run --rm \
    -p 7880:7880 \
    -p 7881:7881 \
    -p 7882:7882/udp \
    -e LIVEKIT_KEYS="xxxxxx: xxxxx" \
    livekit/livekit-server \
    --dev```

so Docker is a bit strange with networking on dev laptops.. you'd want to pass in `--node-ip`

with your machine's IP, instead of the VM's

i did have that set in a previous iteration of this testing, but will try again

Yep, that was it! For future reference if there is no node-ip (or the wrong ip) set it did work for me in a normal chrome window, but would timeout in private chrome window and safari/firefox..

Chrome (non-incognito) has some magic that automatically handles the docker IPs.. no other browser seems to like it

you are sending it to a room SID, instead of participant sids: i.e. `RM_xxxxx`

you could turn off TURN/UDP by setting `udp_port: 0` in your server config. that'll ensure TURN/TLS is the only option available

there's a function in the server-sdk to explictily <https://docs.livekit.io/server-sdk-js/classes/roomserviceclient.html#createroom|create rooms> which allows to customize the room settings (emptyTimeout, maxParticipants, nodeId, etc). But as the documentation says "Explicit room creation is not required, since rooms will be automatically created when the first participant joins"

the only use-case for roomCreate I can think of, is for explict room creation with non-default room settings.

for me it would make more sense if automatic room creation could be disabled, so we can have more control over room's lifecycles.

I mentioned something related to this at this thread <https://livekit-users.slack.com/archives/C01KVTJH6BX/p1637129406216600>

I have an add-on question here, can we enable/disable simulcast on the fly?

<@U02NS4V998Q>, yes you can change a participant's perms via a server API call <https://docs.livekit.io/guides/server-api#updateparticipant>

<@U02Q4J8KS85> simulcast is set per track, and you won't be able to turn it on/off dynamically at the moment.

what is your use case here? it might play well into a feature that's currently on our roadmap

i'll have 80+% of users in a room only ever sending a thumbnail sized video. Every once in awhile these users would be upgraded to a 'stage' where i'd then want simulcast. But I'd prefer to have them locked at small/low bandwidth until that happens (IF that happens)

yup, that's exactly what we are working on.. dynamic simulcast that keeps track of if a layer is needed by any subscribers, and turning them off on the publisher side when unused.

we expect to have this sometimes in the next month

nice! I'm guessing that will change your benchmarks decently, ~do you guys have a plan for updating that section of your docs at any interval?~ Actually I guess this is a pretty niche case, so prob doesn't make sense.

we have a couple of updates queued up :slightly_smiling_face: recently we were able to get 5 publishers (at 540p) + 1000 subs on a 16core node.

dynamic simulcast on the publisher side should help, but most of the performance gains came from dynamic video layer selection on the client side (this is already released in all of our SDKs). since clients are requesting size-appropriate layers from the server, the SFU doesn't have to send out as many bits.

awesome! I have been thinking about more publishers and fewer subs, so dynamic simulcast will help me a lot. Thanks!

<@U01KZ61GV5Z> oops, didn't see that - thanks!

When you update the `publish` permission, does it automatically unpublish the participant's tracks?

no it does not, it only impacts new publications

though I think it should perform these steps automatically.. mind opening an issue?

hey Gabriel, are you referring to single node performance? I think this is hard to answer as it's highly use case dependent.

it comes down to, how much data does the SFU need to send out at any given point. What is your use case? in terms of number of people that needs to subscribe to a single stream

we have a virtual world that has hundreds of simultaneous users. sometimes they all come together in an auditorium, sometimes they are broken up around small groups in the virtual space

<@U02RANXR1UY> are those users publishing audio and video or audio only?

most of them just audio - but everyone is also sending data over datachannels

<@U02RANXR1UY> sorry for the delayed response here! We just updated our docs with a benchmark for larger numbers of publishers. <https://docs.livekit.io/deploy/benchmark#audio-only|https://docs.livekit.io/deploy/benchmark#audio-only>

If groups come together for, say, a presentation in your virtual space, you can use selective subscriptions to manage which streams they can/should be able to access.

What we'd like to do in the near future is publish more comprehensive benchmarks around specific classes of use-cases.

Hey Sayan, this isn't publicly available as it's part of our website. The logic is fairly straight forward though (uses the JS SDK to perform tests). For example, for the TURN test

```this.room = await connect(this.url, this.token, {
        rtcConfig: {
          iceTransportPolicy: 'relay',
        }
      });```

hmm, that’s odd. just to verify, is the room object being held in a ViewModel to preserve it across config changes?

I’m not able to repro this with the sample app, do you have a sample I could try this against?

I've a shared view model that is used by 3 fragments.
    `private val liveSessionViewModel:  LiveSessionViewModel by activityViewModels()`
	
This viewModel contains a room variable.
    `val room = MutableStateFlow&lt;Room?&gt;(null)`

Maybe there is some issue with my code flow. I will try some more.

The code is in a larger project. I will try to extract and share a smaller sample if I am not able to resolve this.

Thanks.

one other thing to verify: this only happens on config change/rotate, and doesn’t happen on initial connect?

Not on initial connect. I am getting streams and able to visualize them... So the basic use case is working...

I had one more issue. May be both are related.. I tried to call room.disconnect() in the fragment onDestroy method, to handle user pressing back button from the live fragment.. That also was crashing... I don't have exact stacktrace with me now.. I can share that as well....

hmm, something very weird’s happening then. local participant gets initialized upon the initial connect, and never gets deinitialized afterwards

ok.. most probably in my code then.. I will keep you posted... What is the right place and way to disconnect in an app based on fragments?

Our app is based on standard Android architecture similar to this app. <https://github.com/android/sunflower>

mmm, should just be able to disconnect the room in the `onCleared` method in the ViewModel containing the room object

that’s the appropriate place for viewmodels to clean up their own objects (`Fragment.onDestroy()` will have a separate lifecycle from the view models)

This is the other exception I am getting on calling room.disconnect()

    `java.lang.IllegalArgumentException: NetworkCallback was already unregistered`
        `at com.android.internal.util.Preconditions.checkArgument(Preconditions.java:47)`
        `at <http://android.net|android.net>.ConnectivityManager.unregisterNetworkCallback(ConnectivityManager.java:3388)`
        `at io.livekit.android.room.Room.handleDisconnect(Room.kt:256)`
        `at io.livekit.android.room.Room.onDisconnect(Room.kt:416)`
        `at io.livekit.android.room.RTCEngine.onLeave(RTCEngine.kt:520)`
        `at io.livekit.android.room.SignalClient.handleSignalResponseImpl(SignalClient.kt:409)`
        `at io.livekit.android.room.SignalClient.access$handleSignalResponseImpl(SignalClient.kt:37)`
        `at io.livekit.android.room.SignalClient$onReady$1$invokeSuspend$$inlined$collect$1.emit(Collect.kt:136)`
        `at kotlinx.coroutines.flow.SharedFlowImpl.collect(SharedFlow.kt:351)`
        `at kotlinx.coroutines.flow.SharedFlowImpl$collect$1.invokeSuspend(Unknown Source:15)`
        `at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)`
        `at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)`
        `at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)`
        `at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)`
        `at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)`
        `at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)`

Do you think this is also because of the codeflow?

This exception is coming after getting a successful callback in onDisconnect(room: Room, error: Exception?) method.

Ah, looks like disconnect was called multiple times

Would you be fixing/handling this in the library?

yeah I’ll fix it to be more robust against this

<https://github.com/livekit/client-sdk-android/pull/32>

Your API requests will need to be secured with certificates. Browsers will refuse to connect to it if not secured.

If you are putting it behind a load balancer, the load balancer should terminate SSL for you. This can be on whatever port you want, but 443 is recommended.

So, if you are running the server on the default port 7880, your load balancer should terminate SSL on port 443 and proxy that traffic to port 7880.

There's more info on the ports here, in case you didn't see it: <https://docs.livekit.io/deploy/ports-firewall>

Yeah, I've read thru that stuff and I'm using the live kit helm charts (other than I'm using a gke managed cert for the ingress).. Is there any way for me to debug the proxy? I'm getting a 200 response from my lb

I haven't used the helm chart version, so I'm probably not the best to help on that part, sorry! The ingress itself should be logging either to its console (if you are running your own ingress) or likely to cloud logging. The LiveKit server will also log to console by default and might provide some insight, like if the traffic is even making to it.

hey <@U02Q4J8KS85>, If a certificate is attached, you should see a 443 port on the load balancer.

did you configure it with `secretName` in there? <https://github.com/livekit/livekit-helm/blob/master/server-sample.yaml#L52>

i'm using a managed cert with gke, so i built my own ingress. Having troubles with it though, so will keep debugging on my end

ah ok, I believe our setup doesn't use managed secrets. (due to a limitation). Here's our ingress config if it's helpful: <https://github.com/livekit/livekit-helm/blob/master/livekit-server/templates/ingress.yaml>

you'd want to point the HTTP to livekit-server deployment, on port 7880 or `livekit.port`  in your values.yaml

ah, any ideas what the limitation was on managed certs?

good question.. it's been awhile so I'm having trouble remembering why.. :sweat_smile:

there's no reason why it couldn't work though.. it's worth taking a look at it.

if you are up for modifying the existing ingress template to work, I'd welcome a PR :slightly_smiling_face:

for the embedded TURNS server, we would require a secretName, because livekit is performing TLS termination.

for clarity --&gt; the LB needs to map to livekit.port (7880) for both 443 and 80 incoming?

Annddd...this just started working. :joy:  Looks like most of the issue was with google provisioning the cert..took a looong time (~4 hours) and gave me ssl cipher mismatch errors. I'll get to a PR for a managed cert at some point. Thanks!

Could you share the steps that you'd taken to reproduce this? what server version, client SDK/versions are you using?

What were the participants doing when it happened?

Very hard to reproduce actually...During the day suddenly the docker was frozen with the above error ! Will try... The "node" seem to do with distributed model but we have only one LiveKit server run.

which version of server are you running? This appears to be a deadlock somewhere. We take these very seriously given the nature of the system.

In a multi-node setup, the system is designed to self-heal (traffic will not be routed to nodes that aren't alive), to avoid the cascading effect.

I use livekit-server latest docker as of yesterday. Did a load test also, 10 pub and 25 sub, works well at 40ms latency summary while doing another chat room in parallel. CPU ~9%.

livekit-livekit-1  | redis: 2022/02/20 11:33:30 pubsub.go:599: redis: &amp;{%!s(*redis.PubSub=&amp;{0xc0003960e0 0x7003c0 0x702a60 {0 0} 0xc00054eb40 map[rtc_channel:ND_a8XMTO5W:{} sig
nal_channel:ND_a8XMTO5W:{}] map[] false 0xc0000c4120 0xc000313620 {1 {0 0}} 0xc000384e80 &lt;nil&gt;}) %!s(chan *redis.Message=0xc000313560) %!s(chan interface {}=&lt;nil&gt;) %!s(chan str
uct {}=0xc0000c4300) %!s(int=100) %!s(time.Duration=60000000000) %!s(time.Duration=3000000000)} channel is full for 1m0s (message is dropped)
livekit-livekit-1  | redis: 2022/02/20 11:34:30 pubsub.go:599: redis: &amp;{%!s(*redis.PubSub=&amp;{0xc0003960e0 0x7003c0 0x702a60 {0 0} 0xc00054eb40 map[rtc_channel:ND_a8XMTO5W:{} sig
nal_channel:ND_a8XMTO5W:{}] map[] false 0xc0000c4120 0xc000313620 {1 {0 0}} 0xc000384e80 &lt;nil&gt;}) %!s(chan *redis.Message=0xc000313560) %!s(chan interface {}=&lt;nil&gt;) %!s(chan str
uct {}=0xc0000c4300) %!s(int=100) %!s(time.Duration=60000000000) %!s(time.Duration=3000000000)} channel is full for 1m0s (message is dropped)
livekit-livekit-1  | 2022-02-20T11:35:09.550ZINFOlivekitservice/rtcservice.go:182server closing WS connection{“room”: “stark-tower”, “participant”: “vivek”,
“connID”: “CO_gFZJV94rfWyn”}

I was in a call suddenly docker freezed with this error

2021-12-16 15-13-49.mkv

:eyes: Whoa, finally someone found our Tron-mode Easter egg.

whoa, that's really odd. Looks like it happens only during rotation?
if you don't rotate, and if it starts out in the other orientations (and without a rotation during the session), do the same artifacts present themselves?

<@U01KZ61GV5Z> it's not triggered by device rotation, if I start in portrait it already looks color shifted, same thing if I start in "landscape left", it only works when the orientation is "landscape right", either by starting in that orientation or rotating to that orientation.

that's odd.. which device are you seeing this with?

it's a Samsung Galaxy S7 (SM-G930F) running android 8.0.0, I haven't tested in any other devices yet. I'll give it a try on iOS and let you know.

sounds good.. to be clear, you are seeing this only on the LOW layer? and the higher layers look ok?

<@U01KZ61GV5Z> how can I tell for sure which layer is being rendered? I set the publication.videoQuality in the TrackSubscribedEvent but sometimes the resolution seems to change over time

on web, you can use the example app that comes with the SDK. then set in JS console. `currentRoom` is exposed to make it easier to debug

i.e. this is almost certainly an issue with the publisher, so using JS sample could give you a way of validating that.

I tested it on iOS, and it doesn't happen there, I'll keep testing and letting you guys know.

hey <@U02HN8GPUBY>, following back up this. does this still happen for you? we've upgraded the underlying webrtc sdk since then, and I can't repro it any more

Drone camera? that sounds AWESOME! does it appear as an android device then?

You should be able to specify a device when creating a track with <https://docs.livekit.io/client-sdk-android/livekit-android-sdk/io.livekit.android.room.participant/-local-participant/#242126939%2FFunctions%2F-885714121|createVideoTrack>. It should let you pass in a <https://docs.livekit.io/client-sdk-android/livekit-android-sdk/io.livekit.android.room.track/-local-video-track-options/#-49501172%2FProperties%2F-885714121|deviceId>.

Thanks, David. The drone camera doesn't appear as an Android device. We receive the video bytestream through the DJI SDK and then create frames from that.

I'll attempt to create a LocalVideoTrack via the constructor and add the track.

the stream is h264 encoded (<https://gstreamer.freedesktop.org/documentation/x264/index.html?gi-language=c|gstreamer x264enc>) and converted to flv format (<https://gstreamer.freedesktop.org/documentation/flv/flvmux.html?gi-language=c|gstreamer flvmux>)

<@U01UB0LHTTQ>: I assume this works for you guys. Can I ask what kind of machine you run the tests on

I just have a MacBook Air perhaps I don't have enough cpu to do the encoding as I see a lot of backwards timestamps

I run them on a 2018 macbook pro. The backwards timestamps are normal (caused by using live audio and video sources), as long as the difference in timestamps is ~100ms or less

to narrow down the issue you can try running <https://github.com/aler9/rtsp-simple-server> and serving the rtmp stream there (our <https://github.com/livekit/livekit-recorder/blob/main/test/integration_test.go#L97|integration test> uses this), and then you can open that rtmp stream with vlc to check if there’s video

<@U01UB0LHTTQ>: ok finally found the issue. I believe this is related to somebody else commenting that the mp4 videos created by the recorder can't be played in QuickTime. I had to convert the pixel format to get a hls stream with video. Not sure if this is a setting you can change in gstreamer. But this may help someone else out later

Extra cmd line parameter to ffmpeg was -pix_fmt yuv420p

ah, good find. I released a compatibility update that should fix this, but it’s waiting for the next livekit-server release

If I use this to convert the video output by the recorder the resulting video also plays in QuickTime 

Ok cool when is the next release slated for?

sounds like you are testing extreme low bandwidth. This is something we've improved significantly in the v0.15 line (soon to be released). Would be great if you can test it on that version when it's released.

There isn't a way to use an external TURN server yet, but that's a great suggestion. I've filed ticket: <https://github.com/livekit/livekit-server/issues/271>

<@U01KZ61GV5Z> Absolutely YES will be the FIRST to test I think please let us know. :sweat_smile:

try to do a cli load test today, but got a websocket handshake error, just wondering if anyone else had this issue, i have made an issue on github <https://github.com/livekit/livekit-cli/issues/8>, any help would be awesome, thanks advance

I can sort of answer my own question..here's some load tester results from my home and my office, home is obviously much slower.
```Summary | Tester | Tracks | Latency      | Total OOO     | Total Dropped
        | Sub 0  | 4/4    | 112.827426ms | 910 (2.523%)  | 767 (2.127%)
        | Pub 0  | 2/2    | 109.257942ms | 198 (1.1%)    | 140 (0.778%)
        | Pub 1  | 2/2    | 109.857324ms | 0 (0%)        | 23 (0.127%)
        | Total  | 8/8    | 111.165071ms | 1108 (1.536%) | 930 (1.29%)



Summary | Tester | Tracks | Latency     | Total OOO   | Total Dropped
        | Sub 3  | 6/6    | 80.961819ms | 5 (0.01%)   | 4 (0.008%)
        | Sub 6  | 6/6    | 79.175272ms | 7 (0.015%)  | 4 (0.008%)
        | Sub 5  | 6/6    | 79.125286ms | 12 (0.025%) | 10 (0.021%)
        | Sub 7  | 6/6    | 79.430945ms | 8 (0.017%)  | 4 (0.009%)
        | Pub 2  | 4/4    | 78.698255ms | 6 (0.017%)  | 3 (0.008%)
        | Sub 0  | 6/6    | 78.851007ms | 6 (0.011%)  | 3 (0.006%)
        | Sub 9  | 6/6    | 81.234329ms | 9 (0.02%)   | 4 (0.009%)
        | Pub 1  | 4/4    | 78.44503ms  | 12 (0.033%) | 7 (0.019%)
        | Sub 4  | 6/6    | 81.225747ms | 11 (0.022%) | 5 (0.01%)
        | Sub 8  | 6/6    | 79.242194ms | 7 (0.015%)  | 4 (0.009%)
        | Pub 0  | 4/4    | 78.953186ms | 0 (0%)      | 0 (0%)
        | Sub 1  | 6/6    | 78.885529ms | 5 (0.01%)   | 3 (0.006%)
        | Sub 2  | 6/6    | 78.960627ms | 7 (0.014%)  | 4 (0.008%)
        | Total  | 72/72  | 79.511174ms | 95 (0.016%) | 55 (0.009%)```

hey <@U02Q4J8KS85> I think you might be seeing limitations of the your local network. if running the server in GCP, and multiple publishers on your local machine (i.e. browsers), it could easily use several megabits of upstream.

browsers will try to accommodate the congestion,  but will show artifacts/pause video entirely when it gets severe.

could u do me a favor and paste the code in a gist instead of in slack? it's hard to follow here :pray::skin-tone-3:

ok here is my gist

<https://gist.github.com/Kangyeongwoo/5227f8dade4a8352cec3ec4bc1b6844e>

and i used  flutter-sdk :  livekit_client 0.5.4

could you remove the segment from the message? we will forward it to the right engineer to investigate

Hey david i solve the problem using version 0.5.5 under development.

i guess flutter_webrtc version or screenShare’s preset is the cause.

But now it works well

Thanks for livekit Team.

you can use stats exposed by webrtc-stats on this. we expose it via `room.engine.subscriber.pc`

Hi, may I know your Xcode ver?
Currently, the example app compiles with `main` branch of <https://github.com/livekit/client-sdk-ios> .
Can you pull both repos into a same parent dir. Then open `LiveKitExample.xcworkspace`

```public func track(_ track: VideoTrack, videoView: VideoView, didUpdate size: CGSize) {```
Cannot find type 'CGSize' in scope

it's fixed by adding
*import* CoreGraphics

into the TrackPublication.swift

&gt; the example app compiles
do you compile example app for macOS or iOS as it works for macOS without any issues

oh yes, I pushed that fix just 20 mins ago :grin:

image.png

that's odd.. TURN shouldn't be needed as long as your server's ports are exposed.

this looks more like an issue with publishing on mobile. maybe certain permissions weren't granted for the tester? does the same thing happen when you run the tests on a desktop browser?

no, it was only tested on mobile browsers, I'll ask for re-test on same network on desktop browsers. But if it were permissions, it wouldn't fail randomly, this gets random results when the test is repeated, also when using the app I built on flutter, the devices randomly can't receive or send audio or video (usually it fails only 1, audio OR video)

btw, is there a source code for that livekit tester? I'd like to know under what conditions it triggers each message, so I can try to implement similar features on my app and at least notify the user when it happens

ah I see, this is for a user. that's really strange..

are you running the server with redis or without?

with redis and only 1 node, I just enabled port 7881 on the docker and it seems to work better now

ah ok, that sounds like ICE related.. when UDP can't get through, we'd use ICE/TCP, which is on 7881

If this is video on mobile/Safari, keep bitrate low may help (500)...