LiveKit Community

Hi <@U03KC2Y3J06>, could you explain a bit more about what this customer is trying to do? How are they planning to save bandwidth with cascading? Are they looking to somehow combine their private and public cloud deployments?

picture_cascade.jpg

Please reference the picture below, the left side is the deployment in private cloud, the right side is the deployment with private and public cloud cascade. When 2 users connect to SFU from public network, the right side solution has less bandwidth usage.

<@U03KC2Y3J06> we do plan to one day introduce a form of cascading into LK OSS, but discussions around this are still very early.

image.png

i'm going to try the advance track management to see if it works. i'll get back to u on that

I think we should surface this so you don't have to use `RTCVideoRenderer`. Will check with <@U02GAC58A79> on this

we can add a nullable `MediaDevice? outputDevice` optional parameter to
<https://github.com/livekit/client-sdk-flutter/blob/main/lib/src/widgets/video_track_renderer.dart#L27>

Setup an output device enumerated from here <https://github.com/livekit/client-sdk-flutter/blob/main/lib/src/hardware/hardware.dart#L71>

I was able to get the unity webapp running building in unity webgl, but couldn’t figure out how it integrates with react and also couldn’t get the webcam to show up correctly

Hey Peggy, I’m not sure about how to integrate Unity with React. There is a pretty popular repo for that: <https://react-unity-webgl.dev/>. I never used it.

For the webcam, do you have any error in the console, is the issue inside the JoinScene?

If you want to use the LiveKit functions inside JS and C# at the same time, there is a small example here: <https://github.com/livekit/client-sdk-unity-web/tree/main/Samples~/JSExample>

A quick question for dev-tools of chrome:
How can this arrow auto collapsed when open devtool

Hey <@U04N2ULR17C>, Theo will announce and share it when it’s ready. We’re getting close, but there’s still a bit more to do before we release a developer preview. We appreciate your patience. :pray::skin-tone-4:  

<@U04N2ULR17C> this is a repeat of <https://livekit-users.slack.com/archives/C048FRL1N2C/p1676977754012759>

Please have a bit more patience. Also, please refrain from tagging individuals unless you are in an existing conversation with them. :pray:

Theo would be able to spend more time to work on the SDK if he's spending less time reading slack notifications :slightly_smiling_face:

Hello guys! I am using `@livekit/components-react` and i want to prevent users closing their tabs accidentally. I am using `window.beforeunload`  event to display warning if this user tries to close the browser or the tab. But when this event triggers inside of the room user disconnects from the room immediately. Before waiting for the task i assume. Is there way to prevent this behaviour.

<https://github.com/livekit/client-sdk-js/blob/main/src/room/Room.ts#L545>

*EDIT*: *Sorry guys i just saw that it is already mentioned and its already fixed.* 

<https://github.com/livekit/client-sdk-js/pull/580>

webrtc_internals_dump.txt

Where is the user connecting from? do you have an example session (link from analytics dashboard)

hi <@U01KZ61GV5Z>, this is link to the latest session <https://cloud.livekit.io/projects/p_62ja5e4q3d9/sessions/RM_3grzSZfA48oX>

User that have problems with connection is identity: 1127856

Hi <@U01KZ61GV5Z>. Sorry to disturb you. Maybe you have some ideas or advises how to debug, or maybe you have some extra information from the servers? After few days of aggregating logs it looks like that users from Russia have such problems with connection.

Maybe will be helpful for someone. After investigation of the client side, we detected that Google Chrome can not get Ice candidates, even if we use third party STUN (for example Google stuns or Twilio).

Next that we noticed is that Firefox can connect to the room.
Also Janus demo (<https://janus.conf.meetecho.com/audiobridgetest.html>) works well in Google Chrome.

This is config for iceServers that we use

Hey Oleksander, sorry for the late response. I think what happened here is certain Russian ISPs blocked DTLS packets between our server and the user.

We use Pion underneath, the same framework used by TOR Snowflake. This means that our DTLS signatures look very similar to Snowflake's..

We've built a couple of things to get around it, one of them being an automatic TCP/TLS fallback. That should have been deployed in the last few days.

If you find that the user continues to have issues, could you tag me again in this thread?

Hi David, I'm very glad to get your response, thank you!
I will check our error logs and give feedback in a few days. Thank you!

Hi <@U01KZ61GV5Z> it's a pity, but I still see errors from the clients side. Please give me feedback, maybe some extra log needed? I can add extra logs to our frontend code, just give me know if something needed.

Will, I'm not seeing anything abnormal with this particular Egress. The logs aren't super clear here on our end, but it appears that we've received a StopEgress request at exactly two minutes after the egress was started.

CC <@U03H6MNQ9G9>

That's really strange, I've completely commented out our only line of code that calls stop egress and it's still happening

I even tried streaming to multiple different services (twitch and youtube) and they all stop after a couple minutes

Here's the ID of my latest test going to twitch `EG_eAAaRVWqHY9x`

I see.. I'll check with the team to see if that could have come from somewhere else.

<@U03DTLWA19P> we found an issue. Web Egresses are getting cleaned up prematurely. We have a fix and will have it deployed early next week.

Awesome, that's great news! Thank you for the quick fix!

Pasted Graphic.png

LocalParticipant state after reconnection:

Another case when participant tries to connect under limited bandwidth. Tracks are published but they are not visible to others. localParticipant.tracks displays a camera track, but remoteParticipant.videoTracks is empty:

Hi ! The problem is still here and definitely looks like an egress problem.

There is event log. Look at the timers. Egress start 10:53 and finished 11:24. So there is about 30 minutes of recording. But in MP4 video first ~8 minutes is white screen with only sound.

Here is a new example link <https://media.fitol.io/prod/sessions/5457a6c0-f2a6-450c-8401-84f5f501a11d/recording/e3ac1768-90fa-4ae2-85b2-b980452a0ff9.mp4>

<@U01KZ61GV5Z> can you take a look please. It is quite important for us

Hey guys, can you share the Egress ID that was used here? CC: <@U03H6MNQ9G9> to take a look.

I can see a few egress IDs in the screenshot. can you share the request that you are making? for both mp4 as well as HLS

may be problem in deprecated segments, but I didn’t found any other examples how to do it

We tried reproducing this but haven't been able so far. How reproducible is it? i.e., does it always happen with the segments version, or does it sometimes happen with the mp4? Is the delay after which the media appears always similar? Do you have any other egress session to share beyond the one above to see if there is anything interesting in the logs for these?

1. It’s reproducing not every time, but time to time. Aprox one of 5 egress 
2. It happens with mp4 and with HLS
3. The delay always different from 3 to 8 minutes
4. I will reproduce few more tomorrow. It’s to late for now
5. I can recommend to start room, start stream. And the just start egresses every minute. One of them will be corrupted for sure

Oleg, I think we've managed to reproduce the issue. This seems to be a bug with the `single-speaker` template. The "white" background you've seen is due to it incorrectly choosing to display the bot user. We'll get a fix out for this soon. In the mean time, you can do one of two things to work around it:
1. ensure that only the user to be recorded is in the room
2. ensure that the participant to be displayed is speaking

<@U01KZ61GV5Z> Hi, I hope you are doing fine, can you give an update on the status of the bug fix,

<@U01KZ61GV5Z> ping, in case you missed my message)

Hey guys, we are working on a fix. it should be deployed sometimes next week

Thank your, is it possible to make it at the beginning of next week?

Hi <@U01KZ61GV5Z>, can you tell us status of fix deployment

sorry, we are not ready to deploy the fix yet. please continue to use the suggested workaround for a bit longer.

we need to test the new updates a bit more before shipping it

<@U01KZ61GV5Z> hi, can you tell us status of fix deployment

this should be going out early next week. the new templates have been merged just today

`If a user joins or re-joins while the recording is in progress`
This is the publisher?

Do you have a custom template that is trying to resubscribe? What does that logic look like?

Yes, the person publishing the track leaves and re-joins, when they re-join the recorder attempts to re-subscribe to the track and get's the above error

Yes so I have a custom page that renders the users it uses the livekit react library controls &lt;VideoRenderer&gt; and &lt;AudioRenderer&gt;

The &lt;AudioRenderer&gt; is being put on the page and getting the correct track but that track gets stuck on "desired" status after that error is thrown

I've tried both with autoSubscribe turned on and with it turned off manually subscribing to the user's audio track, neither seem to work

I actually have more detailed logs I just finished gathering:

and by the way, do you call `MediaStreamTrack.stop` anywhere in your code?

No we don't manually call stop anywhere in our code

what's happening is we would re-use the transceiver when a new track is published. in order to do so, browsers would recycle the exact same MediaStreamTrack object.. if anyone called `stop` on that object, the browser isn't smart enough to use a new one.. so we added a check to ensure if a TrackSubscribed event fires, we log an error if the MediaStreamTrack is not in the right state.

Interesting,

Those are all the logs from the recorder web page with livekit debug logs turned on.

I didn't see anything that odd in the logs but the lifecycle of that track is:
• User connects
• Audio track published (TR_AMkdDGMNy6Qaf7)
• Audio track is subscription status event, track is unsubscribed (maybe that's strange that it unsubscribed right after it was published?)
• Audio track status changes to "desired"
• User is rendered
• trackSubscribed event for the camera track
• Attempt to subscribe to audio -&gt; track is ended

We don't call stop anywhere and I only see a couple places in the client js where stop is called

Under updateInfo if it detects a removed track, but I don't see a log for that so I'm guessing it didn't happen there

In Room.ts when a participant is disconnected.

I'm going to build a local copy of the client js with some additional logging and try that out. But I'm a little baffled

Also just to reiterate, this is ONLY happening on a headless browser attempting to record the page. If you visit that same page in a normal browser it all works fine.

In fact the livekit Web Egress recorder works totally fine as well...

So if the issue I posted above with the web egress recorder is fixed we can just switch to your recorder instead and not worry about this issue

oh interesting.. that's helpful details..

for your own headless browser, is it Chrome or Chromium, and which version?

I'll have to check with the vendor. We're currently using the <http://mux.com|mux.com> web input product for recording currently so we don't know the exact browser they're using

Got it, I think the most likely scenario is they are using a modified/older version of Chromium that isn't behaving correctly.

By the way, the fix for web egress ending too soon has been deployed!

Yeah that's my guess too, we had problems with their browser not supporting `<http://Array.at|Array.at>()` So I think it is a pretty old browser

That's awesome, trying it out now!

Got a response from the mux team about the browser:
```We're running Chrome 90 (heavily sandboxed), which we know is outdated, and we're working towards upgrading it soon.```
So yeah I'm guessing it some problem with that old chrome version

Welcome <@U04RCSTJRQB>, if your target platform is the browser you should use one of the JavaScript SDK’s.

• If you need full control over everything, you should use our battle-tested client-sdk-js: <https://github.com/livekit/client-sdk-js>
• If you need something more high-level, you should check out: <https://github.com/livekit/components-js> (currently in dev-preview and with limited documentation), which will replace the current react package <https://github.com/livekit/livekit-react> package in the not-too-distant future.

Also <@U03RM7G8Q6A> I want to know if there is a course / blog / playlist on working with JavaScript / React SDK? Or we only have to go through documentation to work?
I know LiveKit Doc is good but I go fast with blogs/videos while doing something.

Most packages have an example folder to give you an idea of how to use the packages.
We are currently working on examples, blog posts and videos. So stay tuned, we will start releasing them soon.

If you run into any roadblocks, you can also ask questions here in the community. :slightly_smiling_face:

<@U03RM7G8Q6A>
What's the difference between @livekit/components-react &amp; @livekit/components-js? Or both are same thing?

<https://github.com/livekit/components-js|@livekit/components-js> is the monorepo and  <https://github.com/livekit/components-js/tree/main/packages/react|@livekit/components-react> is a part of it. It is the framework specific implementation of LiveKit Components for React (we will add support for other frameworks in the future).

I tried editing the `<http://audio_device_core_win.cc|audio_device_core_win.cc>` file as a solution, but I'm not sure if this is the best solution.

UPDATE:
I was able to fix this by calling selectAudioOutput before connecting to the room

hey <@U03J7HTAF1C>,you can also check this PR, <https://github.com/livekit/client-sdk-flutter/pull/243>, it can let windows set the correct input/output device before publishing, and it will remember the setting in the next connection

for broadcasts, do you mean data publishes (i.e. sendData to the server api)? If so, this was just fixed. update the sdk to v1.1.7

<@U04MS1V0LFN> FYI I have deleted your duplicated posts in <#C01KZ61HZRR|general> and <#C048FRL1N2C|cloud>

please refrain from posting the same message to multiple channels.

<@U01KZ61GV5Z> Sure, will keep it in mind.

Screenshot 2023-02-28 at 16.08.40.png

<@U04RCSTJRQB> it seems that you are in the `packages/core` directory. Can you please double check that you are in the root directory of the monorepo and try to run `yarn build` again.

Normally you should see a total of 7 tasks, not just 2.

Also getting following errors/warning before it is actually failing:

hi <@U04RCSTJRQB>
looks like you’re on windows?
Could you provide some info about your environment:
• windows version
• node version
It would also be helpful to get the full logs of the freshly cloned repo running
• `yarn`
• `yarn build`
as a .txt file instead of screenshots

<@U02EU2ZPTQU>
Yes I'm on Windows.
• Windows 10: 64 bit, Version: 22H2
• Node 16.13.2
I just cloned the repo again &amp; below are logs of `yarn` &amp; `yarn build`

thanks! looks like the issue is that we’re not handling windows file paths when generating the styles package.
We’ll add an issue to track this, in the meantime if that is an option for you, running the repo from within WSL should work just fine!

Okay <@U02EU2ZPTQU>
Thanks for helping me out.

Hi <@U04S9JQDA8Y> — I believe this should already be there. cc <@U01UB0LHTTQ> 

there is a webhook for that: <https://docs.livekit.io/server/webhooks/#egress-ended>

<@U01KZ0SCDKL> <@U01UB0LHTTQ> thanks for replying !! have a nice day

Was this room composite with a custom template? I've never seen this image, and got no results with reverse image search

It was a web egress pointing to a page we setup that joins a room and renders it's own layout

The image linked above is not from our site as as far as I can tell, I've never seen it before either

I'll DM you the exact link (we have an auth token in the url so don't want to post it publicly)

The link I just DMed is to the same page for a different room, it has a test user with a looping video in there now

Ah so I figured out the image. We are using the livestream provider mux and that is the default image they use when a livestream disconnects: <https://docs.mux.com/guides/video/handle-live-stream-disconnects>

So it looks like the livestream coming from the web egress disconnected when it failed then mux just shut the livestream down 5 minutes later

I see, do you have logs from the egress service related to the disconnection?

We are using livekit cloud hosting so I don't know that we have any way to get the logs

<@U01UB0LHTTQ> Any word on that egress? ^

gstreamer lost connection to the endpoint

something (unrelated) to note, chrome was dumping thousands of these warnings, about once per second: `When using adaptiveStream, you need to use remoteVideoTrack.attach() to add the track to a HTMLVideoElement, otherwise your video tracks might never start`

I’ll update the service to get a more detailed error message. All I can say for now is that it came from this function in gstreamer’s rtmp2sink:
```send_connect_error (GstRtmp2Sink * self, GError * error)
{
  if (!error) {
    GST_ERROR_OBJECT (self, "Connect failed with NULL error");
    GST_ELEMENT_ERROR (self, RESOURCE, FAILED, ("Failed to connect"), (NULL));
    return;
  }

  if (g_error_matches (error, G_IO_ERROR, G_IO_ERROR_CANCELLED)) {
    GST_DEBUG_OBJECT (self, "Connection was cancelled (%s)",
        GST_STR_NULL (error-&gt;message));
    return;
  }

  GST_ERROR_OBJECT (self, "Failed to connect (%s:%d): %s",
      g_quark_to_string (error-&gt;domain), error-&gt;code,
      GST_STR_NULL (error-&gt;message));

  if (g_error_matches (error, G_IO_ERROR, G_IO_ERROR_PERMISSION_DENIED)) {
    GST_ELEMENT_ERROR (self, RESOURCE, NOT_AUTHORIZED,
        ("Not authorized to connect"), ("%s", GST_STR_NULL (error-&gt;message)));
  } else if (g_error_matches (error, G_IO_ERROR, G_IO_ERROR_CONNECTION_REFUSED)) {
    GST_ELEMENT_ERROR (self, RESOURCE, OPEN_READ,
        ("Could not connect"), ("%s", GST_STR_NULL (error-&gt;message)));
  } else {
    GST_ELEMENT_ERROR (self, RESOURCE, FAILED,
        ("Failed to connect"),
        ("error %s:%d: %s", g_quark_to_string (error-&gt;domain), error-&gt;code,
            GST_STR_NULL (error-&gt;message)));
  }
}```

Thanks for the details. I'll keep an eye on future recordings and ping if it disconnects again

As for the adaptive stream errors. I'll look into those. Odd that I'm not seeing that error in normal browser clients

<@U03DTLWA19P> we just shipped an update yesterday that makes RTMP out much more reliable. it would now recover if connection to RTMP destination is interrupted

Oh that's amazing! Thanks for that! We're fully switching over to livekit recording from mux web inputs next week so that's great to hear

may a little help <@U01KZ61GV5Z> <@U01KZ0SCDKL> :point_right::point_left:

We don’t have any way to generate thumbnails right now. As far as I know, the way to do it would be to save the file using egress and then generate the thumbnail using ffmpeg

generating thumbnails would involve decoding the video first. The Go SDK currently doesn't handle that. You can also use Pion <https://github.com/pion/mediadevices/tree/master/pkg/codec|mediadevices> to decode the frames and then encode them into a jpeg/png

Here's the issue tracking this request: <https://github.com/livekit/livekit/issues/1378>

Thanks. I had a comment in this issue.Could you help to comment whether it make sense or not?

yes, you can definitely develop your own service that's compatible with those definitions. we run something like that on our end.

can you share it with me?  Our team needs to try the Unity demo as soon as possible to show it to the boss

What do you need exactly? Which platform? Audio or Video?

can accept the error, we need to go with Unity Live for the head to decide to choose Livekit or Agora.  I like Livekit for experimenting with Flutter, however Unity sdk is not ready yet.  I need voice chat.  Sir

We need to test voice quality and latency.  At the moment the sdk error arises no problem.  Just experiment for the head to decide. <@U022KB72NFJ> :cry: 

<@U04N2ULR17C> you can test voice quality and latency with <http://meet.livekit.io|meet.livekit.io> — the Unity SDK should not (in theory) negatively impact stream quality or latency. 

If Unity native sdk is not handled well it will create lag and bad sound quality.  I have tested many services

Our goal is to have high quality SDKs, it will not happen :slightly_smiling_face:

The Unity SDK ~will~ uses our common core written in Rust

Yes.  Is a beta or official release expected in March?  Can't I get a trial version now?

<@U04N2ULR17C> I’m kindly asking you to stop inquiring about an early release or when the Unity SDK will be ready. You’ve already asked about it a few times in other messages. We do expect that a developer preview release will happen this month (March). 

There’s a lot of work involved in building this SDK and we have a high quality bar for all the things we do.

It may not be what you want to hear, and I apologize for that, but to borrow from 90s-era Blizzard Entertainment: “it’s ready when it’s ready.”

Thank you for your understanding. :pray::skin-tone-4:  

sorry for asking too much.  I want to ask more March preview, can it be used with ios android or just desktop?

The developer preview will be aimed at desktop, but it _should_ work on mobile.

We don’t have any way to generate thumbnails right now - you would need to use ffmpeg or some similar tool to generate them from the saved video

thank you for replying :slightly_smiling_face:

the one in the `/example` folder of the client sdk repo is the only one we have

Question: When using multiple VMs, you need a load balancer before them, so your url for connection points to the load balancer?

• <https://github.com/livekit/components-js/tree/main/examples/nextjs>
• <https://github.com/livekit/meet>

in addition to that also check out <@U04F14MLUJE>’s new spatial audio sample! it uses pixie instead of babylon, but the concepts should be similar:
<https://livekit-users.slack.com/archives/C01KZ61HZRR/p1677695519693109>

you could work around this by querying the room with one of the server-sdks!

Listed here:
<https://github.com/livekit/livekit#server-sdks|https://github.com/livekit/livekit#server-sdks>

Hey <@U04RCSTJRQB> we don’t have anything for babylon but we did just release this sample project that should get you on the right track: <https://livekit-users.slack.com/archives/C01KZ61HZRR/p1677695519693109>

Hi Ankit, are you using cloud or self-hosted?

It’s very difficult to debug what’s happening from the description above. Do you see anything in server logs? Have you tried this using LiveKit Cloud?

Image_20230306_110012_943.png

Image_20230306_110013_356.png

no, i haven't check server log but i show one thing if you can understand there in this only happening while call between iphone and android so when remote audio is coming from both device i show Mic Mode standard and and when audio is not coming through, its showing Mic Mode off you can see both below image

<@U01KZ0SCDKL> i just want to know that  due to slow internet connection it can happen?  means if i am not connected to higher quality internet in iphone  my microphone not getting on??

<@U01KZ0SCDKL> we are also seeing problems with the Flutter SDK and multiple participants on web. It's basically impossible to have calls with more than 2 participants. Someone is always missing audio. Video comes over ok in all cases. Previously we were using the JS sdk and never had this kind of problem. Audio is supposed to just work, so not sure what could possibly be wrong other than the SDK being buggy. All video feeds always appear but each user seems to be able to hear a random subset of the participants.

<@U01KZ0SCDKL> after some debugging, looks like the audio tracks are not being autoplayed. Maybe due to the browser constraint where you can't play audio before a user interacts with a page. Of course since it's depending on "autoplay" for this, it never logs anything and just transparently doesn't autoplay.

Perhaps it should use the play method to play the sound so that it can at a minimum log an error when it doesn't play. However, once you are in this state, It doesn't look like there is a way via the SDK to detect the error, prevent the error, handle the error, or resume the playback.

<@U03S00K6BD5> thanks for the report, the flutter sdk is currently not handling this case.
The way this is currently handled in the js sdk is that there’s a `RoomEvent.AudioPlaybackFailed` being emitted, which gives you the option to display a button and subsequently call `room.startAudio` to force/resume playback after user interaction.
We’ll get this kind of behavior added to the flutter sdk as well!

do you have the adaptiveStream option turned on?

The way how I create a room

```LiveKit.create(
    context,
)```
And I see adaptiveStream is disabled by default

have you tried setting the video quality/video dimensions on the RemoteTrackPublication?

No. Can please tell where I have to specify it ?

I'm new to livekit and now I'm trying to switch from custom solution to livekit )

Let me also provide more details. As I use AI model independently from conference calls or remote video monitoring, I created shared video track

```private val liveKitRoom by lazy {
    LiveKit.create(context)
}

val liveKitVideoTrack by lazy {
    liveKitRoom.localParticipant.createVideoTrack("localVideoTrack")
}

val liveKitAudioTrack by lazy {
    liveKitRoom.localParticipant.createAudioTrack("localAudioTrack")
}```

which means I can in another service use them for object detection

```private fun startDetection() {
        with(mediaService.liveKitVideoTrack) {
            startCapture()
            addRenderer(videoSinkV2)
        }
     
    }

    private fun stopDetection() {
        with(mediaService.liveKitVideoTrack) {
            removeRenderer(videoSinkV2)
            stopCapture()
        }
    }```
In the same time I can use this track to publish in another room

```private val room by lazy {
    LiveKit.create(
        context,
    )
}
 room.connect(
                callOptions.liveKitServerUrl,
                callOptions.token,
            )

          
            room.localParticipant.let {
                it.publishAudioTrack(mediaService.liveKitAudioTrack)
                it.publishVideoTrack(mediaService.liveKitVideoTrack)

                it.setCameraEnabled(callOptions.videoEnabled)
                it.setMicrophoneEnabled(callOptions.audioEnabled)
            }```


and when I want to lease video call I unpublish this track and disconnect a call

```room.localParticipant.let {
    it.unpublishTrack(mediaService.liveKitAudioTrack, false)
    it.unpublishTrack(mediaService.liveKitVideoTrack, false)
}

room.disconnect()
room.release()```


is the AI model on the publishing device or can be remote subscriber?

So when device is not in a call video frames resolution is not dropped under 480p and AI model works fine. When I connect to a call it may be lowered.

I had the same problem in my previous implementation and problem was because of high CPU usage and webrtc was dropping frames. and for this I had to disable image quality throttling but this snippet


```
val pcConstraints = MediaConstraints()

 pcConstraints.optional.add(MediaConstraints.KeyValuePair("googCpuOveruseDetection", "false"))```


AI model on publishing device. It does not do anything with remote tracks. It needs local camera video stream to be processed

``` with(mediaService.liveKitVideoTrack) {
            startCapture()
            addRenderer(videoSinkV2)
        }```
Here you can see I use local video track and add renderer to it

hmm, as far as I know, the addRenderer route just gets back the raw frames that it receives from the capturer, weird that the resolution drops during the call

As I undertood in room it's takes raw video frames from camera and then format to different resolutions

I guess it may be due to googCpuOveruseDetection

Do you know if it's possible to set perrConnectionFactory media contraints ?

right now we don’t support custom media constraints, though that should be possible to add

lemme check on the frames you get back from the track

Also do you think is it right wat to create shared video/audio track and use it in different places ?

ahh, looks like it does slightly get adjusted. is the device camera very close to 480p resolution?

what’s the resolution before and after connecting to a room?

When device is not connected to any of the room. Resolution is always 800 by 600

When device was simultaneously in two rooms, resolution dropped to 300 by 240 or so

hmm haven’t tried using the same track in multiple rooms at the same time

in any case, might be simpler to get the raw frames from the camera. we don’t have this supported right now, but it’s something I’ve been meaning to add

yes. for most cases it's not needed, but this app is not running on user phones, it created for special tablets which running android

does the resolution drop only happen sometimes, or is it always triggered by something?

I liked Livekit server and want to adopt to our product. It's only one limitation what I have

It drops always as I connect to one or more groups

so it drops even when connected to just one room?

I think it's because of hight CPU usage. Please check CPU usage. First only object detection is here. Then It connected to remote monitoring room and then connected to video call with one participant

<@U01NZKTNE1H> I was able to import livekit source code to my project. Can you please point me where I can try to add  pcConstraints.optional.add(MediaConstraints.KeyValuePair("googCpuOveruseDetection", "false")) to check if it's not because cpu overuse ?

<https://github.com/livekit/client-sdk-android/blob/6c0dbe834173a6fdd3c473b8be7045f123112a54/livekit-android-sdk/src/main/java/io/livekit/android/room/RTCEngine.kt#L538|https://github.com/livekit/client-sdk-android/blob/6c0dbe834173a6fdd3c473b8be7045f[…]-android-sdk/src/main/java/io/livekit/android/room/RTCEngine.kt>

I added

```add(
    MediaConstraints.KeyValuePair(
       "googCpuOveruseDetection",
        "false"
    )
)```
But it still dropping resolution

and once I disconnected all rooms it got back for normal image resolution

okay, added the ability to add a `VideoProcessor` when creating the video track:
<https://github.com/livekit/client-sdk-android/pull/192/files>

here’s an example VideoProcessor:
```
import org.webrtc.VideoFrame
import org.webrtc.VideoProcessor
import org.webrtc.VideoSink

class PassthroughVideoProcessor : VideoProcessor {
    private var videoSink: VideoSink? = null
    override fun onCapturerStarted(started: Boolean) {
    }

    override fun onCapturerStopped() {
    }

    override fun onFrameCaptured(adaptedFrame: VideoFrame?) {
        videoSink?.onFrame(adaptedFrame)
    }

    override fun onFrameCaptured(frame: VideoFrame?, parameters: VideoProcessor.FrameAdaptationParameters?) {
        // do something with raw frame here
        super.onFrameCaptured(frame, parameters)
    }

    override fun setSink(sink: VideoSink?) {
        videoSink = sink
    }
}```

should be available soon on the v1.1.8-SNAPSHOT

Ok. Thank you so much. Is the any way I can try it ?

see installation instructions for the snapshot version here: <https://github.com/livekit/client-sdk-android#installation>

Do I have to change something in my source code after upgrading?

will just have to pass the VideoProcessor in when creating the video track

I don't believe it's possible to guarantee that minimum.. due to bandwidth constraints (CPU is a bit easier to control for).

If WebRTC cannot send at the desired bitrate, it'll have to lower that bitrate somehow. I think there's a way to hint that it should prioritize resolution over fps, but that only goes so far

<@U01KZ61GV5Z>

If WebRTC cannot send at the desired bitrate, it'll have to lower that bitrate somehow.

is dropping image resolutions implemented on LiveKit side or on WebRTC library side ?

when there's insufficient bandwidth, something has to give.. either you sacrifice bitrate, or latency.. webrtc is a real-time protocol so the decision to sacrifice bitrate is made deep down in the stack

But is it possible still getting raw video frames from camera to process them locally ?

MediaConstraints videoConstraints = new MediaConstraints();
videoConstraints.mandatory.add(new MediaConstraints.KeyValuePair("minWidth", "720"));
videoConstraints.mandatory.add(new MediaConstraints.KeyValuePair("minHeight", "480"));

PeerConnectionFactory peerConnectionFactory = PeerConnectionFactory.builder().createPeerConnectionFactory();
PeerConnection peerConnection = peerConnectionFactory.createPeerConnection(iceServers, videoConstraints, new YourPeerConnectionObserver());

Is the any way that I can put my own media contraints if for me min resolutions is the most critical ?

ah ok I misunderstood where you were performing frame processing. If it's on the client side, before transmitting, then ignore everything I just said :sweat_smile:

Ok. So You think what <@U01NZKTNE1H> proposed will work. Right ?

<@U01NZKTNE1H> Hi. It helped. I created this video processor

```class VideoProcessorWithObjectDetector: VideoProcessor {
    private var videoSink: VideoSink? = null

    private var objectDetectorVideoSink: VideoSink? = null

    override fun onCapturerStarted(started: Boolean) {
    }

    override fun onCapturerStopped() {
    }

    override fun onFrameCaptured(adaptedFrame: VideoFrame?) {
        videoSink?.onFrame(adaptedFrame)
    }

    override fun onFrameCaptured(frame: VideoFrame?, parameters: VideoProcessor.FrameAdaptationParameters?) {
        // do something with raw frame here
        super.onFrameCaptured(frame, parameters)
        objectDetectorVideoSink?.onFrame(frame)
    }

    override fun setSink(sink: VideoSink?) {
        videoSink = sink
    }

    fun setObjectDetectorSink(sink: VideoSink?) {
        objectDetectorVideoSink = sink
    }
}```
and my object detector always was getting the same resolution despite of how many room and participants.

To verify it I moved object detector to  override fun onFrameCaptured(adaptedFrame: VideoFrame?) and I was experiencing the same behaviour as it was before your changes. Thank you.

<@U01NZKTNE1H> Hi. Now I'm facing another issue. Looks like local video track stops working in some time after disconnecting from video call


I have shared video and audio track with new video processor

```MediaService.kt file

val videoProcessor: VideoProcessor by lazy {
    ObjectDetectorVideoProcessor()
}

val liveKitRoom by lazy {
    LiveKit.create(context)
}

val liveKitVideoTrack by lazy {
    liveKitRoom.localParticipant.createVideoTrack(
        "localVideoTrack",
        videoProcessor = videoProcessor)
}

val liveKitAudioTrack by lazy {
    liveKitRoom.localParticipant.createAudioTrack("localAudioTrack")
}```

When I connect to the call I publish these public tracks to the room

```LiveKitCallEngine.kt 

 val room by lazy {
        LiveKit.create(
            context,
            options = RoomOptions(
                adaptiveStream = true,
                dynacast = true
            )
        )
    }

room.localParticipant.let {
    try {
        it.publishVideoTrack(mediaService.liveKitVideoTrack)
        it.setCameraEnabled(callOptions.videoEnabled)
    } catch (e: Exception) {
        e.printStackTrace()
    }

    if (callOptions.audioEnabled) {
        it.publishAudioTrack(mediaService.liveKitAudioTrack)
    }
    it.setMicrophoneEnabled(callOptions.audioEnabled)

}```


When I disconnect from the call first I unpublish tracks with stopOnUnpublish as false not to stop object detection

```room.localParticipant.let {
    it.unpublishTrack(mediaService.liveKitAudioTrack, false)
    it.unpublishTrack(mediaService.liveKitVideoTrack, false)
    it.setCameraEnabled(false)
    it.setMicrophoneEnabled(false)
}

room.disconnect()
room.release()```

After a call disconnected I still see my video processor is getting streams and it was working fine when I was connecting/disconnecting from the calls during the evening. But when I left my app over the night object detection video sink stopped getting tracks and when I try to join a call I see the next error as on screenshot.

Do you have any ideas what can be ?

ahh, I didn’t think about this, but media stream tracks get disposed on disconnect at the moment. actually surprised that you still get frames post disconnect

Because of this first I unpublish tracks

``` it.unpublishTrack(mediaService.liveKitAudioTrack, false)
 it.unpublishTrack(mediaService.liveKitVideoTrack, false)```


If I don't do it it will immediatelly will dispose tracks

are you still calling room.release between connects?

that should only be called if you’re completely done with the room object

Yes when I disconnect from call I completely done with this room object. When I have a new call I create a new room object

hmm another approach that should work better would be to create just the video capturer to share for each room

do you need to process frames while disconnected?

As you see I have one room created which has life as long as app is running

```val liveKitRoom by lazy {
    LiveKit.create(context)
}```

and then when I connect to calls I create a new room

do you need to process frames while disconnected?

Yes. I have 3 separate processes.

• Object Detection
• Remove monitoring when we have 1 publisher and multiple subscribers
• Many to Many call

yeah I think owning the video capturer and intercepting frames there would work better there then