LiveKit Community

image (8).png

<@U01KZ61GV5Z> <@U04VDH778AV> any idea on this?

yeah, we have a PR WIP about fixing video blur on iOS, maybe this issue will be resolved soon

Excellent. Will it have an impact for Android?

yeah, don’t worry, we’ll look into it android and try to fix it

a large factor here is your hosting provider, how much the machines are utilized, and how far the users are from the server.

can you reproduce this with LiveKit Cloud?

Servers have 3 vCpu 4 GB Ram 10Gbps internet speed. Servers are too close to the country where the broadcasters are located. They share the same country  border.
The blurring problem can also occur when there is 1 broadcast and 2 viewers, so there is no need for an excessive number of users.
If it is still not resolved after <@U02GAC58A79>'s fix, we will focus on the server side.

Is there a fix for Android? Is there a version I can test?

<@U01KZ61GV5Z> <@U02GAC58A79> Would a low maxBitrate cause this problem?

are you still seeing this on the latest version of the SDK?

image (1).png

yes, In some android users, the face of the person is almost never clear.

We will test again when the problem in the link is resolved.
<https://livekit-users.slack.com/archives/C01KVTJH6BX/p1686945675600479>

Are you looking to achieve this with the Android SDK?

Pretty sure this is what handles the flip <https://github.com/livekit/client-sdk-js/blob/50ab51cb8a731f944064cfcdab5689354ca0a65c/example/sample.ts#L542>

With this said, I don't think there's actual documentation around this specifically.

If you were referring to switching from back camera to front, Id say this would be it. <https://github.com/livekit/client-sdk-js/blob/50ab51cb8a731f944064cfcdab5689354ca0a65c/example/sample.ts#L244>

You can refer to the following example for its usage: <https://github.com/livekit/client-sdk-js/blob/50ab51cb8a731f944064cfcdab5689354ca0a65c/example/index.html#L104>

You are welcome <@U058G0W9JHG> :slightly_smiling_face:

Another quick question while I have you: Is it possible to access the roomInfo variable in the Room object? When I reference room.roomInfo.*creationTime* it complains that roomInfo is a "private variable".

Since roomInfo is private, you won't be able to achieve this. I'm not super sure, tbh.

You could get this through the Server SDK, but unsure with the Client SDK.

<https://docs.livekit.io/server/room-management/#listrooms> this would return you the information you are after.

Is this helm? I think you’re missing `egress:` in your config
```replicaCount: 1

egress:
  api_key: key
  api_secret: secret
  ws_url: ws://&lt;livekit-host&gt; no port
  redis:
    address: &lt;redis-address&gt;:25061
    username: &lt;redis-username&gt;
    password: &lt;redis-pw&gt;
    use_tls: true
    db: 0

  health_port: 8080
  prometheus_port: 9090
  log_level: debug
  insecure: false

resources:
  limits:
    cpu: 7500m
    memory: 7168Mi
  requests:
    cpu: 6500m
    memory: 6144Mi```


Yooo that did it! Thank you! I tried egress: with child props indented before, but I probably had some other issue going on for those experiments.

please reference <https://livekit.io/connection-test|connection tester>, it should give you more information about what type of connection issues it's seeing

IMG_20230520_190933_878.jpg

Here is a screenshot from one of the session dashboards

hmm, that's definitely unexpected. Do you know what conditions would typically cause them to show up?

Hi <@U01KZ61GV5Z>. Since it doesn't happen every time, it's hard to tell but it looks like multiple successive reconnects may help to reproduce it.

got it, we'll take a look to see if we can repro

Hi <@U01KZ61GV5Z>. Here is an additional note. Not sure if it is important.
Since we don't know if someone entered the room first, each participant sends a request for room creation with egress (with configuration settings) before entering the room.

<@U01KZ61GV5Z> Here is one more session. One of our testers experiencing often app crashes (apparently it's something on our side) during the call. She is reconnecting to the same room after that. The average duration of the session is ~7.5-8 minutes. Maybe those "ghost" participants are caused by the improper  room disconnect?
For context: I'm talking about our app that is written in Flutter.
<https://cloud.livekit.io/projects/p_519ctfqf1bf/sessions/RM_HCCsuNFSV7eT>

Hi <@U01KZ61GV5Z>. Update on room creation. I've asked my coworker about how he configured room creation with egress and it turns out that he actually checks if the room already exists. So I'm not sure why it is even happening.

<@U03GQ47GNF4> I haven’t been able to reproduce the `pipeline frozen` issue yet, but I found a few problems that might have potentially been causing it. I plan to have fixes deployed tomorrow

Hi <@U01UB0LHTTQ>. Thanks for response. Hope it'll help.
I have one more question. I didn't notice at first but it turns out that there can be multiple session per one room name. It is really confusing. What are those sessions and how are they related to rooms?

I guess I figured it out. "Session" is a session that has at least one participant and ends after the last participant leaves the room. Is that right?

a session is an instance of the room (like classes and objects :blush: 

image.png

Hi <@U01UB0LHTTQ>. Any idea on those "ghost" users with egress IDs? I'm still getting those in my session dashboards. And the main cloud dashboard shows that those are considered as Go clients

As far as I understand the recording works as follows:
Go client joins the room, subscribes to the track and saves it to the file.
If that's true then it explains the "ghost" participants. Can you do something on a server side or change clients so they won't appear in the list of participants and SDK users won't need to filter them out?

ok, those are Egress bots that join the room and record them.

and you are correct that they should not be visible to clients. Egress participants have identities with prefix `EG_` , and they are joining the room as `hidden` participants

OK, so what is the eventual recommendation on this one? Should I, as SDK user, manually filter them out so the end users of my app doesn't see them or they should not be visible to me (as a developer) in the first place and that's something you'll fix in the future?

That's an interesting concept of hidden participants. Is it possible for me to create one and if yes then how I do that?

Thanks

You should not need to filter them out. They are visible in the dashboard, but should not be visible to your client SDK. If you are seeing them in the UI, that would mean it's a bug. Is that what you are observing in recent sessions? (we did have a bug where allowed them to slip through in some conditions. I believe we've fixed it)

You can create hidden participants too (tho they cannot publish anything, otherwise they'll become visible). You'd just need to give it the `hidden` grant in their token: <https://docs.livekit.io/concepts/authentication/#video-grant>

When this bug was fixed? I didn't turn it off (just in case) when making new builds for our testers. That's why I may not have noticed the fix.

Oh, OK. That may be the case. I didn't see anything related to this in the changelog so I didn't remove the filter. I'll try removing the filter.

Sounds good. if it remains an issue, please let us know. We have not gotten any recent reports for this.

Hi I think I solved the problem by using `importmap` to get the `import` functionality working. In particular, for `livekit-client` package import from `npm`:
```&lt;script type="importmap"&gt;
  { "imports": {
      "livekit-client": "<https://ga.jspm.io/npm:livekit-client@1.9.6/dist/livekit-client.esm.mjs>"
  } }
&lt;/script&gt;```
Once this is established, I could then define `&lt;script type="module"&gt;&lt;/script&gt;` and finally within this script block the `import` functionality works. Hope this helps out anyone who might be stuck with vanilla JS for platform compatibility reasons. Though having a Livekit JS variable that could be accessed directly would ideally be easier.

hey <@U058PP2A94K>,
your workaround should work just fine (on all browsers that support js modules natively).
For setups like yours we bundle an additional umd version. If you load the `umd` version of livekit client, you can access it with `window.LivekitClient` directly in the browser. Hope that helps.
You can also load it directly in a script tag from unpackage from `<https://www.unpkg.com/livekit-client@1.9.6>`

it's difficult to tell from your screenshot alone.. what the UI indicates is that the 2nd participant is not subscribed to anyone else.

My first thought would be, does your token have the permission

Can you share some code pls <@U0575PCGSN4> ?

`import {AudioPresets, ConnectionQuality, DataPacket_Kind, Room, RoomEvent, Track, VideoPresets} from 'livekit-client';`

`const encoder = new TextEncoder()`
`const decoder = new TextDecoder()`

`class Client {`
  `constructor() {`
    `this.room = new Room({`
      `videoCaptureDefaults: {`
        `resolution: VideoPresets.h216.resolution`
      `},`
      `adaptiveStream: false,`
      `publishDefaults: {`
        `videoCodec: "h264",`
        `red: false,`
        `dtx: false,`
        `simulcast: false,`
      `}`
    `});`

    `this.room.on(RoomEvent.DataReceived, OnDataReceived)`
  `}`

  `async sendMessage(_data_, _participantSIDs_) {`
    `const strData = JSON.stringify(_data_)`
    `const encodedData = encoder.encode(strData)`
    `await this.room.localParticipant.publishData(encodedData, DataPacket_Kind.RELIABLE)`
  `}`

  `async connect(_token_) {`
    `await this.room.connect(`<ws://localhost:7880>`, _token_, {`
      `autoSubscribe: true, maxRetries: 3, peerConnectionTimeout: 15_000, rtcConfig: {`
        `bundlePolicy: "max-bundle", rtcpMuxPolicy: "require"`
      `}`
    `})`
  `}`
`}`

`function OnDataReceived(_payload_, _participant_, _kind_) {`
  `const strData = decoder.decode(_payload_)`
  `console.log('data received', strData, _participant_, _kind_)`
`}`

`const client = new Client()`
`export {client}`

Simply to validate, in your video grant, you do have `canPublishData: true` ? Ref: <https://docs.livekit.io/concepts/authentication/#video-grant>

`func getJoinToken(apiKey, apiSecret, room, identity string) (string, error) {`
  `at := auth.NewAccessToken("devkey", "secretsecretsecretsecretsecretsecretsecretsecret")`
  `CanPublish := true`
  `CanSubscribe := true`
  `CanPublishData := true`

  `grant := &amp;auth.VideoGrant{`
    `RoomCreate:           false,`
    `RoomList:             false,`
    `RoomJoin:             true,`
    `Room:                 room,`
    `CanPublish:           &amp;CanPublish,`
    `CanSubscribe:         &amp;CanSubscribe,`
    `CanPublishData:       &amp;CanPublishData,`
    `CanPublishSources:    nil,`
    `CanUpdateOwnMetadata: nil,`
    `Hidden:               false,`
  `}`
  `at.AddGrant(grant).SetIdentity(identity).SetValidFor(time.Hour)`
  `return at.ToJWT()`
`}`

I read the whole doc of livekit itself , js and go sdk

Your `sendMessage` method seems correct. May I ask you if you are using the LK Cloud or your own LK Server?

I just simply downloaded the binary and ran it

Are you attempting to send a data message to the same participant? I don’t think that works. (Full disclosure: I’ve only taken a quick glance)

why should I do that ? :grin: 
I simply just want to send a json to server

Well, it looks like you’re sending data messages between participants (facilitated via server).

yeah some messages should be send to other users through livekit server and some messages should be send to Go sdk only

<@U01KZ0SCDKL> do you have a working example of sending message with js and go sdk so I can compare it to my code??

<@U0575PCGSN4> You should be able to look up the examples provided here : <https://github.com/livekit/client-sdk-js/tree/main/example>

Which reference: <https://github.com/livekit/client-sdk-js/blob/main/example/sample.ts#L277>

it seems to be exactly like mine
its really weird that is doesnt work :thinking_face:

<@U01KZ61GV5Z> any opinion about this problem ?

As a suggestion, I'd probably create an account on cloud, see if you are facing the same issue or not. This would validate if its something with the LK Server Setup or the code.

Sorry, I don't have enough context here. But if it works with JS example app that Pat pointed you to, but not your own. My guess is there are differences in how you are using the APIs?

Are you expecting to deliver a message to the same instance of the client?

As Russ mentioned, you can send data messages to other participants, but not to itself.

yeah , I passed empty string array (also tried undefined) as participant sids in order to send to all other participants but does not have any affect

<@U0575PCGSN4> you need to have multiple participants in a room to be able to send messages back and forth. We use this feature extensively in <https://github.com/livekit-examples|example apps>.

Hi,
if you just want to get local preview you dont need to use sdk just simply use js web apis

Hi thanks for the quick reply.

What if I want to switch camera/audio device before connecting? can I do this with the component-react?

I didnt use react component till now but in js sdk you can do that so you should be able to do that in react component too and if you couldnt do that , again you can do that with js native functions too :)

Ah I see, thank you, I will play around with the libs :slightly_smiling_face:

Here’s an example invocation of the Prejoin component:
<https://github.com/livekit-examples/meet/blob/main/pages/rooms/%5Bname%5D.tsx|https://github.com/livekit-examples/meet/blob/main/pages/rooms/%5Bname%5D.tsx>

And here’s the component itself:
<https://github.com/livekit/components-js/blob/main/packages/react/src/prefabs/PreJoin.tsx|https://github.com/livekit/components-js/blob/main/packages/react/src/prefabs/PreJoin.tsx>

Thanks <@U01KZ0SCDKL>, any chance `usePreviewDevice` would be exposed? I would like to do my own custom UI :slightly_smiling_face:

Hi <@U04HZ52BXFD>, it looks like this was an oversight on our part. The function is marked `@public`, but I guess we forgot to expose it. As a temporary fix, you can copy the function. We will add it to the public API in a next release.

Hi <@U03RM7G8Q6A>, any chance `useStartAudio` can be exposed as well?

<https://github.com/livekit/components-js/blob/65991d010549a7a9eda9f416a9ce6fcab8271a19/packages/react/src/components/controls/StartAudio.tsx#LL13C10-L13C23|https://github.com/livekit/components-js/blob/65991d010549a7a9eda9f416a9ce6fcab8271a19/packages/react/src/components/controls/StartAudio.tsx#LL[…]23>

just to understand your usecase better (and potentially improve how the components work today), why wouldn’t you want to use the `StartAudio` button component directly?

Hi <@U02EU2ZPTQU>, I want to make the startAudio hidden in the mute button, so initially after connecting, the audio icon will be muted, and the user unmutes (which sets volume to 100 and handles any startAudio related stuff)

ah, but that way the user might not be able to hear everyone else until they unmute themselves, would this be intended?

<https://github.com/livekit/components-js/releases/tag/%40livekit%2Fcomponents-react%401.0.3> exposes `useAudioPlayback` hook and `usePreviewDevice`

are you initiating the disconnect from the same instance? Or is the disconnect triggered by the server?

I'm not sure if we would be firing the disconnect callback if it was initiated locally. CC: <@U02EU2ZPTQU>

I am calling `onDisconnected` from client side:

```&lt;LiveKitRoom
          room={room}
          token={token}
          serverUrl={env.NEXT_PUBLIC_LIVEKIT_SERVER_URL}
          connect={isConnectionToggled}
          onConnected={() =&gt; {
            setIsConnected(true);
            setIsProgressBarDisplayed(false);
            setIsPreview(false);
          }}
          onDisconnected={() =&gt; setIsConnected(false)}
          audio={true}
          video={true}
          screen={false}
          style={{ width: "100%" }}
        &gt;
          &lt;Studio /&gt;
        &lt;/LiveKitRoom&gt;```

I think that's just calling your callback directly?

To trigger a disconnect, you should use `room.disconnect()`

this is a bad error message.. but it's likely indicating incorrect API keys used to verify the webhook. you can determine the key used by base64 decoding the middle section of the JWT token that's sent in `Authorization` header

yeah , I regenerated api key and secret key using livekit cli and now it's working thanks :)

Hi I am trying to set device using `setActiveMediaDevice` from `useMediaDeviceSelect` hook of component-react (calling it inside `LiveKitRoom` context). I am calling it before connecting to room (sort of like choosing the camera device in pre-join stage, and I want the selected device to be the published device when I click connect):
```const {
    devices: videoDevices,
    activeDeviceId,
    setActiveMediaDevice,
  } = useMediaDeviceSelect({
    kind: "videoinput",
  });```
I am getting this error (when `setActiveMediaDevice`):

&gt;  Failed to select the desired device. Desired: e7f7638c0538deb435cc2600e4f4ca1378c1b8e8550c2496912cfe5e3f370c4e. Actual: undefined
Or in general how do I initialize the deviceId using a selected device before connecting to room?

Hi. I am using Flutter and LiveKit to develop an app. I was wondering if there is a way to get access to the audio data from a session locally.

I am trying to create something similar to Twitch live clips. My plan is to keep audio data of the most recent 15 seconds with buffer and export when needed.

Is this possible to achieve? Or is Egress the only way to record audio?

hey ewan!
this sounds like you are not looping over existing publications when the participant connects.
you will want to handle `RoomEvent.TrackPublished/TrackSubscribed` but also invoke the same flow on `RoomEvent.Connected` and basically loop over all existing participant publications

are you using `autoSubscribe`? if so it should automatically subscribe you to everyone in the room.

Thank you both of you! Will do a test and update 

One more issue is this, some browser and I mean only some phones they don't get connected at all, the rest do mostly

I experience this with iPhone especially 

do you have an example session where the user could not connect at all?

this should not happen.. Cloud is covered by comprehensive connectivity tests across platforms

I understand Livekit should be working, it is highly something we are doing wrong on our ende

still not working at all, when we are late into the room we don't get the subscription at all. We only use audio services in Livekit

you would generate one with your API key/secret, located in `Settings`, there's a token generator there too

i want my app to join a room on a press of a button where i dont want a user to enter a token it should be autogenerated on user press

Also i m using flutter for my application

<@U04NSL1PQBS> pls do not repost messages from a thread into the top level of the channel :pray::skin-tone-4:  

<@U01KZ0SCDKL> Can you help me with this i want to have a way to get tokens for users without going to livekit.cloud

<@U04NSL1PQBS> have you read through any of the docs? There’s quite a bit written there about how auth (i.e. tokens) works and how you can accomplish what you need. You definitely do not need LiveKit Cloud’s dashboard to generate user tokens. Check out the code for <https://github.com/livekit-examples/meet?ref=meet|Meet> or other LiveKit <https://github.com/livekit-examples|examples>.

Hi <@U03GQ47GNF4>, thanks for reaching out. Most people familiar with this are in the Pacific time zone, so expect a reply later today.
cc <@U02BH3ZHJ0Y> <@U02MC5V3C2H>

<@U03GQ47GNF4> are you using Cloud or self-hosted?

Hi <@U01KZ0SCDKL>. Cloud, since we decided to eliminate any possible problems that can be related to improper backend configuration/setup.

And you’re still having connectivity/quality issues? This is unexpected. Can y’all help here <@U02GAC58A79> <@U01NZKTNE1H> <@U02MAJLGN65>?

I haven’t been following your other threads <@U03GQ47GNF4>. Have you shared more info about the issues testers are complaining about? Do you surface connection quality in your UI? Do you have sessions/participant URLs in your cloud dash you can share with us, which identify some problematic sessions?

Are there video quality issues besides the slowdown in framerate you mentioned in the other thread?

I have a few more questions in addition to the above
• Where are the testers located?
• What type of network connection are they using? (and any firewalls?)
• Which platforms &amp; devices are the issues experienced on?


Hi <@U01NZKTNE1H>.  If I use configuration from flutter example app, then I get remote participants framerate slowdown over time in my app on Android devices. In example app framerate also degrades over time when `adaptiveStream` is set to `true` . When `adaptiveStream`  is off, example app seems to work just fine, but in my app framerate slowdown is perceptible when `adaptiveStream=false` , but I've managed to improve the situation by increasing max bitrate of the layers. Without it video of remote participants is blocky or quality is very poor/blurry.

Hi <@U01KZ61GV5Z>.
• Testers are mostly located in Ukraine. Exceptions are: one in Cyprus and one in UK
• No firewalls. By most part they are using WiFi but sometimes 4G
• Here is example of something our testers called "bad" (said that connection was very bad and wasn't able to carry out a call): <https://cloud.livekit.io/projects/p_519ctfqf1bf/sessions/RM_2vwLSKihFKfD>

Hi <@U01KZ0SCDKL>. Yes I shared more info in that threads. I show connection indicator in UI but it doesn't seem to work properly. Sometimes it shows perfect connection quality by video quality is poor and sometimes vice versa.

I'm looking at the session that you sent, looks like it has only a single participant that left within seconds. there was no sign of anyone else that they may be having a call with (only an egress instance). Is this the right session?

what indicated to the tester that the connection is bad? could they record a screen recording that allows us to see what they are experiencing?

for the flutter/adaptivestream issue, could you paste the publish settings that you've used? that'd help to see what bitrates/layer sizes are at play

Which one: before "fix/improvement" or after I've made adjustments?

has it been improved to a point of satisfaction for you? or does it still appear choppy?

Our testers are currently testing: some that it's OK, others - not. But default framerates seem to be improper for given resolutions. Also increased max bitrate seems to increase device power consumption and it seems to (thermally) throttle over time and video starts to lag every now and then.

ok, please paste the current settings you are using.

Here one more session. I guess that the same testers. They tries to create another room.
<https://cloud.livekit.io/projects/p_519ctfqf1bf/sessions/RM_PL2QxqQNUNfi>

here are the current settings that seem to work +-
```final videoSimulcastLayers = Platform.isAndroid
        ? const [
          VideoParameters(
            dimensions: VideoDimensionsPresets.h180_169,
            encoding: VideoEncoding(
              maxBitrate: 500 * 1000,
              maxFramerate: 20,
            ),
          ),
          VideoParameters(
            dimensions: VideoDimensionsPresets.h360_169,
            encoding: VideoEncoding(
              maxBitrate: 1000 * 1000,
              maxFramerate: 20,
            ),
          )
        ]
        : const [
            VideoParameters(
              dimensions: VideoDimensionsPresets.h360_169,
              encoding: VideoEncoding(
                maxBitrate: 1000 * 1000,
                maxFramerate: 20,
              ),
            ),
            VideoParameters(
              dimensions: VideoDimensionsPresets.h540_169,
              encoding: VideoEncoding(
                maxBitrate: 3000 * 1000,
                maxFramerate: 20,
              ),
            )
          ];
      await room.connect(
        roomServerUrl,
        roomToken,
        connectOptions: ConnectOptions(
          autoSubscribe: currentUser?.isCoach ?? false,
        ),
        roomOptions: RoomOptions(
            adaptiveStream: !Platform.isAndroid,
            dynacast: true,
            defaultVideoPublishOptions: VideoPublishOptions(
              simulcast: true,
              videoSimulcastLayers: videoSimulcastLayers,
            ),
            defaultCameraCaptureOptions:
                CameraCaptureOptions(cameraPosition: newCameraPosition)),
      );
    }```

also I added this event handling for android devices. Looks like it helps a bit too:
```..on&lt;TrackSubscribedEvent&gt;((e) async {
      if (e.publication.kind == TrackType.VIDEO &amp;&amp; Platform.isAndroid) {
        await e.publication.setVideoQuality(VideoQuality.LOW);
      }
    })```

I’ll look into that, also, I’m making Stats monitoring API for flutter might be able to help find the reason for the bitrate/framerate drop.

thanks for the details. I think the 20fps cap on the simulcast layers is interesting.. have you tried to remove them? we'll do some testing on our side too. We've seen some devices that use hw encoder that aren't able to encode at a different fps for the simulcast layers.

Hi <@U01KZ61GV5Z>. Do you mean custom layers? I've tried the default configuration when i skip `videoSimulcastLayers` and it gave bad results: some testers complained about bad video quality and other/same testers complained about remote participant framerate degradation.
Or do you mean setting `maxFramerate` to those numbers that are given in `video_parameters.dart`?

hey <@U03GQ47GNF4>. If you are using fast connect options in your test, there may be blurry/low frame rate caused by defaultOptions parameters are not passed correctly , you can try this patch
h<https://github.com/livekit/client-sdk-flutter/pull/289/files|ttps://github.com/livekit/client-sdk-flutter/pull/289/files>

Hi <@U02GAC58A79>. It looks like those were added in 1.3.2 version of flutter livekit client. I've switched to this version yesterday and asked our testers to test new build

Sorry, I didn't notice `defaultCameraCaptureOptions`. I'll try to modify it too

Record_2023-05-24-13-46-27.mp4

Screenshot_2023-05-24-13-45-28-33_79d0ff8f9752a3fc32487e6d62330a6a.jpg

Doesn't seem to help. Speed test shows good connection but connection indicator shows poor internet connection quality.

oh no, Looks like the framerate is very low.

Can you try to open another flutter web (chrome) or iOS app to watch the same publish track?

I will try merge the `Stats Monitor API` soon so we can track the RTC stats. This should help us locate the cause of the low frame rate/bit rate.

I've done the following. Entered the same room through my app on android, iPhone and via <https://example.livekit.io/> app on my laptop.
Web app shows both videos OK. But participant from android is always shown with very bad internet connection.
iPhone app shows video from android OK too and also shows poor internet connection for android participant
Android app shows web video OK, but video from iPhone is very laggy (low framerate)
All devices are connected to WiFi

Screenshot 2023-05-16 at 19.51.07.png

Screenshot 2023-05-16 at 19.52.37.png

<@U02GAC58A79>, interesting thing I've noticed recently:
When I open profiler and monitor memory usage the RSS memory usage "sticks" to Dart/Flutter level over time and keeps that way on (first picture). While before it was much higher and wasn't lowering over time (was at the level like on the most left in the second picture).
And it happens only on android devices. On iPhone it is has the plot where RSS memory is almost x2 times higher than Dart/Flutter and never lowers over time (when on the same page in app).
Maybe this behaviour is responsible for those lags.

It seems to be related to this information. I see that the receiving and decoding frame rate is close to 30 fps in the stats monitor layer, but the visual frame rate is only about 5 fps

IMG_0413.jpg

Yes,  From rtc stats, nearly half of the frames dropped 

`disconnecting from surface 0xb4000070c31e0010, reason disconnectFromSurface`
After this log prints, the decoder is reset and starts dropping frames

That's good. At least now we know that it is not something related only to my app. Because it was driving me crazy last few weeks :sweat_smile:

we'll investigate this on our side. having a repro helps!

By the way, when you find the root of the problem can you please give a rough estimate on how much time it may take to fix?
Don't want to be pushy or something, just need to plan our future development steps.

Yes, we have located the issue. The frame loss seems to be related to the HW decoder of libwebrtc. we tested that using the SW decoder can get smooth and clear video. since this is a tricky issue, it may take more time, but I will try to fix it within 1 week if possible.

<@U03GQ47GNF4> we do have a workaround if you are interested. There's a way for us to disable the use of H.264 for your project entirely, so that it will instead use VP8 for the entire project. This would bypass the bug that you've identified with HW decoding.

Let me know if that's of interest and I can enable on our side while the bug is being looked at.

Is that a backend thing you will do or I should change H264 to VP8 in `VideoPublishOptions`?

I can set it in the backend, but it'll take a bit of time for it to clear the caches/etc

If you are sure that it'll improve the video quality, then OK.
By the way, is it something we would be able to change from the dashboard in the future?

Screenshot 2023-05-27 at 1.21.37 PM.png

Yup, we are working on it. here's a preview.

hey <@U03GQ47GNF4> can you try to add this override dependency to your flutter projects `pubspec.yaml`
and check if dropped frames are fixed,  In addition, if you update to the `client-sdk-flutter/example` of the latest commit, you can see video statistics, bit rate, frame rate, resolution, etc. from the main video Widgets.
```dependency_overrides:
  flutter_webrtc:
    git:
      url: <https://github.com/flutter-webrtc/flutter-webrtc.git>
      ref: fix/frame-drops-on-android-hw-decoder```

Hi <@U02GAC58A79>. Sorry for late response. I haven't used statistics display yet but the issue seems to be resolved.

Hi <@U02GAC58A79> <@U01KZ61GV5Z>. It looks like there is similar problem with audio for Android.
When I spawn `ffmpeg` process to strem music to previously generated rtmp URL with this code (JS)
```const ffmpeg = childProcess.spawn(
                `ffmpeg`, [`-re`, `-ss`, `${toHHMMSS(req.body.seekSeconds)}`, `-stream_loop`, `-1`, `-i`, `${concat}`, `-c:v`, `libx264`, `-c:a`, `aac`, `-f`, `flv`, `${req.body.rtmpUrl}`],
                {
                    cwd: process.cwd() + '/songs'
                }
            );```
It looks like some parts of a song are dropping and it becomes choppy. I haven't tried it using cloud project with h264 turned off yet, but I'll let you know.

IMG_0099.MOV

I tried with cloud project where h264 is turned off and got the same behaviour. For Web and iPhone everything works fine. Watch the attached video (playback progression bar is not configured yet so don't mind about it).

<@U03GQ47GNF4> are you using an Ingress instance for this? What comes after RTMP in this workflow?

I see, and it's audio-only? or are you sending video with it as well?

But we are planning to use this flow for video too

In this case, it's possible it's a compatibility issue with Ingress and Flutter for Android. Can you file an issue in the <https://github.com/livekit/ingress|Ingress> repo so we can investigate?

having the ffmpeg command you've used to reproduce would be helpful

<https://livekit-users.slack.com/archives/C01KVTJH6BX/p1686308751356949?thread_ts=1684834597.002079&amp;cid=C01KVTJH6BX|https://livekit-users.slack.com/archives/C01KVTJH6BX/p1686308751356949?thread_ts=1684834597.002079&amp;cid=C01KVTJH6BX>

We are spawning ffmpeg process using node.js

In this code snippet `toHHMMSS` transforms seconds to MM:SS format, and we pass list of mp3 files with `concat`

<@U01KZ61GV5Z> I opened new issue:
<https://github.com/livekit/ingress/issues/115>

<@U03GQ47GNF4> thank you. Yes I meant including the details in the GH issue so the developer taking a look could reproduce.

It's easy for things to get lost in Slack threads.

Hi <@U01KZ61GV5Z>. It looks like I've managed to find a solution to my problem. I switched `-c:a` from `aac` to `mp3` and added `-vbr 5`.
It would be great if someone added section to docs on how to use ffmpeg to stream video/audio to room using Ingress and potential pitfalls like one I experienced so it would be not just try and error method. There is a section in livekit-cli REAME but it's pretty small. This PR conversation was helpful too when I first tried to stream something to a room: <https://github.com/livekit/livekit-cli/pull/25>
Thanks

Hi <@U02GAC58A79>. It looks like the problem with dropped frames arised once again. To reproduce I've used the code from this branch
<https://github.com/livekit/client-sdk-flutter/tree/feat/prejoin-example>
In cloud dashboard I set supported codecs to VP8 only in this case (see attached screenshots). Something very strange is happening with stats: sum of decoded and dropped frames is not equal to the number of received

Hello again. Any updates on this one? Still can reproduce with version 1.4.2 of livekit client. I've added logging to write stats to influx DB every 20-30 seconds. Below are the plots of decoded and dropped fractions of received frames for android device (first) and iphone (second). Both have inconsistency with sum of decoded and dropped fractions is not equal to 1, but iphone device has 0 dropped frames and video from android is clean without delays, whereas the video from iphone on android device has 10-15 seconds delay.

<@U02GAC58A79> in order to make everything work made a build with version 1.3.3 (prior to dropped frames fix for Android) of livekit client and enabled support of VP8 only in cloud dashboard. This build works fine somehow (none of our testers complained about video quality issues).

The biggest difference between 1.3.3 and 1.4.x is that 1.4.x uses libwebrtc m114, I will check again next week

<@U03GQ47GNF4> to understand the issue, is the problem when encoded in vp8, the receiver (Android) exhibits a low framerate?

Hi <@U01KZ61GV5Z>. Not a low framerate but a delay. It can be quite big: 10-15 seconds

huh. wow.. that's insane.. and the publisher is also Flutter client on Android?

oh nvm, it's from iPhone.. I see in your original message now

I may have provided the wrong version number. I guess I actually used 1.3.2 version of client. Sorry for that. I also got lost in all these versions while finding the correct configuration for everything to work properly

no problem.. 1.3.2 works properly? or do you mean that is failing for you too

I mean that the build with version 1.3.2 of livekit client together with VP8 being the only codec enabled at backend works fine

ok, thanks, we'll look through to see what the difference might be

Confirmed again, 1.4.x uses WrappedVideoDecoderFactory correctly. When using VP8 or H264 decoding, the only difference is that m104 is upgraded to m114

<@U03GQ47GNF4> Which Android phone model are you using? There is another possibility that when decoding multiple VP8 tracks, there will be frame drops due to device performance limitations, perhaps through the subscriber setting the quality layer to medium or low

Hi <@U02GAC58A79>. We are using EB2103, ONEPLUS A6000, Umidigi F2

Hey <@U01KZ61GV5Z> <@U02GAC58A79>. I've made one mistake once again (I wrote about it to David some time ago). In my project I forgot to commit one quite important update: set video parameters for camera track to 360p in preview page. The track was created with default parameters (which means 720p) and passed to fast connect options. Which may explain frame drops. I don't know how I missed this part but I'm very sorry for that. I'll test again and let you know if anything have changed.

sorry, I'm not quite following. are you still experiencing drops and/or delays? if so could you describe the repro steps in detail? ideally it could be filed as an GH issue. Slack is a bit difficult to track

As I said earlier: currently I'm using version 1.3.2 with VP8 only enabled at the backend. I'll use this configuration for a while since it works best. After I finish more important tasks on my project, I'll try a newer version of livekit client and let you know if something is not working correctly.

Hey, what do you mean by the audio recognition rate?

The GPTTrack is the downstream track, this is where Kitt “speaks”

Since we’re not reencoding the audio on the server side, you need to edit the client parameters to edit the sample rate

When I started to talk to the ai, the subtitles displayed were a little different from what I said. I wanted to improve the recognition effect of this part. How should I adjust it?

<@U022KB72NFJ> I think audio recognition is the sample rate passed into Google STT.

The code is <https://github.com/livekit-examples/kitt/blob/main/lkgpt-service/pkg/service/transcriber.go|here>, I don’t think GCP gives options to edit the interim results

The options are defined <https://github.com/livekit-examples/kitt/blob/24ffc914e65de967b166af11f15e2650a8330860/lkgpt-service/pkg/service/transcriber.go#L288|here>

(We’re just forwarding the Opus data from the client to GCP)

Are the parameters here the result of your tuning?
```config := &amp;sttpb.RecognitionConfig{
		Model:             "command_and_search",
		UseEnhanced:       true,
		ProfanityFilter:   true, // replacing all but the initial character in each filtered word with asterisks, e.g. "f***".
		Encoding:          sttpb.RecognitionConfig_OGG_OPUS,
		SampleRateHertz:   int32(t.rtpCodec.ClockRate),
		AudioChannelCount: int32(t.rtpCodec.Channels),
		LanguageCode:      t.language.TranscriberCode,}```