LiveKit Community

if you want to send the message to everyone in the room, destinationSids can be left empty/nil

Which version of livekit server are you running?

Egress works only with livekit 1.0+. please update to the latest and it should work

const info = await egressClient..startWebCompositeEgress(_roomName_, 'grid-light', output);

Hi Romain, a couple of questions:

1. does `deleteRoom` always fail for you? or does it only fail when you are deleting an already-closed room?
2. which version of the server are you running

1. It seem to fail very regularly.
I first encountered the error when trying to close a not-already-closed room so I assumed I have to move the function call into the webhook callback to make sure it was closed before deleting it.

But to no avail.

2. I am using
• Docker version 20.10.17, build 100c701
• NodeJS docker container v16.13.0
• livekit-server-sdk v1.0.1 (nodeJS dependency)
• Egress docker container v1.0
• Livekit docker container v1.0

Thank you for the report. we'll investigate

x-posting for reference - <https://livekit-users.slack.com/archives/C01KVTJH6BX/p1655972260669129?thread_ts=1642070438.096200&amp;cid=C01KVTJH6BX>

image.png

Hi <@U02CM28PB4Y>, ~the version 1.1.0 does not appear to be published on the NPM registry as of now~
EDIT: derp

Sorry <@U03KRLGTXTK> I was referring to the livekit server version - <https://github.com/livekit/livekit/releases/tag/v1.1.0>. Having said that and looking back through the thread, I do not see which version you were using before. If you are using something before v1.1.0, can you please try it?

Oh, I see. I will try it and report back later

I just tried with the livekit docker container v1.1.0

*DISCLAIMER: I am using egress to record the audio and video tracks*

• when invoking `deleteRoom()` before the `room_finished` webhook triggers, I still get a 500 error stating `operation cannot be completed`
``` response: {
    status: 500,
    statusText: 'Internal Server Error',
    data: { "code": "internal", "msg": "operation cannot be completed", "meta": { "cause": "*errors.errorString" } }
}```
• when invoking `deleteRoom()` in the `room_finished` webhook, I get a 500 error but that time it is complaining about the room not existing (which in my opinion should not throw an error)
``` response: {
    status: 500,
    statusText: 'Internal Server Error',
    data: { "code": "internal", "msg": "could not find object", "meta": { "cause": "*errors.errorString" } }
}```

Would you happen to have server logs you can attach here?

is `INFO` level sufficient or would you need  `DEBUG` logs?

If you have `DEBUG` , better :slightly_smiling_face: If not, `INFO` is good.

If you don't mind, can you also put down the sequence you are doing?
• Are you creating room from `server-sdk-js` or some web client creating it?
• The egress tracks you are pulling, who is publishing them? The web client?

1. The room is first created by the <https://github.com/livekit/server-sdk-js|server-sdk-js> 
2. Then the <https://github.com/livekit/client-sdk-js|client-sdk-js> creates the tracks for the given user, and publishes them
3. Then the user starts the recording by clicking a UI button which triggers the creation of the TrackCompositeEgress on the server side
4. Upon leaving the call, the TrackCompositeEgress is stopped and then we try deleting the room (we tried either before or after the webhook `room_finished` triggers)

• Livekit v1.1.0 debug logs when trying to delete the room during the `room_finished` webhook: <https://pastebin.com/sczvdaAm>
• Livekit v1.1.0 debug logs when trying to delete the room BEFORE the `room_finished` webhook triggers: <https://pastebin.com/bGN0HB83>

Thank you Romain. Not spotting anything strange in the logs.

On the error when there is no room, yeah, 500 does not make sense. It might be a twirp thing. Will check how we can massage that.

I am trying to reproduce this, but I am unable to. This is the exact sequence I am doing (I am using server SDK Go as I am not set up to use server SDK JS yet, but I don't think that matters).
• Create room from `server-sdk-go`
• Join that room from `client-sdk-js` sample app and publish audio/video
• Start track composite egress for the tracks published by the sample app
• `DeleteRoom` under three scenarios
    ◦ When both the web client and egress is in the room - deletes the room correctly
    ◦ After the JS client leaves (which ends egress also because tracks are gone), but before the room is closed (there is a wait before room is close) - deletes the room correctly.
    ◦ After the room is closed - returns the `could not find object` error.
I am not able to reproduce the first error you are seeing. Can you spot difference between your sequence and my sequence?

In the scenario n°2 (`After the JS client leaves (which ends egress also because tracks are gone), but before the room is closed (there is a wait before room is close) - deletes the room correctly`) I believe in my case the egress instance(s) are stopped prior to the JS client actually disconnecting

Okay, will try
• stopping egress
• disconnect JS sample client
• `DeleteRoom`.

Not able to reproduce with the above sequence as well :disappointed:

My setup involves
• one NodeJS docker container for the backend server (`v16-alpine`, running with livekit-server-sdk `v1.0.1`)
• one Livekit docker container (`v1.1.0`)
• one Egress docker container (`v1.0.4`)
• JS client running locally with livekit-client (`^1.0.1`)

In my case , the error `operation cannot be completed` is thrown reliably (100% of the time) if the invocation of `deleteRoom` is made prior to receiving the webhook *`room_finished`*

• Is there anything the JS client should be doing on call closure? -- aside from `disconnectFromRoom()`
• I also wondered what happens if Egress has not yet finished processing or uploading the data when `deleteRoom()` is invoked. Could it be the cause of the error `operation cannot be completed`?

`disconnectFromRoom` should be enough. Don't think Egress will be an issue either. I will try to reproduce your set up (will also look at adding more logs to the server to see if that provides more insight).

Thank you very much :slightly_smiling_face:

have you deployed an egress cluster? is it pointed at the same Redis instance? Can you see what egress logs say?

Yes, could I send you via private chat? I would not leak any private info. I will post later here the remain conversation

hey Angel, we would prefer to help folks in the public channels since it'll be useful for others with similar issues, and for me to pull in others if I don't know the solution.

Also, it would be really helpful if you can keep the response concise, it'll make it easier for me and others to follow along. :pray:

ok, this the egress logs in the cluster. It seems to connect fine to redis instance and livekit:

`$ kubectl logs instance-livekit-XXXX-egress-cbXXXXX-qXXXXXX`
`+ pulseaudio -D --verbose --exit-idle-time=-1 --system --disallow-exit`
`W: [pulseaudio] main.c: Running in system mode, but --disallow-module-loading not set.`
`N: [pulseaudio] main.c: Running in system mode, forcibly disabling SHM mode.`
`I: [pulseaudio] main.c: Daemon startup successful.`
`+ exec ./egress`
`{"level":"info","ts":1654888703.8246362,"logger":"egress","caller":"messaging/bus.go:16","msg":"connecting to redis work queue","nodeID":"NE_6H94rNpSmE5j","addr":"<http://10.110.XXX.XXX:6379|10.110.XXX.XXX:6379>"}`

I get this fixed.

The issue was that setting number of replicas * max cpu in resources * nodes, was excedding CPU.

I lower replicas to 1 and put CPU as 1000m since I have 2000m per node.

After that, I set log_level: debug for egress helm values.yml

So when I inspect the logs for the pod, It was saying when they have low cpu.

Also since livekit 1.0.4 was released, there are new config for cpu_cost available, thanks to <@U01UB0LHTTQ>

And for digital ocean S3, in request.json I have to set the endpoint without the bucket name, only the CDN (with region) like:
`"endpoint": "<https://ams3.digitaloceanspaces.com>",`

ah ha! thanks for the update. This makes sense

<https://docs.livekit.io/|https://docs.livekit.io/>

Can I only see the live stream from others?

Hello <@U01KZ0SCDKL> I'm working with <@U03KSLUHA66> on a small MVP. I'm not a developer but I'll try to put it in plain English. 

The core idea is we want to offer a TVOD service that enables multi users to create live sessions and broadcast to multi clients, so it's not a conference call nor collaborative session, more like YouTube live stream, from source A —-&gt; to B, C, D, E,… viewers 

Thanks <@U03K5G2V5RT> and <@U03KSLUHA66>. Yes, you can do this. LiveKit offers selective subscriptions, so certain users can be broadcasting audio/video (publishers/producers) and others can simply be subscribers/consumers.

These broadcasters can be in separate “rooms” if they should be isolated, or, if subscribers can view multiple broadcasts simultaneously, then they can be in the same room.

Or, subscribers can also be in multiple rooms simultaneously. So you have flexibility on how to organize things.

Thanks <@U01KZ0SCDKL>. Another question. How do go about permissions on livekit? I.e: if creator/broadcaster wants to charge viewers a fee to have an access to their own live session, is there anything specific we needs to look at? 

<@U03K5G2V5RT> clients usually join LiveKit rooms via a JWT that is provisioned using the SDK. You can build the permission logic by generating the token only if the creator allows it or a fee has been paid. You should have everything you need in the docs

Hi Ihor, can you please thread your messages? Thank you!

hi, could you provide a minimal reproduction repo for the issue you are seeing?

<@U02EU2ZPTQU>

Getting a better understanding of rendering:
<https://alexsidorenko.com/blog/react-render-props/>

I've exctracted the AudioButton into a self contained component.
Works for me so far :wink:

Did it the same way with the other ControlButtons

1655342407438.png

Hi <@U03KC2Y3J06> please refrain from posting the same messages in multiple channels. we are all reading messages in both channels :slightly_smiling_face:

Can you please help on this question? Thanks.

If you are using kubernetes, you need create a service of type LoadBalancer for expose the ports, something like this config:

*YOURNAMESPACE-livekit-load-balancer.yaml* 
`apiVersion: v1`
`kind: Service`
`metadata:`
  `name: YOURNAMESPACE-livekit-load-balancer-service`
  `namespace: YOURNAMESPACE`
  `labels:`
    `app: YOURNAME-livekit-deploy`

  `annotations:`
    `<http://service.beta.kubernetes.io/do-loadbalancer-name|service.beta.kubernetes.io/do-loadbalancer-name>: "your domain name for your loadbalancer"`
    `<http://service.beta.kubernetes.io/do-loadbalancer-protocol|service.beta.kubernetes.io/do-loadbalancer-protocol>: "https"`
    `<http://service.beta.kubernetes.io/do-loadbalancer-certificate-id|service.beta.kubernetes.io/do-loadbalancer-certificate-id>: "yourcertificate-file-loadbalancer-livekit"`
    `<http://service.beta.kubernetes.io/do-loadbalancer-disable-lets-encrypt-dns-records|service.beta.kubernetes.io/do-loadbalancer-disable-lets-encrypt-dns-records>: "false"`
    `<http://service.beta.kubernetes.io/do-loadbalancer-healthcheck-port|service.beta.kubernetes.io/do-loadbalancer-healthcheck-port>: "7880"`
    `<http://service.beta.kubernetes.io/do-loadbalancer-healthcheck-protocol|service.beta.kubernetes.io/do-loadbalancer-healthcheck-protocol>: "tcp"`
    `<http://service.beta.kubernetes.io/do-loadbalancer-healthcheck-path|service.beta.kubernetes.io/do-loadbalancer-healthcheck-path>: "/"`
    `<http://service.beta.kubernetes.io/do-loadbalancer-healthcheck-check-interval-seconds|service.beta.kubernetes.io/do-loadbalancer-healthcheck-check-interval-seconds>: "3"`
    `<http://service.beta.kubernetes.io/do-loadbalancer-healthcheck-response-timeout-seconds|service.beta.kubernetes.io/do-loadbalancer-healthcheck-response-timeout-seconds>: "5"`
    `<http://service.beta.kubernetes.io/do-loadbalancer-healthcheck-unhealthy-threshold|service.beta.kubernetes.io/do-loadbalancer-healthcheck-unhealthy-threshold>: "3"`
    `<http://service.beta.kubernetes.io/do-loadbalancer-healthcheck-healthy-threshold|service.beta.kubernetes.io/do-loadbalancer-healthcheck-healthy-threshold>: "5"`
`spec:`
  `selector:`
    `app: livekit-server`
  `ports:`
    `- name: https`
      `protocol: TCP`
      `port: 443`
      `targetPort: 80`
    `- name: http`
      `protocol: TCP`
      `port: 80`
      `targetPort: 7880`
  `type: LoadBalancer` 

7880 port bascially to connect with livekit. In local environment this will be working but in production environment you'll require to use a load balancer &amp; SSL certificate otherwise user won't be able to connect with livekit.

How about if I don't have load balancer before 7880，then 7880 is exposed, right?

nope, this is really for logging/analytics. the main option that has an impact is `protocol`, that dictates what the client could support

Wonderful. Thanks <@U01KZ61GV5Z>. Much appreciated.

oops.. had a typo in the reply.  corrected :)

All Good. I have all this working in Unreal, but I wasn't seeing any changes when modifying these 2 properties, so I was curious. Thanks for clarifying.

<@U02MAJLGN65> Anything I can do about this?

We were previously using OpenVidu and we didn’t have this problem

for openvidu, did you just use the default webrtc audio device module or were there any modifications there? I'll see if I can compare the differences in the underlying webrtc library

<@U03DURSG2G6>, which devices are you noticing echo problems on?

<@U01KZ61GV5Z> it is several ones. for example a Samsung S8 (with Android 9): this one generates echo for all the other participants in the room if the `audioTrackCaptureDefaults` are true, but it doesn’t if they are false

in other cases, for example a `pixel 2` we need to enable hardware cancellation not to get echoes &amp; audio artifacts (by setting `UseHardwareAcousticEchoCanceler` to true in the audio module)

thanks, we'll file an issue and will try to repro + resolve. The goal is to have these kind of things be contained in the SDK itself.

<https://github.com/livekit/client-sdk-android/issues/93>

thanks! we suspect we may not want to set the `goog*` flags at all by default? maybe they should be optional. We noticed that with OpenVidu we didn’t set those at all and it was working better.  We used only the `UseHardwareAcousticEchoCanceler` ones. I’ll try to check if I can confirm that or not

Hey <@U03FWUU3P7C>, empty_timeout is actually used *before* a participant joins. Once a participant joins and then leaves the room, it'll be subject to a standard closure timeout.

think of it as "create the room and close it after this timeout if no one joins"

Hello David, I got it,  thank you for your reply.

<@U01KZ61GV5Z> when you say:
_“Once a participant joins and then leaves the room, it’ll be subject to a standard closure timeout.”_
We have rooms created in the past still returned in the room list and with zero participants. Do you know why this happens and if is it possible to clean the room that is already with no participant from a while?

An individual room is hosted on a single node. When a second participant connects, they connect to the room via that node, in whichever region it resides.

Does this works in case of a <https://docs.livekit.io/deploy/distributed/|distributed setup>? Should the redis be common between regions?

You probably should hide your api key and secret :wink: Putting public is a risk

Restarting the redis instance (which is a data loss) solve the issue. So, livekit does support a load balancing redis setup?

You have to set up a HA Redis instance, but remember to point LiveKit at the Redis master, not read slaves.

read-slaves cannot be written to, so that explains the error message you are seeing

The error is because is filled fully, in that state is only read mode, because no more space in redis, no because is master or slave

you can publish static files to the room as a participant/track. you can do it either with our <https://github.com/livekit/livekit-cli#publish-files-as-tracks|CLI>, or <https://github.com/livekit/server-sdk-go#publishing-tracks-to-room|Go SDK>. you'd have to encode the files into opus first.

In case you're curious, my goal is to play live music in the "Lobby" of virtual offices. Seems like I'd have to stand up a separate service that creates a streaming track for every lobby. Technical cost is too high for now, but I'll keep it on the backlog :slightly_smiling_face:

hmm we don't have a dedicated way to do this at the moment; you'd have to create your own `VideoCapturer` to generate the video frames and then create a local video track with it

i see,
is there any code snippets for reference?

hmm i'm not sure about this actually we are processing every frame captured from camera through `onSurfaceTextureUpdated`  and after that we are processing bitmap from it and then we want to push that frame.

if you already have the frame processing from the camera in place, then you can simply create a `VideoCapturer` wrapper that emits frames to the capture observer.

<https://github.com/livekit/client-sdk-android/blob/main/video-encode-decode-test/src/main/java/io/livekit/android/videoencodedecode/DummyVideoCapturer.kt|This test VideoCapturer emits a single color at a time, but you can reference it for emitting full frames>.

livekit3.mov

Hey <@U01NZKTNE1H> thanks for guidance.

i have implemented as you have mentioned in your previous message but there are some issue with sending video frames
```val buffer = NV21Buffer(bitmap.compress(Bitmap.CompressFormat.JPEG, 100).toBytes(), bitmap.width, bitmap.height, null)

val frame = VideoFrame(buffer, 0, TimeUnit.MILLISECONDS.toNanos(SystemClock.elapsedRealtime()))

this.capturerObserver?.onFrameCaptured(frame)

frame.release()```
this is how i am preparing frames is there anything wrong and you can se output in video.

you're passing in a bitmap buffer to NV21Buffer, but they're incompatible types. you need to convert the bitmap bytes to nv21 format if you want to use NV21Buffer.

<https://stackoverflow.com/questions/5960247/convert-bitmap-array-to-yuv-ycbcr-nv21/13055615#13055615>

alternatively, you can also use the similar `encodeYUV420SP` method from the previously linked DummyVideoCapturer to use JavaI420Buffer

<@U01NZKTNE1H>
It is good solution but there are some issue with this approach, like device heating issue and not supporting all screen size devices

is there any better and optimised approach?

not sure why screen size would matter. the encoding type isn't dependent on screen size.

you can try using YuvHelper methods to use the libyuv native methods which should be faster if that's an issue

```private fun getVideoFrame(bitmap: Bitmap, width: Int, height: Int): VideoFrame.Buffer {
        val byteCount = bitmap.byteCount
        //get bytes from bitmap into buffer, bitmap config is ARGB_8888
        val argbBuffer = ByteBuffer.allocate(byteCount)
        bitmap.copyPixelsToBuffer(argbBuffer)
        argbBuffer.rewind()

        //reorder argb to abgr
        val abgrBuffer = ByteBuffer.allocate(byteCount)
        for (i in 0 until byteCount step 4) {
            abgrBuffer.put(argbBuffer.get(i + 0))
            abgrBuffer.put(argbBuffer.get(i + 3))
            abgrBuffer.put(argbBuffer.get(i + 2))
            abgrBuffer.put(argbBuffer.get(i + 1))
        }
        abgrBuffer.rewind()

        //convert abgr to YUV
        val yuvBuffer = JavaI420Buffer.allocate(width, height)
        YuvHelper.ABGRToI420(
            abgrBuffer,
            bitmap.rowBytes,
            yuvBuffer.dataY,
            yuvBuffer.strideY,
            yuvBuffer.dataU,
            yuvBuffer.strideU,
            yuvBuffer.dataV,
            yuvBuffer.strideV,
            width,
            height
        )

        return yuvBuffer
    }```

this is what i found online but it seems to be not working
actually i am not very familiar with webrtc concepts...

you have code in there that does `//reorder argb to abgr`. try doing `argb to rgba` instead

```private fun getVideoFrame(bitmap: Bitmap, width: Int, height: Int): VideoFrame.Buffer {
        val byteCount = bitmap.byteCount
        //get bytes from bitmap into buffer, bitmap config is ARGB_8888
        val argbBuffer = ByteBuffer.allocate(byteCount)
        bitmap.copyPixelsToBuffer(argbBuffer)
        argbBuffer.rewind()

        //reorder argb to abgr
        // rgba instead of abgr
        val abgrBuffer = ByteBuffer.allocate(byteCount)
        for (i in 0 until byteCount step 4) {
            abgrBuffer.put(argbBuffer.get(i + 1))
            abgrBuffer.put(argbBuffer.get(i + 2))
            abgrBuffer.put(argbBuffer.get(i + 3))
            abgrBuffer.put(argbBuffer.get(i + 0))
        }
        abgrBuffer.rewind()

        //convert abgr to YUV
        val yuvBuffer = JavaI420Buffer.allocate(width, height)
        YuvHelper.ABGRToI420(
            abgrBuffer,
            bitmap.rowBytes,
            yuvBuffer.dataY,
            yuvBuffer.strideY,
            yuvBuffer.dataU,
            yuvBuffer.strideU,
            yuvBuffer.dataV,
            yuvBuffer.strideV,
            width,
            height
        )

        return yuvBuffer
    }```
there isn't any method in YuvHelper for RGBA
this is what i found <https://github.com/alzybaad/RGB2YUV> is this helpful?

did you try it with ABGRToI420? it mentions that it uses RGBA in memory format on its description, so it's worth a try

Screenshot 2022-07-06 at 1.30.18 PM.png

Hi, I am very new to LiveKit. Do you have any video tutorials on LiveKit - Unity integration for WebGL

Hey Neil! I'm not too familiar with contentHint, what fps do you see when setting `simulcast: false` . which API are you using to capture screenshare?

hey! simulcast is already set to false. I'm using:
```const screenTracks: LocalTrack[] = await createLocalScreenTracks({ audio: true, resolution: { width: 1280, height: 720, frameRate: 24 } });```


docs around content-hint: <https://www.w3.org/TR/mst-content-hint/>

tried with both h264 and vp8 both with the same results. And see the low fps issue on the vanilla webrtc sample: <https://www.webrtc-experiment.com/Pluginfree-Screen-Sharing>

FPS being captured is high, FPS being sent is low unless I sent contenthint to "motion" in which case fps is fine, resolution is low

Screen Shot 2022-06-21 at 7.38.00 PM.png

:thinking_face: looks like the low fps is specifically an issue with VP9: <https://bugs.chromium.org/p/webrtc/issues/detail?id=13016>

I'm not seeing people mention this being a problem with VP8. just tried sharing it with our example app. we capped the capture resolution at 15 by default

as indicated in your doc.. contentHint is used by browsers to make the trade off between fps and resolution *when there is resource constraint*. I don't believe it's used when you have sufficient resources

Hrmm i'm setting fps to 24 and setting max bitrate but I'm definitely seeing this either-fps-or-resolution behavior.

It's possible that webrtc thinks I'm limited by something. I'm not bandwidth constrained but perhaps it thinks i'm cpu constrained even though task manager shows my cpu is fine. Is there an easy livekit way I can try the gpu encoder? I'm not sure i've seen it in the docs

we don't have any control over the encoder used. Safari generally tries to use hw encoding for H.264

i see, do you know of any sdp jui jitsu that accomplishes it?

I'm not sure. 720p 24fps seems pretty straight forward to encode though.

yeah certainly something like this on OBS with h264 on ultrafast or very fast preset would be easy on my setup

&gt; I'm not seeing people mention this being a problem with VP8. just tried sharing it with our example app. we capped the capture resolution at 15 by default
are you able to see the full 15fps 720p in the example app on the receiving end?

this is the only example app I see: <https://example.livekit.io/>

Is there one for screenshare?

hrmm i'm able to get good results with simulcast turned on. Seems I can only reproduce my issue when simulcast is off - but anyways I don't need it off so no big deal :shrug:

that's strange.. I wonder what flags/toggles is influencing Chrome behavior

If not, can I have some idea to further reduce the pressure at both client &amp; server side?

<@U03LNC59919> What sort of pressure are you seeing on server side?

Yes, it is possible to limit the number of forwarded tracks with some work. Some pointers
• This worker runs periodically and determines active speakers - <https://github.com/livekit/livekit/blob/120110fbbee1375905e06a629461112895f19bc6/pkg/rtc/room.go#L914>
• That active speaker list is sorted by audio level.
So, based on that list, it is possible to change forwarding dynamically.

A few things to keep in mind
• That is at a participant level. If a participant has multiple microphone tracks (mostly it won't), it will check if any of them is active and report the one with the loudest level.
• The levels are not normalised across participants. So, if a participant has some loud background noise, it is possible that it is in the top 5 all the time.
• Once you determine which tracks are forwarded and which should be throttled, that can flow back via 1 `pkg/rtc/participant.go -&gt; pkg/rtc/mediatrackreceiver.go -&gt; pkg/sfu/receiver.go`
• You can either stop forwarding in `receiver.go` or broadcast that change to all the subscribers (`downtrack.go` - an example is where video layer availability changes are broadcast to down tracks - <https://github.com/livekit/livekit/blob/120110fbbee1375905e06a629461112895f19bc6/pkg/sfu/receiver.go#L394>) and let `DownTrack` forward/throttle.
• This could happen several times a second (active speaker determination runs every 400 ms by default).
• As this happens only every 400 ms by default, there is potential for front end clipping, i. e. a silent user may start talking and enter the top 5, but a portion of the start of their speech could be lost because of sampling.
• If you want to prevent the above, it will be a bigger change. i. e., the algorithm in this file (<https://github.com/livekit/livekit/blob/master/pkg/sfu/audio/audiolevel.go>) will have to change to look for onset of speech/voice activity and notify to rest of the system and the top 5 determination run again in those notification path. That could add pressure on server side unless these events are throttled as large number of unmuted publishers means potentially some or the other could detect voice activity and constantly acting on it.

<@U02CM28PB4Y>

Thanks for ur wonderful explanation. It confirms that my understanding of the code is correct.

Yes, we want to prevent the clipping effect and we don’t mind a bigger change to the code.

There are two possible ways,

1.
We make use of `room.GetActiveSpeakers()` and write a background worker that is similar to `audioUpdateWorker()` but with less sleep time, maybe 20ms and determine which track should be throttled and constantly notify the system.

But the clipping effect can still happen, we cannot guarantee the notification will come first or the packet will be dropped first?

2.
Do it at package level, we add a hook to all track receiver in one room and add the SmoothedAudioLevel to `buffer.ExtPacket`.

So when a package is prepared to be broadcasted, we query the hook to check if it is the current loud package and decide whether to be broadcasted or not.

However, in this way, I am afraid it will introduce too many extra server overhead, and the hook itself may be hard to be implemented (if update in real time, need a lock and need fight for update; if update in cron job, clipping effect can still happen).


Can I have ur opinion on Method 2? Is it doable? How the hook may be implemented?

Meanwhile, can I have ur thought on which method is better also?

THX so much!!! Really impressed by the community~

Sure <@U03LNC59919>.

See benchmarking results here for the `Audio Only` case - <https://docs.livekit.io/deploy/benchmark/>. It could handle close to 1M packets on a 16-core machine in GCP. That test was using 10 publishers and 3000 subscribers. The ingress rate is 300+ pps. So, subscription side is 3000 * 300 = ~900K pps.

In your case, all 1000 are publishing and unmuted. DTX would send about 2.5 pps IIRC (one packet every 400 ms). Assuming 10 people are actively speaking and the rest are silent, that would be an input packet rate of 10 * 50 + 990 * 2.5 = 2975. At 1000 subscribers, you are looking at 2.975M.  That is significant packet load on the server.

First thing would be to analyse that number, compare against the bench marks and figure out what is reasonable.

But, if you are forwarding only 5 and if all 5 are speaking all the time, the packet load would be 250 * 1000 = 250K which is much more manageable on 12 core machine.

That will give you enough head room to implement #2. Yes, it is per packet, but should not be too bad. The way I would think of #2 working is that there is a threshold for the room. Basically, the room worker is running every 20 ms (your method 1) and figuring out the top 5 and setting a threshold for that window. So, anything above that threshold will go through.

But, all this will lead to clipping. The way I have thought about this selective forwarding is using VAD per track. i. e. change the audio level algorithm (or add a VAD module to each track). It will declare whether a track has voice activity or not. In fact, the top bit in audio level RTP extension is supposed to indicate if that packet has voice activity, but I don't think browsers set that properly. If that was set properly, you could use that directly. So, forward packets as long as that module says there is voice activity. This algorithm will avoid front-end/back-end clipping and will also have the benefit of not forwarding silent tracks. The one down side is if all 1000 people speak, then this will blow up. Some other guard rail would be needed for that. But, from a quality perspective, this would work out the best.

BTW, have you checked limitation on client/browser side? Maybe, browsers cannot handle that many `&lt;audio&gt;` elements either?

BTW, I work at LK. So, I delve in this code daily :slightly_smiling_face: Happy to be of help :slightly_smiling_face:

THX for your replay, I will check client side limitation also.

Please keep us posted on what you find <@U03LNC59919>. Very interested in the results. Also open to accepting contributions to the repo if you pick a strategy that works for your use case.