LiveKit Community

Well you can make the request yourself, but I don’t know if that’s easier than using livekit-cli

The thing is that we want to embed the publish logic in a docker module, where we want easy control over starting and stopping publishing. If use the livekit-cli this would mean starting a subprocess in python (or some other language) and killing it, I guess?

for just voice, it should be fine to use the sdks without utilizing any ui elements

can't say I have any experience with capcitator, but from what I gather, it's rendered in a web view, so I'm guessing you'd be working with the JS SDK? (you could write wrappers around the native sdks, but for your purposes, that might be overkill)

there shouldn't be any problems there; we've tested the JS SDK working on mobile browsers

for back-end sdks, we have <https://docs.livekit.io/references/server-sdks/> which is oriented for controlling the livekit-server, but given you want to play audio over the webrtc voice channel, you'll need WebRTC functionality which only the Go server SDK supports currently.

Thank you <@U01NZKTNE1H> for the information!

nvm, I see it's Cloud. We are investigating a couple of issues with some of the Room APIs with Cloud. will keep you posted. btw, we also have <#C048FRL1N2C|> for cloud specific issues

This issue has been resolved. Cloud should not return 500 errors anymore.

Let me know if you guys have any other issues with it.

<@U01KZ61GV5Z> we observe the above same issue with server-sdk-js calling in OSS (with own hosted k8s livekt servers), will there be a patch for that as well ?

Hey Ashan, can you start a new thread on what you are seeing? thread with code snippets would be helpful. :pray: 

Sure <@U01KZ61GV5Z> will share. Thank you for the update.

you can change the source resolution with <https://docs.livekit.io/client-sdk-js/classes/LocalTrack.html#replaceTrack|replaceTrack> API

In the react SDK I couldn't find this API inside the given hierarchy.

Can you point me out the hierarchy to reach the replaceTrack API

starting from the localParticipant it’s `localParticipant.getTrack(...).track.replaceTrack()`

Thanx, found it. But replceTrack only accept the MediaStreamTrack, so how can we replace or change the resolution / quality of the ongoing video stream with it.

Hi all, I'm having an issue in call conferences with China users. I'm using LiveKit Cloud as server and flutter sdk on client.
It seems that China users are not able to publish their tracks (on LiveKit Cloud instance dashboard I am not able to see the TRACK_PUBLISHED event).
I have noticed that China users connect to a Cloud instance located in USA.
Can this be an issue related to China firewall?
Thanks
Stefano

we are working on it! The repo isn't public yet as we are just wiring the pieces together and there's a ton of flux. It should keep almost an identical API to the WebGL SDK though.

CC <@U022KB72NFJ>

Hi <@U04D2CC8TTL>, for subscribers it's quite straight forward. You can connect to the Room just the playground example, the only thing you'd need to do is to not publish your own video as a viewer. The audience will receive video of the publisher with very little effort. more docs on this <https://docs.livekit.io/client/receive/|here>

<@U01KZ61GV5Z> Thank you. I managed to do it. Works great. I am wondering what is the best way to achieve best videoquality.

it's automatic! you can publish high quality and enable adaptive stream on the subscriber side

hi, how did you get the server setup? the one thing to check is if the needed <https://docs.livekit.io/oss/deployment/ports-firewall/|ports> are open. You can also use the <https://livekit.io/connection-test|connection tester> to see if your deployment is set up correctly.

I have tried the below function, while encountered this warning
`adaptive stream is enabled, cannot change track settings`
```publication.setEnabled(false)```

Can I simply remove the video element to achieve this?

If you want to mute the remote participant only for yourself (on receiver side) you can unsubscribe his video track.
If you want to mute it for everybody I guess you’ll have to implement some kind of communication over signalling layer and participant will unpublish his video himself

Gocha, thanks <@U03L2F74CF7>. While I think Unsubscribe is too expensive…, it’s really a trade off

Quite contrary… If you unsubscribe you stop receiving the video completely - thus saving bandwidth and data.
However hiding the video or detaching it from DOM will do the same thing (unsubscribe) due to adaptive streaming.
With manual unsubscribe you’re just in better control over what’s happening and not relying on livekit “magic”

Yeap. With `adaptive` will also detect the video’s size to save bandwidth as long as the visibility

<@U03REEFGQQL> as you said, you could either detach the Track from all video elements, or simply keep them hidden. Adaptive stream will kick in and ensure the layers are disabled.

<@U04D5P5N8G5> please ask your question once and thread your messages. I’m going to delete the root message here since I see you already have a thread going on <#C01KVTJH6BX|>.

Can you please help me i want to implement room setup where only admin can access all thing like admin can mange audio video of user 


If user want to speak than user need to ask to admin for permission if admin give permission than only user can enable their audio.

that's excellent news thanks <@U01KZ61GV5Z>!

does this work for you? <https://github.com/livekit/livekit-cli#simulating-audio-publishers>

I shall try thanks <@U01NZKTNE1H> :slightly_smiling_face:

seems like you've ran out of file descriptors. <https://github.com/livekit/livekit-cli#configuring-system-settings>

Thank you for the feedback <@U01KZ61GV5Z>. my test initiator server spec is  x4 core , 16GB server in GCP.

After applying system settings in load testing server, in order to isolate the issue, I set the pod resources to below and set replica to be 1 (min an max) and same error can be seen now with ~200-300 connected users in livekit-server

Pod:
```autoscaling:
  enabled: true
  minReplicas: 1
  maxReplicas: 1 
  targetCPUUtilizationPercentage: 60

resources:
 limits:
   cpu: 300m
   memory: 2048Mi
 requests:
   cpu: 200m
   memory: 1024Mi  
```
Issue:
```2022-12-02T20:30:15.388Z | 2022-12-02T20:30:15.388Z INFO livekit service/utils.go:12 error handling request {"participant": "xtxlt_261", "room": "ashan-load-test-video-only", "remote": false, "status": 500, "error": "could not lock room"}
```
So after going through the code found the culprit;

```pkg\service\errors.go ::L15
ErrRoomLockFailed        = errors.New("could not lock room")
```
```pkg\service\redisstore.go :: L248

func (s *RedisStore) LockRoom(_ context.Context, roomName livekit.RoomName, duration time.Duration) (string, error) {
    token := utils.NewGuid("LOCK")
    key := RoomLockPrefix + string(roomName)

    startTime := time.Now()
    for {
        locked, err := s.rc.SetNX(s.ctx, key, token, duration).Result()
        if err != nil {
            return "", err
        }
        if locked {
            return token, nil
        }

        // stop waiting past lock duration
        if time.Since(startTime) &gt; duration {
            break
        }

        time.Sleep(100 * time.Millisecond)
    }

    return "", ErrRoomLockFailed
}
```
It seems livekit was *unable to acquire a lock from redis*. Do you have any recommend settings for the redis cluster that we need to use if we are support 2000 -5000 at least on single room. For the test setup we have setup helm redis - 1x master and 3x slave nodes with replication.

<@U01KZ61GV5Z> just want to make sure that the above issue related to redis rather livekit pod system settings restrictions ( file descriptors , network throttle, etc.). Highly appreciate your opinion on this

<@U01KZ61GV5Z> my bad, thank you for the feedback, <https://docs.livekit.io/oss/test-monitor/#file-handles> -  should apply in pod / node.

When we set the redis to redis-cluster and attempt , we get following error in load testing initiating server, no errors in livekit-server in regard to `"error": "could not lock room"`  .

Load test server, server spec is x4 core, 16 GB. Already applied   <https://github.com/livekit/livekit-cli#configuring-system-settings> in load tester server;

```Test:
livekit-cli load-test --url wss://<doamin.com> --api-key XXXXXX --api-secret XXXX --room ashan-load-test-video-only --audio-publishers 1 --subscribers 2000```
```....
subscribed to track fwrvf_491 TR_AMa5pWb5YtLovB audio 2/5
subscribed to track fwrvf_491 TR_AMScJtg4h3np4U audio 3/5
subscribed to track fwrvf_233 TR_VCH5JpGCYN9Wup video 1/5
subscribed to track fwrvf_233 TR_AMScJtg4h3np4U audio 2/5
subscribed to track fwrvf_233 TR_AMa5pWb5YtLovB audio 3/5
subscribed to track fwrvf_441 TR_VCH5JpGCYN9Wup video 2/5
subscribed to track fwrvf_441 TR_AMScJtg4h3np4U audio 2/5
subscribed to track fwrvf_441 TR_AMa5pWb5YtLovB audio 3/5
panic: runtime error: slice bounds out of range [:6992] with capacity 4096

goroutine 42675 [running]:
bufio.(*Reader).Read(0xc02dced620, {0xc00f5670f4?, 0x40c?, 0x1500?})
        /opt/hostedtoolcache/go/1.18.7/x64/src/bufio/bufio.go:249 +0x2ed
<http://github.com/gorilla/websocket.(*messageReader).Read(0xc0216e1d78|github.com/gorilla/websocket.(*messageReader).Read(0xc0216e1d78>, {0xc00f5670f4, 0x40c, 0x40c})
        /home/runner/go/pkg/mod/github.com/gorilla/websocket@v1.5.0/conn.go:1050 +0x1c5
io.ReadAll({0x7f37867b4b90, 0xc0216e1d78})
        /opt/hostedtoolcache/go/1.18.7/x64/src/io/io.go:645 +0xfe
io/ioutil.ReadAll(...)
        /opt/hostedtoolcache/go/1.18.7/x64/src/io/ioutil/ioutil.go:27
<http://github.com/gorilla/websocket.(*Conn).ReadMessage(0x4?)|github.com/gorilla/websocket.(*Conn).ReadMessage(0x4?)>
        /home/runner/go/pkg/mod/github.com/gorilla/websocket@v1.5.0/conn.go:1097 +0x46
<http://github.com/livekit/server-sdk-go.(*SignalClient).readResponse(0xc010b2b5e0?)|github.com/livekit/server-sdk-go.(*SignalClient).readResponse(0xc010b2b5e0?)>
        /home/runner/go/pkg/mod/github.com/livekit/server-sdk-go@v1.0.5/signalclient.go:210 +0x95
<http://github.com/livekit/server-sdk-go.(*SignalClient).Join(0xc010b2b5e0|github.com/livekit/server-sdk-go.(*SignalClient).Join(0xc010b2b5e0>, {0x7ffe0489d7b6?, 0xc005a73b70?}, {0xc02549e240, 0x10b}, 0xc02adc3a50)
        /home/runner/go/pkg/mod/github.com/livekit/server-sdk-go@v1.0.5/signalclient.go:92 +0x292
<http://github.com/livekit/server-sdk-go.(*RTCEngine).Join(0xc0036945a0|github.com/livekit/server-sdk-go.(*RTCEngine).Join(0xc0036945a0>, {0x7ffe0489d7b6, 0x25}, {0xc02549e240, 0x10b}, 0xc02adc3a50)
        /home/runner/go/pkg/mod/github.com/livekit/server-sdk-go@v1.0.5/engine.go:99 +0x45
<http://github.com/livekit/server-sdk-go.(*Room).JoinWithToken(0xc0122d07e0|github.com/livekit/server-sdk-go.(*Room).JoinWithToken(0xc0122d07e0>, {0x7ffe0489d7b6, 0x25}, {0xc02549e240, 0x10b}, {0xc005a73d48, 0x1, 0x0?})
        /home/runner/go/pkg/mod/github.com/livekit/server-sdk-go@v1.0.5/room.go:170 +0xc5
<http://github.com/livekit/server-sdk-go.(*Room).Join(0xc0122d07e0|github.com/livekit/server-sdk-go.(*Room).Join(0xc0122d07e0>, {0x7ffe0489d7b6, 0x25}, {{0x7ffe0489d7e6, 0xf}, {0x7ffe0489d803, 0x2b}, {0x7ffe0489d836, 0x1a}, {0x0, ...}, ...}, ...)
        /home/runner/go/pkg/mod/github.com/livekit/server-sdk-go@v1.0.5/room.go:158 +0x299
<http://github.com/livekit/livekit-cli/pkg/loadtester.(*LoadTester).Start(0xc002f97810)|github.com/livekit/livekit-cli/pkg/loadtester.(*LoadTester).Start(0xc002f97810)>
        /home/runner/work/livekit-cli/livekit-cli/pkg/loadtester/loadtester.go:102 +0x368
<http://github.com/livekit/livekit-cli/pkg/loadtester.(*LoadTest).run.func1()|github.com/livekit/livekit-cli/pkg/loadtester.(*LoadTest).run.func1()>
        /home/runner/work/livekit-cli/livekit-cli/pkg/loadtester/loadtest.go:252 +0x78
<http://golang.org/x/sync/errgroup.(*Group).Go.func1()|golang.org/x/sync/errgroup.(*Group).Go.func1()>
        /home/runner/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 +0x64
created by <http://golang.org/x/sync/errgroup.(*Group).Go|golang.org/x/sync/errgroup.(*Group).Go>
        /home/runner/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:72 +0xa5 ```
Upon above error, we get following errors in livekit-server which is expected as cli stop load testing script;

```2022-12-02T18:59:53.175Z | 2022-12-02T18:59:53.174Z INFO livekit rtc/participant_signal.go:180 could not send message to participant {"room": "ashan-load-test-video-only", "roomID": "RM_D5kPyJb3uaj5", "participant": "dlnsl_1263", "pID": "PA_bsj3piPuBNLM", "remote": false, "message": "*livekit.SignalResponse_Update", "error": "no response sink"}
2022-12-02T18:59:53.175Z | 2022-12-02T18:59:53.174Z ERROR livekit rtc/room.go:865 could not send update to participant {"room": "ashan-load-test-video-only", "roomID": "RM_D5kPyJb3uaj5", "participant": "dlnsl_1263", "pID": "PA_bsj3piPuBNLM", "error": "no response sink"}
2022-12-02T18:59:53.175Z | <http://github.com/livekit/livekit-server/pkg/rtc.(*Room).sendParticipantUpdates|github.com/livekit/livekit-server/pkg/rtc.(*Room).sendParticipantUpdates>
2022-12-02T18:59:53.175Z |  /workspace/pkg/rtc/room.go:865
2022-12-02T18:59:53.175Z | <http://github.com/livekit/livekit-server/pkg/rtc.(*Room).subscriberBroadcastWorker|github.com/livekit/livekit-server/pkg/rtc.(*Room).subscriberBroadcastWorker>
2022-12-02T18:59:53.175Z |  /workspace/pkg/rtc/room.go:960
2022-12-02T18:59:53.175Z | 2022-12-02T18:59:53.175Z INFO livekit rtc/participant_signal.go:180 could not send message to participant {"room": "ashan-load-test-video-only", "roomID": "RM_D5kPyJb3uaj5", "participant": "dlnsl_1216", "pID": "PA_Fu8hK6Cd5qKe", "remote": false, "message": "*livekit.SignalResponse_Update", "error": "no response sink"}
2022-12-02T18:59:53.175Z | 2022-12-02T18:59:53.175Z ERROR livekit rtc/room.go:865 could not send update to participant {"room": "ashan-load-test-video-only", "roomID": "RM_D5kPyJb3uaj5", "participant": "dlnsl_1216", "pID": "PA_Fu8hK6Cd5qKe", "error": "no response sink"}
2022-12-02T18:59:53.175Z | <http://github.com/livekit/livekit-server/pkg/rtc.(*Room).sendParticipantUpdates|github.com/livekit/livekit-server/pkg/rtc.(*Room).sendParticipantUpdates>
2022-12-02T18:59:53.175Z |  /workspace/pkg/rtc/room.go:865
2022-12-02T18:59:53.175Z | <http://github.com/livekit/livekit-server/pkg/rtc.(*Room).subscriberBroadcastWorker|github.com/livekit/livekit-server/pkg/rtc.(*Room).subscriberBroadcastWorker>
2022-12-02T18:59:53.175Z |  /workspace/pkg/rtc/room.go:96```


Livekit server error pt. 2:

```2022-12-03T22:39:17.491Z | 2022-12-03T22:39:17.491Z INFO livekit rtc/participant_signal.go:180 could not send message to participant {"room": "ashan-load-test-video-only", "roomID": "RM_Tuer7m42Ef6x", "participant": "npeyp_783", "pID": "PA_eczZgeQ2SuQu", "remote": false, "message": "*livekit.SignalResponse_Update", "error": "no response sink"}
2022-12-03T22:39:17.491Z | 2022-12-03T22:39:17.491Z ERROR livekit rtc/room.go:865 could not send update to participant {"room": "ashan-load-test-video-only", "roomID": "RM_Tuer7m42Ef6x", "participant": "npeyp_783", "pID": "PA_eczZgeQ2SuQu", "error": "no response sink"}
2022-12-03T22:39:17.491Z | <http://github.com/livekit/livekit-server/pkg/rtc.(*Room).sendParticipantUpdates|github.com/livekit/livekit-server/pkg/rtc.(*Room).sendParticipantUpdates>
2022-12-03T22:39:17.491Z |  /workspace/pkg/rtc/room.go:865
2022-12-03T22:39:17.491Z | <http://github.com/livekit/livekit-server/pkg/rtc.(*Room).subscriberBroadcastWorker|github.com/livekit/livekit-server/pkg/rtc.(*Room).subscriberBroadcastWorker>
2022-12-03T22:39:17.491Z |  /workspace/pkg/rtc/room.go:960
2022-12-03T22:39:17.492Z | 2022-12-03T22:39:17.492Z INFO livekit rtc/participant_signal.go:180 could not send message to participant {"room": "ashan-load-test-video-only", "roomID": "RM_Tuer7m42Ef6x", "participant": "npeyp_779", "pID": "PA_nXHvXiqFbTDk", "remote": false, "message": "*livekit.SignalResponse_Update", "error": "no response sink"}
2022-12-03T22:39:17.492Z | 2022-12-03T22:39:17.492Z ERROR livekit rtc/room.go:865 could not send update to participant {"room": "ashan-load-test-video-only", "roomID": "RM_Tuer7m42Ef6x", "participant": "npeyp_779", "pID": "PA_nXHvXiqFbTDk", "error": "no response sink"}
2022-12-03T22:39:17.492Z | <http://github.com/livekit/livekit-server/pkg/rtc.(*Room).sendParticipantUpdates|github.com/livekit/livekit-server/pkg/rtc.(*Room).sendParticipantUpdates>
2022-12-03T22:39:17.492Z |  /workspace/pkg/rtc/room.go:865
2022-12-03T22:39:17.492Z | <http://github.com/livekit/livekit-server/pkg/rtc.(*Room).subscriberBroadcastWorker|github.com/livekit/livekit-server/pkg/rtc.(*Room).subscriberBroadcastWorker>
2022-12-03T22:39:17.492Z |  /workspace/pkg/rtc/room.go:960
2022-12-03T22:39:17.494Z | 2022-12-03T22:39:17.493Z INFO livekit rtc/participant_signal.go:180 could not send message to participant {"room": "ashan-load-test-video-only", "roomID": "RM_Tuer7m42Ef6x", "participant": "npeyp_792", "pID": "PA_5zkhyk6rmM6N", "remote": false, "message": "*livekit.SignalResponse_Update", "error": "no response sink"}
2022-12-03T22:39:17.494Z | 2022-12-03T22:39:17.494Z ERROR livekit rtc/room.go:865 could not send update to participant {"room": "ashan-load-test-video-only", "roomID": "RM_Tuer7m42Ef6x", "participant": "npeyp_792", "pID": "PA_5zkhyk6rmM6N", "error": "no response sink"}
2022-12-03T22:39:17.494Z | <http://github.com/livekit/livekit-server/pkg/rtc.(*Room).sendParticipantUpdates|github.com/livekit/livekit-server/pkg/rtc.(*Room).sendParticipantUpdates>
2022-12-03T22:39:17.494Z |  /workspace/pkg/rtc/room.go:865
2022-12-03T22:39:17.494Z | <http://github.com/livekit/livekit-server/pkg/rtc.(*Room).subscriberBroadcastWorker|github.com/livekit/livekit-server/pkg/rtc.(*Room).subscriberBroadcastWorker>
2022-12-03T22:39:17.494Z |  /workspace/pkg/rtc/room.go:960
2022-12-03T22:39:17.601Z | 2022-12-03T22:39:17.601Z INFO livekit rtc/transport.go:472 ice connection state change {"room": "ashan-load-test-video-only", "roomID": "RM_Tuer7m42Ef6x", "participant": "npeyp_649", "pID": "PA_57i6WLKgQhfd", "remote": false, "transport": "SUBSCRIBER", "state": "connected"}
2022-12-03T22:39:17.602Z | 2022-12-03T22:39:17.602Z INFO livekit rtc/transport.go:479 selected ICE candidate pair {"room": "ashan-load-test-video-only", "roomID": "RM_Tuer7m42Ef6x", "participant": "npeyp_649", "pID": "PA_57i6WLKgQhfd", "remote": false, "transport": "SUBSCRIBER", "pair": "(local) udp4 host 10.244.0.169:56469 &lt;-&gt; (remote) udp4 relay 10.244.1.70:61367 related 10.128.0.2:41296"}
2022-12-03T22:39:17.727Z | 2022-12-03T22:39:17.727Z INFO livekit connectionquality/connectionstats.go:250 low connection quality {"room": "ashan-load-test-video-only", "roomID": "RM_Tuer7m42Ef6x", "participant": "npeyp_787", "pID": "PA_GvzwXEXvwRR7", "remote": false, "trackID": "TR_AMF3tFSiRgo7zs", "relayed": false, "score": 1.1520737, "params": {"Duration":4999597885,"Codec":"red","PacketsExpected":227,"PacketsLost":201,"Bytes":13770,"Frames":227,"FrameRateExpected":0,"Jitter":167520.83333333334,"Rtt":720,"DtxEnabled":false,"Width":0,"Height":0,"IsReducedQuality":false}}
2022-12-03T22:39:17.824Z | 2022-12-03T22:39:17.823Z INFO livekit rtc/participant.go:419 received answer {"room": "ashan-load-test-video-only", "roomID": "RM_Tuer7m42Ef6x", "participant": "npeyp_531", "pID": "PA_CKkL6vaZPTt2", "remote": false, "transport": "SUBSCRIBER"}
2022-12-03T22:39:17.825Z | 2022-12-03T22:39:17.825Z INFO livekit rtc/transport.go:472 ice connection state change {"room": "ashan-load-test-video-only", "roomID": "RM_Tuer7m42Ef6x", "participant": "npeyp_531", "pID": "PA_CKkL6vaZPTt2", "remote": false, "transport": "SUBSCRIBER", "state": "checking"}
2022-12-03T22:39:17.825Z | 2022-12-03T22:39:17.825Z INFO livekit rtc/participant.go:1143 sending offer {"room": "ashan-load-test-video-only", "roomID": "RM_Tuer7m42Ef6x", "participant": "npeyp_531", "pID": "PA_CKkL6vaZPTt2", "remote": false, "transport": "SUBSCRIBER"}
2022-12-03T22:39:17.849Z | 2022-12-03T22:39:17.848Z ERROR livekit service/rtcservice.go:286 error reading from websocket {"room": "ashan-load-test-video-only", "roomID": "RM_Tuer7m42Ef6x", "participant": "npeyp_776", "remote": false, "error": "websocket: close 1002 (protocol error): RSV1 set, RSV2 set, RSV3 set, bad opcode 14"}
2022-12-03T22:39:17.849Z | <http://github.com/livekit/livekit-server/pkg/service.(*RTCService).ServeHTTP|github.com/livekit/livekit-server/pkg/service.(*RTCService).ServeHTTP>
2022-12-03T22:39:17.849Z |  /workspace/pkg/service/rtcservice.go:286
2022-12-03T22:39:17.849Z | net/http.(*ServeMux).ServeHTTP
2022-12-03T22:39:17.849Z |  /usr/local/go/src/net/http/server.go:2487
2022-12-03T22:39:17.849Z | <http://github.com/urfave/negroni.Wrap.func1|github.com/urfave/negroni.Wrap.func1>
2022-12-03T22:39:17.849Z |  /go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:46
2022-12-03T22:39:17.849Z | <http://github.com/urfave/negroni.HandlerFunc.ServeHTTP|github.com/urfave/negroni.HandlerFunc.ServeHTTP>
2022-12-03T22:39:17.849Z |  /go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:29
2022-12-03T22:39:17.849Z | <http://github.com/urfave/negroni.middleware.ServeHTTP|github.com/urfave/negroni.middleware.ServeHTTP>
2022-12-03T22:39:17.849Z |  /go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:38
2022-12-03T22:39:17.849Z | net/http.HandlerFunc.ServeHTTP
2022-12-03T22:39:17.849Z |  /usr/local/go/src/net/http/server.go:2109
2022-12-03T22:39:17.849Z | <http://github.com/livekit/livekit-server/pkg/service.(*APIKeyAuthMiddleware).ServeHTTP|github.com/livekit/livekit-server/pkg/service.(*APIKeyAuthMiddleware).ServeHTTP>
2022-12-03T22:39:17.849Z |  /workspace/pkg/service/auth.go:84
2022-12-03T22:39:17.849Z | <http://github.com/urfave/negroni.middleware.ServeHTTP|github.com/urfave/negroni.middleware.ServeHTTP>
2022-12-03T22:39:17.849Z |  /go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:38
2022-12-03T22:39:17.849Z | <http://github.com/rs/cors.(*Cors).ServeHTTP|github.com/rs/cors.(*Cors).ServeHTTP>
2022-12-03T22:39:17.849Z |  /go/pkg/mod/github.com/rs/cors@v1.8.2/cors.go:266
2022-12-03T22:39:17.849Z | <http://github.com/urfave/negroni.middleware.ServeHTTP|github.com/urfave/negroni.middleware.ServeHTTP>
2022-12-03T22:39:17.849Z |  /go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:38
2022-12-03T22:39:17.849Z | <http://github.com/urfave/negroni.(*Recovery).ServeHTTP|github.com/urfave/negroni.(*Recovery).ServeHTTP>
2022-12-03T22:39:17.849Z |  /go/pkg/mod/github.com/urfave/negroni@v1.0.0/recovery.go:193
2022-12-03T22:39:17.849Z | <http://github.com/urfave/negroni.middleware.ServeHTTP|github.com/urfave/negroni.middleware.ServeHTTP>
2022-12-03T22:39:17.849Z |  /go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:38
2022-12-03T22:39:17.849Z | <http://github.com/urfave/negroni.(*Negroni).ServeHTTP|github.com/urfave/negroni.(*Negroni).ServeHTTP>
2022-12-03T22:39:17.849Z |  /go/pkg/mod/github.com/urfave/negroni@v1.0.0/negroni.go:96
2022-12-03T22:39:17.849Z | net/http.serverHandler.ServeHTTP
2022-12-03T22:39:17.849Z |  /usr/local/go/src/net/http/server.go:2947
2022-12-03T22:39:17.849Z | net/http.(*conn).serve
2022-12-03T22:39:17.849Z |  /usr/local/go/src/net/http/server.go:1991
2022-12-03T22:39:17.849Z | 2022-12-03T22:39:17.848Z INFO livekit service/rtcservice.go:220 finishing WS connection {"room": "ashan-load-test-video-only", "roomID": "RM_Tuer7m42Ef6x", "participant": "npeyp_776", "remote": false, "connID": "CO_uaCpMPSqMFi7"}
2022-12-03T22:39:17.850Z | 2022-12-03T22:39:17.850Z INFO livekit service/roommanager.go:443 RTC session finishing {"participant": "npeyp_776", "pID": "PA_KxoJvHdBnXcM", "room": "ashan-load-test-video-only", "roomID": "RM_Tuer7m42Ef6x"}```
Any insight feedback to to make successful test for 1000, 2000 and 5000 users per room is highly appreciated.

Team,  upon your experience,  *can you kindly mention what is rough HW spec that* *we need to include for 1000 , 2000 and 5000 users (listeners with 1 x audio-publisher) in single room respectively (if possible).*  This will help us to clearly set required HW resources for the given benchmarks.

This appears to be an error in the load tester with the underlying websocket library as the number of participants scale.

As a workaround, can you try to start multiple load testers?

for example, each subscriber node to run 600 subscribers, and starting 5 of them.

the CLI load tester is at least 2x less efficient as the server. So ensure you are running it on an instance with sufficient resources.

Thank you for the feedback <@U01KZ61GV5Z>. Really appreciate the insights. 
Per your suggestion what is the recommended HW.  to put 600 subscribes, is
 x4 core, 8 GB or
 x2 core , 4GB  is enough.

In our experience, when we hit 700-1000, in x4core  , 16 GB server, we only observe *25-30% cpu usage and 1GB Ram usage and get above load tester servers error in between (700-1000 users  loading)*

the number of users is highly use-case dependent. You'll know better than I do after load testing it to simulate your specific scenario :slightly_smiling_face:

Noted with thanks, will share the insights so others will help, after testing.

Just quick feedback,  According to <https://docs.livekit.io/oss/benchmark/#benchmarks> server spec `All benchmarks were ran with the server running on a 16-core, compute optimized instance on Google Cloud. ( c2-standard-16)`  , We quickly setup x16 core, 32GB dedicated server and applied <https://github.com/livekit/livekit-cli#configuring-system-settings> settings (issue a reboot after making the changes permeant to be make sure :sweat_smile:) as load testing server and still unable to push the 650 -750 (subs with 1x audio-publisher) margin and we get following errors;

CLI:
```turnc ERROR: 2022/12/04 23:50:13 fail to refresh permissions: all retransmissions failed for ZWwnvGVPypgHHxH9```
Livekit server (on and after cli errors) :

```
.....
2022-12-04T18:22:54.874Z | <http://github.com/pion/turn/v2/internal/allocation.(*Allocation).packetHandler|github.com/pion/turn/v2/internal/allocation.(*Allocation).packetHandler>
2022-12-04T18:22:54.874Z |  /go/pkg/mod/github.com/pion/turn/v2@v2.0.8/internal/allocation/allocation.go:267
2022-12-04T18:22:54.874Z | 2022-12-04T18:22:54.874Z ERROR livekit.turn allocation/allocation.go:267 Failed to send ChannelData from allocation 178.128.117.59:58488 write tcp4 10.130.0.12:3478-&gt;10.130.0.5:50364: write: broken pipe
2022-12-04T18:22:54.874Z | <http://github.com/pion/turn/v2/internal/allocation.(*Allocation).packetHandler|github.com/pion/turn/v2/internal/allocation.(*Allocation).packetHandler>
2022-12-04T18:22:54.874Z |  /go/pkg/mod/github.com/pion/turn/v2@v2.0.8/internal/allocation/allocation.go:267
2022-12-04T18:22:54.874Z | 2022-12-04T18:22:54.874Z ERROR livekit.turn allocation/allocation.go:267 Failed to send ChannelData from allocation &lt;PUBLIC IP&gt;:58488 write tcp4 10.130.0.12:3478-&gt;10.130.0.5:50364: write: broken pipe
2022-12-04T18:22:54.874Z | <http://github.com/pion/turn/v2/internal/allocation.(*Allocation).packetHandler|github.com/pion/turn/v2/internal/allocation.(*Allocation).packetHandler>
2022-12-04T18:22:54.874Z |  /go/pkg/mod/github.com/pion/turn/v2@v2.0.8/internal/allocation/allocation.go:267
2022-12-04T18:22:54.874Z | 2022-12-04T18:22:54.874Z ERROR livekit.turn v2@v2.0.8/server.go:184 error when handling datagram: failed to handle CreatePermission-request from 10.130.0.5:50364: write tcp4 10.130.0.12:3478-&gt;10.130.0.5:50364: write: broken pipe
2022-12-04T18:22:54.874Z | <http://github.com/pion/turn/v2.(*Server).readLoop|github.com/pion/turn/v2.(*Server).readLoop>
2022-12-04T18:22:54.874Z |  /go/pkg/mod/github.com/pion/turn/v2@v2.0.8/server.go:184
2022-12-04T18:22:54.874Z | 2022-12-04T18:22:54.874Z ERROR livekit.turn v2@v2.0.8/server.go:184 error when handling datagram: failed to handle CreatePermission-request from 10.130.0.5:50364: write tcp4 10.130.0.12:3478-&gt;10.130.0.5:50364: write: broken pipe
2022-12-04T18:22:54.874Z | <http://github.com/pion/turn/v2.(*Server).readLoop|github.com/pion/turn/v2.(*Server).readLoop>
2022-12-04T18:22:54.874Z |  /go/pkg/mod/github.com/pion/turn/v2@v2.0.8/server.go:184
2022-12-04T18:22:54.878Z | 2022-12-04T18:22:54.877Z ERROR livekit.turn v2@v2.0.8/server.go:184 error when handling datagram: failed to handle Refresh-request from 10.130.0.5:50364: write tcp4 10.130.0.12:3478-&gt;10.130.0.5:50364: write: broken pipe
2022-12-04T18:22:54.878Z | <http://github.com/pion/turn/v2.(*Server).readLoop|github.com/pion/turn/v2.(*Server).readLoop>
2022-12-04T18:22:54.878Z |  /go/pkg/mod/github.com/pion/turn/v2@v2.0.8/server.go:184
2022-12-04T18:22:55.306Z | 2022-12-04T18:22:55.305Z INFO livekit rtc/transport.go:472 ice connection state change {"room": "ashan-load-test-video-only", "roomID": "RM_zUFXdSKXvmv3", "participant": "rtlya_948", "pID": "PA_KfzHMBhVQuSZ", "remote": false, "transport": "SUBSCRIBER", "state": "connected"}
2022-12-04T18:22:55.306Z | 2022-12-04T18:22:55.305Z INFO livekit rtc/transport.go:479 selected ICE candidate pair {"room": "ashan-load-test-video-only", "roomID": "RM_zUFXdSKXvmv3", "participant": "rtlya_948", "pID": "PA_KfzHMBhVQuSZ", "remote": false, "transport": "SUBSCRIBER", "pair": "(local) udp4 host &lt;PUBLIC IP&gt;:53068 &lt;-&gt; (remote) udp4 host 85.14.245.18:40150"}
...```
Just out of curiosity, per doc (<https://docs.livekit.io/oss/benchmark/#benchmarks>) you guys ran the benchmark from `16-core, compute optimized instance on Google Cloud. ( c2-standard-16)`
, Are there any additional OS / Kernal level changes that you guys used to trigger 3000 subs (with 1 x audio-subs /  set benchmarks under <https://docs.livekit.io/oss/benchmark/#benchmarks>) apart from <https://github.com/livekit/livekit-cli#configuring-system-settings> changes or you guys also use multiple load testing servers to simulate the shared benchmarks ?

Would like do any server performance tuneup to load tester server (x16 core, 32GB dedicated server) if that worth the effort over creating multiple servers (will try this in parallela and share revert feedback if there any progress).  Can you kindly share your revert feedback on this <@U01KZ61GV5Z>. This will help many who try do the load testing with livekit.

the errors you report are from the TURN client &amp; server. Have you used the <https://livekit.io/connection-test|connection tester> to ensure the server is set up correctly? In general TURN errors can be ignored during the load test.

we have not needed to apply any other changes to the OS in order to achieve those stats.

Capture_2022-12-04-009.png

Capture_2022-12-05-002.png

Capture_2022-12-05-001.png

Capture_2022-12-04-007.png

Yes, the <https://livekit.io/connection-test|connection tester> passed successfully for livekit-server.

Just to make sure, for `load tester you use a single instance of 16-core, compute optimized instance on Google Cloud. ( c2-standard-16)`, correct?

you'd want to use a larger machine, or multiple VMs to run the load testers.

as mentioned, if you are trying to stress a 16 core livekit-server instance to its max capacity.. you'll need a machine with 2x the amount of computing and networking capacity to run the tester. -- the tester is much less efficient than the server itself

<@U01KZ61GV5Z> just to clarify and sumup;

Our setup as follows;
• *k8s cluster* [just quick test cluster, *which runs livekit-servers*] have -  *x8 core , 16 GB* (x2 worker nodes each having x4 core, 8GB )
• *load tester* *Initiator* server - *x16 core, 32GB* server
Can you kindly confirm the below so i can clarify (i think i may have jumbled ) ;

1. In <https://docs.livekit.io/oss/benchmark/#benchmarks> you guys mentioned, `16-core, compute optimized instance on Google Cloud. ( c2-standard-16)`  . Is this the livekit-server itself HW spec or load tester initiator server HW spec ?
2. If option 1 answer is livekit-server itself HW spec, then how many load tester initiator server(s) (according to your previous feedback, it should be x2 times the livekit-servers resources ) you guys used to generate the benchmarked traffic and can you kindly share the rough spec of one of load tester initiator server

<@U01KZ61GV5Z> quick update;

- Single load initiator test server
	- load test initiator - x16 core, 32 GB RAM  /
	- Livekit servers - K8s - x2 servers (each server set with x8 core , 16 GB)
	
	- Tests
		- livekit-cli load-test --room ashan-load-test-video-only --video-publishers 5 --subscribers 3000
			- when there are 500 users online we get socket error in cli testing server
		- livekit-cli load-test --room ashan-load-test-video-only --video-publishers 1 --subscribers
			-  when there are 1332 users online we get socket error in cli testing server
		- livekit-cli load-test --room ashan-load-test-video-only --audio-publishers 1 --subscribers 2000
			- Success - all 1000 users were online successfully
		- livekit-cli load-test --room ashan-load-test-video-only --audio-publishers 1 --subscribers 4000 --num-per-second 10
			- Success - all 2000 users were online successfully (~1950 seems be the margin for load test single server limit)
			
- Multiple load initiator test servers
	- *load test initiator servers* -
		*- x1 server - 16 core, 32 GB RAM* 
		*- x3 servers - x8 core, 32 GB RAM*
	- Livekit servers - K8s - x2 servers (each server set with x8 core , 16 GB)
	
	- Tests
		- Single test
			- livekit-cli load-test --room ashan-load-test-video-only --audio-publishers 1 --subscribers 2000 --num-per-second 30
			- Success - all 1000 users were online successfully
		
		- *group test*
			- livekit-cli load-test --room ashan-load-test-video-only  --audio-publishers 1 --subscribers 3000 --num-per-second 30
			- livekit-cli load-test --room ashan-load-test-video-only  --subscribers 2500 --num-per-second 30
				
			- *After ~2200-2500 (users online, ~5000 subs)* the in *load initiator test server* &gt; cli get an error of *panic: runtime error: slice bounds out of range [:8192] with capacity 4096*	
```subscribed to track iqugl_2862 TR_VCu6NYnh39scmF video 1/2
subscribed to track iqugl_2862 TR_AMU7RmTvPyt3aG audio 2/2
subscribed to track iqugl_2846 TR_VCu6NYnh39scmF video 1/2
subscribed to track iqugl_2827 TR_AMU7RmTvPyt3aG audio 1/2
panic: runtime error: slice bounds out of range [:4305] with capacity 4096

goroutine 2783570 [running]:
bufio.(*Reader).Read(0xc22fcd3200, {0xc23dce4000?, 0xfbd?, 0x0?})
        /opt/hostedtoolcache/go/1.18.7/x64/src/bufio/bufio.go:249 +0x2ed
io.(*LimitedReader).Read(0xc23dce2048, {0xc23dce4000?, 0x0?, 0x0?})
        /opt/hostedtoolcache/go/1.18.7/x64/src/io/io.go:476 +0x45
io.discard.ReadFrom({}, {0x1be8e00, 0xc23dce2048})
        /opt/hostedtoolcache/go/1.18.7/x64/src/io/io.go:610 +0x72
io.copyBuffer({0x1bea080, 0x2169b80}, {0x1be8e00, 0xc23dce2048}, {0x0, 0x0, 0x0})
        /opt/hostedtoolcache/go/1.18.7/x64/src/io/io.go:412 +0x14b
io.Copy(...)
        /opt/hostedtoolcache/go/1.18.7/x64/src/io/io.go:385
io.CopyN({0x1bea080, 0x2169b80}, {0x1be5560?, 0xc22fcd3200}, 0xfbd)
        /opt/hostedtoolcache/go/1.18.7/x64/src/io/io.go:361 +0x9a
<http://github.com/gorilla/websocket.(*Conn).advanceFrame(0xc22897b080)|github.com/gorilla/websocket.(*Conn).advanceFrame(0xc22897b080)>
        /home/runner/go/pkg/mod/github.com/gorilla/websocket@v1.5.0/conn.go:798 +0x5e
<http://github.com/gorilla/websocket.(*Conn).NextReader(0xc22897b080)|github.com/gorilla/websocket.(*Conn).NextReader(0xc22897b080)>
        /home/runner/go/pkg/mod/github.com/gorilla/websocket@v1.5.0/conn.go:1009 +0xc5
<http://github.com/gorilla/websocket.(*Conn).ReadMessage(0xb1865e?)|github.com/gorilla/websocket.(*Conn).ReadMessage(0xb1865e?)>
        /home/runner/go/pkg/mod/github.com/gorilla/websocket@v1.5.0/conn.go:1093 +0x19
<http://github.com/livekit/server-sdk-go.(*SignalClient).readResponse(0xc0ff672960?)|github.com/livekit/server-sdk-go.(*SignalClient).readResponse(0xc0ff672960?)>
        /home/runner/go/pkg/mod/github.com/livekit/server-sdk-go@v1.0.5/signalclient.go:210 +0x95
<http://github.com/livekit/server-sdk-go.(*SignalClient).readWorker(0xc0ff672960)|github.com/livekit/server-sdk-go.(*SignalClient).readWorker(0xc0ff672960)>
        /home/runner/go/pkg/mod/github.com/livekit/server-sdk-go@v1.0.5/signalclient.go:295 +0xd7
created by <http://github.com/livekit/server-sdk-go.(*SignalClient).Start|github.com/livekit/server-sdk-go.(*SignalClient).Start>
        /home/runner/go/pkg/mod/github.com/livekit/server-sdk-go@v1.0.5/signalclient.go:54 +0x6a                      ```
*Any idea how we can push subs for 10k-20k subs (5k-10k users ) for the same meeting room* without above `panic: runtime error: slice bounds out of range [:8192] with capacity 4096`  error ? Kindly mark all  load initiator servers have applied the <https://github.com/livekit/livekit-cli#configuring-system-settings> performance tuneups.

In order to isolate the issue we had apply the following changes and result are same;
• Increase k8s cluster HW resources (from Horizontal and Vertical) -&gt; ~ max reach is *~2200-2500 (users online, ~5000 subs) for given room (1x audio-publisher)*
• *Distribute load testing among multiple load initiator servers* -&gt; *max reach is ~2200-2500 (users online, ~5000 subs) for given room (1x audio-publisher)* 
<@U01KZ61GV5Z> highly appreciate your valuable feedback on this.

Ashan, I mentioned <https://livekit-users.slack.com/archives/C01KZ61HZRR/p1670136095618269?thread_ts=1669972332.348759&amp;cid=C01KZ61HZRR|this above>, but I think the workaround is to reduce the concurrency for each load tester. Try ~600 testers per process.. and starting multiple processes

Thank you for the feedback <@U01KZ61GV5Z>, will try 600 subs per processor and confirm the stats.

Capture_2022-12-05-008.png

Capture_2022-12-06-006.png

Capture_2022-12-05-007.png

Capture_2022-12-06-004.png

Capture_2022-12-06-005.png

<@U01KZ61GV5Z> as mentioned above, tried distributing 600 subs per processor in multiple servers. In a single multiple 600 subs processors are running

- load test initiator servers -
    - x1 server - 16 core, 32 GB RAM
    - x3 servers - x8 core, 32 GB RAM


- group test 2
	- livekit-cli load-test --room ashan-load-test-video-only  --audio-publishers 1 --subscribers 600 --num-per-second 50
	- livekit-cli load-test --room ashan-load-test-video-only  --subscribers 600 --num-per-second 50 (multiple processors in same load testing server)
		- x2 servers - 5 processors in each server
			- After ~2400-2500 the cli get an error of panic: runtime error: slice bounds out of range [:8192] with capacity 4096
		- *x4 servers - 4 processors in each server*
			- *After ~4433* *users (8k-9k subs)* the cli get an error of panic: runtime error: slice bounds out of range [:8192] with capacity 4096

I will do further testing and confirm (see if we can handle 4.4k+ with 600 subs x3 processors in each server and try to expand the cluster worker nodes and see if that affects) . Is there any way optimization to do to squeeze more subs?

Attached cluster HW stats as well for your reference.

this seems to be an issue with the load tester.. we'll file an issue under livekit-cli to track. the only advise I have now is to use less concurrency in each of the CLI processes.. and starting more of them

Noted with Thanks, will wait for livekit-cli issue update

Hey Charlie,

Most of the latency will be coming from the network between your user and the server. If the client and server are running in the same datacenter, then end to end latency is under 100ms.

Practically, low-hundreds of ms is the right ballpark.

<@U01KZ61GV5Z> thanks for a straight response; was expecting more 'it depends' than that

when you say 'same datacenter' - you mean it would be a relatively local end-to-end connection and not going through remote servers correct?

yeah, that's just a hypothetical scenario. livekit itself adds &lt;1ms of latency during forwarding. so it really depends on how quickly the publisher could encode and send those bits to us (and then for us to forward to subscriber, then for it to render)

100ms includes the entire process (encode, publish + forward + decode)

out of curiosity, what question are you trying to answer with these numbers? As far as applications go, low-hundreds-of-ms is good enough for voice conversation, not good enough for music performance (which is constrained by digital audio &amp; the speed of light)

<@U02MC5V3C2H> the application is more for the former than the latter

I'm curious because in our first implementation with LiveKit, we're hitting serious lag, but it's obviously something that needs troubleshooting. One of our teammembers will post about it this weekend for assistance.

Also curious because similar solutions to ours in the market - I have noticed ~2secs of lag with remote connections - I'm sure they aren't using LiveKit though.

<@U01KZ61GV5Z> understood - so it sounds like in real world applications your team has seen ~100ms for local connections and likely &lt;1sec for remote connections through AWS / Azure / Google cloud? (I understand it depends on where the datacenter is, what the connection of the client looks like, etc.) Is that fair to say? Just thinking of realistic targets here.

if this is a broadcast streaming use-case and other solutions have ~2secs of lag, they might be using HLS rather than WebRTC, which introduces some unavoidable lag

also if using our hosted Cloud service is an option for you, that will also improve the network connection as interdatacenter data transmission happens through backbone connections rather than public internet connections

gotcha - thanks <@U02MC5V3C2H> - appreciate the responses

Hey LiveKit team -- im working with Charlie on a project where we would like to use LiveKit as the streaming backend for cameras. mostly following the publishing examples <https://github.com/livekit/livekit-cli#publishing-to-a-room|here>.

we are using `gstreamer` and i got a working proof-of-concept with the LiveKit CLI -- however as Charlie said, the audio is super delayed when joining the same room as the camera via the <https://example.livekit.io/|LiveKit React Example> (by super delayed i mean ~10 seconds). the video in comparison has minimal latency.

here is the gstreamer pipeline i am running on the camera:
```&lt;camera feed plugin&gt; ! video/x-h264,width=1920,height=1080,stream-format=byte-stream ! queue ! tcpserversink host=0.0.0.0 port=16104 autoaudiosrc ! queue ! audioresample ! audioconvert ! opusenc bitrate=24000 ! audio/x-opus ! oggmux max-page-delay=20000 max-delay=20000 ! queue ! tcpserversink host=0.0.0.0 port=16105```
here is the LiveKit CLI command we are running:
```livekit-cli join-room --room my-room --identity bot --publish <h264://camera.local:16104> --publish <opus://camera.local:16105> --api-key devkey --api-secret secret```
as mentioned, the camera can join the room and the video has minimal latency. audio is obviously not synced up and super delayed.

i know it has something to do with how LiveKit is handling the audio, because when i connect to the audio stream with a `gstreamer` tcpclientsrc pipeline, there is almost no delay

e.g.
```./gst-launch-1.0 tcpclientsrc host=10.0.0.34 port=16105 ! oggdemux ! opusparse ! opusdec ! audioconvert ! audioresample ! autoaudiosink```
this is all experimentation on a local setup, so network delay is not a factor. im wondering if maybe the opus decoder is optimized for particular settings? it took a lot of trial and error to even figure out i had to adjust the page delay settings to get it to work. any feedback would be appreciated and let me know if i can provide any more info.

I think what's happening here is the client is buffering the audio for playback. As mentioned, livekit forwards everything received immediately without buffering.

as to why the buffering is taking place.. my best guess is it has to do with how it's encoded.. As an example, publishing a pre-encoded file (instead of a live source) <https://github.com/livekit/server-sdk-go#h264--opus|like this> produce no lag at all.

thanks for the insight -- i figured it might have something to do with the audio encoding esp since im not super familiar there. ill continue messing with the audio settings from gstreamer

thank you for your comment David, was just what i needed to figure it out!

```gst-launch-1.0 &lt;camera feed plugin&gt; ! video/x-h264,width=1920,height=1080,stream-format=byte-stream ! tcpserversink host=0.0.0.0 port=16104 autoaudiosrc ! audioresample ! audioconvert ! opusenc bitrate=24000 ! audio/x-opus ! oggmux max-page-delay=0 max-delay=0 ! tcpserversink host=0.0.0.0 port=16105```
just had to remove all the `queue` elements in the pipeline, which was the primary source of the buffering. also set the oggmux delay parameters to 0 as well

ok so it did not quite fix it, though it may have marginally helped  -- i guess i started the `gstreamer` pipeline and LiveKit CLI around the same time when i tested without the queue elements, so it appeared those changes fixed it.

regardless it led me to figure out that *the audio delay is based on how long you wait to hook the LiveKit CLI into the stream*. so if i start the pipeline, and then run LiveKit CLI 10 seconds later, there will be around a 10 second delay in pulling in the audio in the LiveKit room, and it will be 10 seconds behind.

seems like a few folks have run into this before with `gstreamer` -- i found this <https://gitlab.freedesktop.org/gstreamer/gst-plugins-base/-/issues/249|unanswered issue> that describes a similar problem and a StackOverflow question that described it as well. tried some settings around synchronization and buffering etc which would have supposedly worked and nothing changed. might have to come up with another way to hook into LiveKit from `gstreamer`

ah interesting.. that would be an issue with GStreamer + live src.. what you'll need is something like a leaky bucket for a live source

added a queue in front of the audio tcpserversink that leaks old buffers to no avail. need to try some other debugging approaches to see if it's `gstreamer` or something with the OggReader in Pion. `gstreamer` itself with `tcpclientsrc` always picks up the live audio from the feed fwiw

im wondering -- could the Go SDK read a UDP stream in theory? we are going to have to move away from the CLI and use the SDK anyway. if it is an issue with tcpserversink then it might be a viable option to use udpsink instead

yeah, it should be possible to use a UDP stream. I haven't tried it but seems possible with a small modification.

So we figured out the audio latency thing with tcpserversink. However we realized that true audio/video synchronization is infeasible with two different sockets which is the approach the LiveKit CLI takes, so I am assuming that most folks using gstreamer with LiveKit have a more custom approach that involves streaming a container format directly instead of two different sockets. This is not my team’s expertise by any means, so I’m curious if there is a more production-ready approach or example that keeps audio/video in sync streaming from gstreamer.

are you setting framerate with video?  the CLI does allow you to pass in a video framerate as well.

usually sync issues happen because video and audio are being played back at different speeds. I don't believe the separate sockets is the root cause of sync problems.

can you share the pipeline that you are using as well as livekit-cli flags?

Interesting ok. We are not passing a frame rate

Here is the gstreamer pipeline:

```HOST=0.0.0.0 gst-launch-1.0 olcamerasrc ! video/x-h264,width=1920,height=1080,stream-format=byte-stream ! queue max-size-bytes=0 max-size-buffers=0 ! tcpserversink host=$HOST port=16104 recover-policy=keyframe sync-method=latest-keyframe autoaudiosrc ! queue  max-size-bytes=0 max-size-buffers=0 ! audioconvert ! opusenc bitrate=48000 ! audio/x-opus ! oggmux max-page-delay=0 max-delay=0 ! tcpserversink host=$HOST port=16105 recover-policy=keyframe sync-method=latest-keyframe```
LiveKit CLI command:

```livekit-cli join-room --room my-room --identity bot --publish <h264://oclea.local:16104> --publish <opus://oclea.local:16105> --api-key devkey --api-secret secret```

what fps is your source publishing at? is it possible to get the framerate there?

Most likely — I’ll have to check the docs or ask our camera provider

Ok picking up where Mike left off 3 months later :disappointed: . I have added --fps but I am definitely experiencing the delay. Open to any and all ideas. Planning on switching to the Rust SDK once it supports publishing.

Recreating this delay is really easy using the following:

Start Livekit Server
```livekit-server --dev```
Create Token and join room using example app: <https://example.livekit.io/#/>
```livekit-cli create-token \
--api-key devkey \
--api-secret secret \
--join --room room1 \
--identity user1 \
--valid-for 24h```
Start Stream
```gst-launch-1.0 autoaudiosrc ! audioresample quality=10 ! audio/x-raw ! queue max-size-bytes=0 max-size-buffers=0 ! opusenc bitrate=24000 ! oggmux max-page-delay=0 max-delay=0 ! tcpserversink host=localhost port=16107 recover-policy=keyframe sync-method=latest-keyframe```
Publish Stream
```livekit-cli join-room \
    --fps 30 \
    --room "rooms1" \
    --identity "stream1" \
    --publish <opus://localhost:16107> \
    --api-key devkey \
    --api-secret secret```
Depending on the time it takes you to Start/Publish your audio will be delayed. You can exit and re-join the room and you will see the audio delay grows exponentially

Follow up here, now I am wondering if the issue could be related to how livekit reads the opus streams.

Setup audio src
```gst-launch-1.0 autoaudiosrc \
    ! audioresample \
    ! audioconvert \
    ! opusenc hard-resync=true audio-type=2048 \
    ! oggmux max-page-delay=0 max-delay=0 \
    ! tcpserversink host=127.0.0.1 port=16107 sync=0 async=0```
Join with gstreamer
```gst-launch-1.0 tcpclientsrc host=127.0.0.1 port=16107 \
    ! oggdemux \
    ! opusdec \
    ! audioconvert \
    ! audioresample \
    ! autoaudiosink```
notice there is no delay regardless of how late you join with `tcpclientsrc`

Ha you are going through the same exact testing and thought process I went through. I have said before that I am almost positive it has to do with the OPUS decoding, might even be in that doc I put together. I have also tested the gstreamer client command with no issue in latency etc

I have been trying to get udpsink to work as well its kicking my ass :disappointed:

That would confirm if it was the OPUS decoding if the same thing happened on UDP. It is almost like the OPUS decoder is looking at something that indicates how long the stream has been running.

Yeh I think you also need to do RTP on top of UDP. It seemed challenging. I know there was another approach mentioned in a GitHub issue where you could potentially do a Unix Domain Socket with shmsink, though our camera’s GStreamer install doesn’t have it on there by default.

I have been testing with macos lately but if we get it working I can likely copy it over to the camera, I will poke around at that. I know when I use the ffpmeg demo for unix from the docs it always starts from the beginning of the file though.

I also still don’t fully understand how video/audio gets synchronized when they are on two different sockets :thinking_face:  

I believe I have resolved the issue here: <https://github.com/livekit/server-sdk-go/issues/192>

Hi Vadim,

The paragraph that you are quoting is the current connection model for LiveKit Open Source. The blog post is describing our Cloud solution, which builds on top of the Open Source SFU. Cloud does not have a single node per room limitation.

Do you have different source code for Cloud?
I mean if I launch some OSS nodes on my server with Redis - will I face this limitation?

as indicated in the post, Cloud is built on top of OSS - it has more code to deal with the orchestration of SFU instances and media relay.

The post does not go in much detail about the mesh architecture. Is this planned to stay closed for now?

<@U04CG6UBGN6> , yeah it'll be closed for now. the mesh architecture is quite more complex to deploy. we may decide to open source it in the future when we are able to simplify it a bit.

Indeed it must be. Thanks for your transparency :sparkles:

Hey Matt!

1. Yes
2. Yes
3. Yes
4. Not yet, but we have plans to and I believe (cc <@U02EU2ZPTQU>) some stuff in a pre-release state?
5. Yes
See: <https://theportal.to/> and <https://hyperfy.io/> (tho the latter isn’t using Unity) for live examples with similar needs to yours:

also cc <@U047ZTF8EHH> who i think implemented his own audio spatialization (and switched from Agora in like 3-4 hours)?

Hey <@U01KZ0SCDKL>, thank you for the very fast response! All of this seems promising. I will check out these platforms!

<@U02EU2ZPTQU> and <@U047ZTF8EHH> if you have any insight to offer here, I'm all ears. :slightly_smiling_face:

Agora's updates to their (in-beta) Unity WebGL plugin are fairly slow (for example, we haven't implemented Agora's full-spatial audio yet since there are breaking bugs in Firefox), so I am looking for any/all reasons I should switch!

Is there any timeline for when the full-spatial audio feature will be added? And will it have HRTF filtering? If I had that info, it would help me build a more compelling case for our team to switch to LiveKit.

hey <@U04DUL22NF5>,
what we have already today is an experimental feature for piping all incoming audio on the js side through WebAudio. You can then set whatever webaudio nodes for all remote audio tracks, thus enabling you to use e.g. the panner node to do do full-spatial audio. It might be that we still have to find a way to expose similar functionality for handling spatial audio on the unity side (cc <@U022KB72NFJ>).
and yes, <https://developer.mozilla.org/en-US/docs/Web/API/PannerNode/panningModel|WebAudio does support HRTF> as an option :slightly_smiling_face:

Very interesting, thank you for the information, <@U02EU2ZPTQU>!

Would all of this be managed by the LiveKit SDK in C#? Our ideal solution would be something turnkey that our team could integrate with C# in Unity (feeding in the local client's audio listener position+rotation and up to 100 remote users' voice positions in 3D space) and then deploy across multiple platforms - for example we'd create a Unity WebGL build for people accessing our platform via web browser, and we'd potentially do another native Unity Android build for users entering our platform via the Meta Quest 2 VR headset (or Vive Focus, Pico, etc).

Currently our platform is browser-only, but we plan to add VR support in the future, so just keeping that in mind.

Is this type of functionality on the roadmap for LiveKit?

Yeah, there’s a unity sdk for all platforms on the roadmap!
AFAIK it’s still not clear whether this will use unity’s built in audio spatialization for the webGL target as there seem to be some restrictions there. But the goal would be to have unity take over that part (so that there’s no livekit-specific bridging to webAudio necessary).

<@U02EU2ZPTQU> do you have an example for me how to implement spatial web audio, so how to use the panner node with livekit?

Hi Dennis,
we don’t have an example for it yet, but that’s definitely something we want to provide!

Is there a link to a third party impl or any other documentation/website you could provide, so I can try to implement it by myself? 

Ah but I need to know how the experimental audio pipe feature works or how I have to use it, right?
And that is currently not documented somewhere.
 

yeah, sorry about that. There are some basic descriptions of the functionality embedded in the source as js-docs, they just don’t show up in the reference yet.
Also to really make good use of it, it makes sense to pass in a custom audio context, that functionality will only be part of the sdk in the next release though. We are still ironing out some issues, but hopeful that a new release will be available tomorrow.

Will try to come up with an example for the panner node some time next week!

That would be reeeaaalllyyyyy cool :heart: 
Thank you so much :pray: 

hello - has anyone found a creative way to handle sound bleed when there is audio chat between players enabled, but one player is consuming live stream audio? We’re running into situations where the sound from the livestream comes through the mic and causes issues with the other players. Obviously if everyone used headphones, no worries - but as we all know, we can’t require that. Thoughts/tips?

follow instruction <https://docs.livekit.io/oss/deployment/vm/>

i run `docker run --rm -it -v$PWD:/output livekit/generate`

image.png

what is the error that's given? did you open up ports as docs indicated?

credential.go

Code to generate time limited credentials for coturn:

In this case, I would recommend leaving `turn_servers` blank on the server side, and instead pass in `rtcConfig.iceServers` on the client side when <https://docs.livekit.io/client-sdk-js/interfaces/RoomConnectOptions.html|connecting>.

If you are passing in `rtcConfig` as a reference to your own object, when you update .iceServers on your end, we'll use those updated values automatically.

Thanks <@U01KZ61GV5Z>.  I was able to pass via client side and I now see those ICE Servers in <chrome://webrtc-internals> and don’t see any authentication errors (so the creds are working).  However, a WebRTC session is not established between my participants (connection state goes: new =&gt; connecting =&gt; closed).  I know this TURN server works because in the same session I can share my screen with a participant using our application’s WebRTC implementation (not yet integrated with LiveKit).  When I compare the webrtc-internals, the `iceServers` list is identical.  In the screen sharing case that works, I see a `addIceCandidate (relay)` message, but in the LiveKit one I only see the `icecandidate (relay)` messages but not the one for the add.  Any ideas what could be the issue here?

I attempted this (external TURN servers) later and it worked.  I’m wondering if sometimes I am experience a timeout?  Or is it possible that since I was able to negotiate a connection with the screensharing that the browsers “remember” somehow and are able to establish the connection?  I’ll do more testing and see if I discover anything else.

What is also strange is that I had the situation again where the PeerConnection failed and I could not establish the call.  In my TURN server logs, I could see the `ALLOCATE` and `CHANNEL_BIND` packets, but there was a long delay before I saw some `BINDING` packets.  I was stuck in this state until I restarted the LiveKit server - then everything connected right up and I saw `ALLOCATE`-&gt; `CREATE_PERMISSION`  -&gt; `CHANNEL_BIND` -&gt; `BINDING`.  I am completely perplexed why only after I restart the LiveKit server do I start getting the `CHANNEL_BIND` packets.  I have already been able to reproduce this behavior two times (didn’t work until a LiveKit server restart).

Just to be clear. The screen sharing <@U04C2CDU5KP> is mentioning is not going through the LiveKit SFU, it’s a direct WebRTC P2P connection forced to go via the same TURN server.

<@U04DHDHMBT8> There was an edge case and it was addressed in this commit of server - <https://github.com/livekit/livekit/pull/1208>. We do not have a release of the server with that. But, if you can point to a hash of server and try it, that would be great.

Hi Justin, did you pull the submodules <https://github.com/livekit/client-sdk-android|here> . this is only needed when opening the source repo itself

yes, we pulled the submodules down and unzipped it, got a folder named "client-sdk-android-main" and drag the whole folder to AS and encountered the errors.

did you clone the repo? or downloaded via GH as a zip? I'm not sure if submodules get pulled down correctly with zip downloads

Hmm.. I think so, but from the screenshot it's clear to see that with "import io.livekit...", AS got it right, but with "import livekit.LivekitRtc" or "import livekit.LivekitModels", AS doesn't seem to resolve this.

so I think what David’s saying is that to get the sample app you want to run

```git clone git@github.com:livekit/client-sdk-android.git
git submodule update --init```
… and then open that directory in AS.  If you’re manually unzipping any files, I’m not sure that will work.

Are you developing on Windows?

Ahh I see. I'll try that. And yes, on Windows.
Thank you, I'll try it out now.

still no luck.. got the same as result
any chance to figure out the problem from the screenshot?
thanks again!

oh wow...it's empty.
how come it's empty even if i use git clone?

oddly i did run "git submodule update --init"!

Is this still in the folder you downloaded with the zip?

You should delete that folder and only use the folder created by git clone

so i did get a "protocol" folder, but it's empty.
and this is what i did with git.

Oh oops, after the git clone you have to cd into the cloned directory

no one here is going to claim that `git submodule …` is intuitive the first time you use it :sweat_smile:

That's true haha
Anyway big thanks to all staffs for fast responses!

<@U04CQDLBVQA> <https://developer.mozilla.org/en-US/docs/Web/Media/Autoplay_guide>

If you have an audio track along with video on that `&lt;video&gt;` element, the browser can block autoplay entirely without user interactions.

We do have a way to handle this browser restriction. Take a look at <https://github.com/livekit/client-sdk-js#audio-playback|this>

the case looks like
```#EXTM3U
#EXT-X-VERSION:4
#EXT-X-PLAYLIST-TYPE:EVENT
#EXT-X-ALLOW-CACHE:NO
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-TARGETDURATION:1
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00000.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00001.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00002.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00003.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00004.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00005.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00006.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00007.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00008.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00009.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00010.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00011.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00012.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00013.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00014.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00015.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00016.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00017.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00018.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00019.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00020.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00021.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00022.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00023.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00024.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00025.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00026.ts
#EXTINF:1.000,
f569a4ed1c94ae8730ed69faee843990c07e97766af716339c36d0aa534aa121_00027.ts```

are you seeing a `pipeline frozen` error in your logs?

yes, there it is
```{"level":"error","ts":1670529806.1926138,"logger":"egress","caller":"pipeline/pipeline.go:464","msg":"pipeline frozen","nodeID":"NE_kBwVriih8EKx","egressID":"EG_kXvvMxZbpYPJ","stacktrace":"<http://github.com/livekit/egress/pkg/pipeline.(*Pipeline).SendEOS.func1.1.1\n\t/workspace/pkg/pipeline/pipeline.go:464|github.com/livekit/egress/pkg/pipeline.(*Pipeline).SendEOS.func1.1.1\n\t/workspace/pkg/pipeline/pipeline.go:464>"}```

It’s a known issue - it can lag behind if it doesn’t have enough CPU (we recommend 8 cores), especially if recording at 1080p, and we kill it after 30s to make sure it doesn’t run out of memory and crash. For now, if the end tag is important, you can use more CPUs and/or a lower resolution, and we will be looking into fixing this soon

thank you, currently before sending m3u8 file to player, I'm checking if it has closed or not, will do close method if it was still open

We don’t currently support CBR :disappointed:

hi <@U04F5J001LG>
there’s different scenarios depending on your use case.

For ultra-low-latency in a one-to-many application you would still want to use livekit-server itself. (be aware that if you’re *not* using livekit cloud you might run into scaling issues at some point, have a look at these benchmarks to see if the self hosted version fits your needs <https://docs.livekit.io/oss/benchmark/#video-room>)

If what you are looking for is more “traditional” live-streaming like HLS, then that service is called `LiveKit Egress`. If you self host, then this is a separate service that you have to deploy in addition to the livekit server.
More in depth explanation here:
<https://docs.livekit.io/server/egress/>

in this case it is for a livecommerce application.  average of 60 minutes of Live for 15,000 people.  would the livekit alone be enough?  As for scaling, I can configure it on my server side so I don't have problems, right? <@U02EU2ZPTQU> 

I cannot give you specific numbers for what will work and what will not. For this kind of scale it probably makes sense if you run your own benchmarks using the load tester on your own infrastructure!

Just to reiterate: you don’t need to worry about scaling if you consider livekit cloud :slightly_smiling_face:

Luiz, for live shopping applications, latency of traditional HLS streams can be a challenge. LiveKit is designed for applications like that.

If self-hosting, a single server would not be able to serve 15k users in a single stream. With Cloud, we run the servers in <https://blog.livekit.io/scaling-webrtc-with-distributed-mesh/|a distributed manner> to support 100k users in a single session.

<@U01KZ61GV5Z> I will bring the possibility of the cloud to the team.  but just for the record, what would be the ideal setup for a high volume livecommerce stream?  between 20,000 and 50,000 people

Another question: is there already a defined way of testing the load in the livekit?  in the similar case the sessions to see if it holds

With Cloud, you don't worry about the setup.. you can simply spin up participants to join.

for load testing your own installation, check out <https://docs.livekit.io/oss/benchmark/#load-tester|load testing docs>. There is no need to load test Cloud, we guarantee its scalability and performance.

Is there any opensource chat service you recommend to work with livekit?

you can use data messages to accomplish this: <https://docs.livekit.io/client/data-messages/#data-messages>

<@U01KZ61GV5Z> is there any tool to calculate the cost of the cloud or a formula to have a base? in the cloud, what would be the average cost for a livecommerce app that will do an average of 3 simultaneous lives with one person transmitting and in each live 10,000 viewers?

On the cloud website it informs that on average it is 44% cheaper than competitors. But I just did a simulation of a 60 minute live stream for 5000 people at 740p and it estimates a cost of $801 versus $320 for aws ivs. I don't know if I did the calculation correctly, but if it's correct, the difference between cloud and aws is very big. Wouldn't it be cheaper to host locally?

<@U04F5J001LG> LiveKit and IVS are very different products. IVS is a one to many solution with about 5-10s of latency from publisher to subscribers.

LiveKit is a real-time framework where multiple users can interact with each other with only ~100ms of latency. because of that, our pricing page is comparing to other real-time services.

Do you have a latency requirement? if not, you could use livekit for the streamer, then publish it out to IVS for distribution to viewer.

it's a liveshop application, low latency is important.  What would your last suggestion to use both solutions look like?  would it reduce the cost? <@U01KZ61GV5Z> 

Can you tell me if installing and configuring it on a server would considerably reduce the cost?  I understand that we would have the responsibility to maintain the infrastructure, but could we have this cost estimate with local installation?

that really depends on how you run it and what resources you'd have to pay for. I'm not able to comment on what hosting providers charge

How would this option you suggested to use the livekit and the IVS together?

<@U04F5J001LG> the “hybrid” approach would work like this:

You use LiveKit for your broadcasters/streamers (and perhaps some of your subscribers, if there are those you want to have on ultra low latency). For everyone else—all or most of the subscribers where you don’t care as much about higher latency—you can use LiveKit (egress) to publish out an RTMP url which IVS can consume and livestream (using HLS) to them.

I refer to it as “hybrid” because some of your users are using WebRTC (via LiveKit) and others are consuming an HLS stream (via IVS).

With this approach, self-hosted LiveKt could support as many sessions as you want, where each individual session has 1-2 broadcasters and low-to-mid thousands of subscribers all with ~100ms latency. If you need to handle additional subscribers beyond that number, they’d be consuming a higher latency livestream and not able to interact in real-time.

Does that make sense?

Let me know if I can clarify anything.

Are you still interested in an answer to this specific question, or did you already get it answered in the other thread?

I should maybe clarify that the main problem behind this is that ideally the main program audio should be recordable locally by the host (and allows things like post-processing the main feed using studio hw). Recording the egress is not good enough.

Actually, answering to myself, this is mainly an issue with setting up the hw and recording locally, everything *just works* using normal LiveKit tracks. The host just needs to send one clean (ie. mix-minus) program feed to the audience, and then the app just needs to figure out which tracks the audience needs to subscribe to. There's no need to send the full program feed back to the audience bc the latency is so small!

Sorry for the noise! I guess I had been thinking this from the software perspective so long that forgot how easy this is to do when you add proper hw in to the mix :smile:

Still, being able to do this purely with sw would be pretty cool!

Hey David, that’s correct. Under GDPR, your deployment of LiveKit is a data processor for CID in-flight, but not at rest, since no CID is stored. I can confirm this is the general strategy for the core of LiveKit now and in the future. Auxiliary services may deviate from this, however.

For example, Egress does temporarily store CID in memory (sandboxed within a Docker container) before uploading a recording (or segment) to your specified cloud.

That’s perfect, thanks for the fast and accurate feedback :+1: