LiveKit Community

Hi <@U03PARRPHS4>, missing 20s seems unexpected. I’ll let <@U01UB0LHTTQ> comment, but putting any issues aside, we’re also working on Chrome pooling, as well.

the `START_RECORDING` log occurs as soon as a participant joins the room (normally happens ~500ms after chrome is launched) - are there already participants in the room, or are you starting egress before people join?

<@U01UB0LHTTQ>, The egress start recording after more than one participant joined the meeting for this case.
```{"level":"debug","ts":1670902069.5913672,"logger":"egress","caller":"service/service.go:168","msg":"request received","nodeID":"NE_nWkNtRgAdZUz","egressID":"EG_7u2nYaurU5k4","requestID":"RPC_XJPiETsFPLX6","senderID":"ND_KU22jczb"}
{"level":"debug","ts":1670902069.5914743,"logger":"egress","caller":"stats/monitor.go:157","msg":"cpu request","nodeID":"NE_nWkNtRgAdZUz","accepted":true,"availableCPUs":9.999798573127235,"numCPUs":10}
{"level":"info","ts":1670902069.5926645,"logger":"egress","caller":"service/service.go:211","msg":"request accepted","nodeID":"NE_nWkNtRgAdZUz","egressID":"EG_7u2nYaurU5k4","requestID":"RPC_XJPiETsFPLX6","senderID":"ND_KU22jczb"}
{"level":"debug","ts":1670902069.5928519,"logger":"egress","caller":"params/params.go:724","msg":"writing to file","nodeID":"NE_nWkNtRgAdZUz","egressID":"EG_7u2nYaurU5k4","filename":"/out/data/EG_7u2nYaurU5k4/1052185049105313792.mp4"}
{"level":"debug","ts":1670902069.6199641,"logger":"egress","caller":"server/main.go:120","msg":"handler launched","nodeID":"NE_dyJzJvKPqBFn"}
{"level":"info","ts":1670902069.6200974,"logger":"egress","caller":"server/main.go:124","msg":"setting TMPDIR environment and creating path","nodeID":"NE_dyJzJvKPqBFn","path":"/tmp/lkegress/EG_7u2nYaurU5k4"}
{"level":"info","ts":1670902069.6202557,"logger":"egress","caller":"redis/redis.go:54","msg":"connecting to redis","nodeID":"NE_dyJzJvKPqBFn","cluster":true,"addr":["192.168.100.30:6379","192.168.100.30:6380","192.168.100.31:6381","192.168.100.31:6382","192.168.100.32:6383","192.168.100.32:6384"]}
{"level":"debug","ts":1670902069.6342988,"logger":"egress","caller":"params/params.go:724","msg":"writing to file","nodeID":"NE_dyJzJvKPqBFn","egressID":"EG_7u2nYaurU5k4","filename":"/out/data/EG_7u2nYaurU5k4/1052185049105313792.mp4"}
{"level":"debug","ts":1670902069.6406364,"logger":"egress","caller":"web/source.go:49","msg":"launching xvfb","nodeID":"NE_dyJzJvKPqBFn","egressID":"EG_7u2nYaurU5k4","display":":400328796","dims":"1920x1080x24"}
{"level":"debug","ts":1670902069.6443985,"logger":"egress","caller":"web/source.go:83","msg":"launching chrome","nodeID":"NE_dyJzJvKPqBFn","egressID":"EG_7u2nYaurU5k4","url":"<https://egress-composite.livekit.io?layout=speaker-dark&amp;token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NzA5ODg0NjksImlzcyI6IkFQSUNGRWJ1RjVoSHo3QyIsIm5iZiI6MTY3MDkwMjA2OSwic3ViIjoiRUdfN3UybllhdXJVNWs0IiwidmlkZW8iOnsiY2FuUHVibGlzaCI6ZmFsc2UsImNhblB1Ymxpc2hEYXRhIjpmYWxzZSwiY2FuU3Vic2NyaWJlIjp0cnVlLCJoaWRkZW4iOnRydWUsInJlY29yZGVyIjp0cnVlLCJyb29tIjoiMTc3ODE2MzE4ODM2NzM3XzE1Mjc3MDAwNzA3Mjc2OSIsInJvb21Kb2luIjp0cnVlfX0.450A9cNv1RrwGtVdJlaas6NUyIc2rbhcmAD5SdLQLYs&amp;url=wss%3A%2F%2Fqm-live.xiangxiang.com.cn>"}
{"level":"debug","ts":1670902081.769755,"logger":"egress","caller":"web/source.go:172","msg":"chrome log: START_RECORDING","nodeID":"NE_dyJzJvKPqBFn","egressID":"EG_7u2nYaurU5k4"}```
And the event log of webhook like below:
```INFO 12/13-11:27:45.283 /root/go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/hooks.go:28 livekit的webhook: event:"room_started"  room:{sid:"RM_Z2ibi6k7thc4"  name:"177816318836737_152770007072769"  empty_timeout:300  creation_time:1670902065  turn_password:"t4FC6aB9LWd2QxYtJajE0flOfl91LLKPh3uLPRBZiN5"  enabled_codecs:{mime:"audio/opus"}  enabled_codecs:{mime:"audio/red"}  enabled_codecs:{mime:"video/VP8"}  enabled_codecs:{mime:"video/H264"}}  id:"EV_mbUqaBoEw5mK"  created_at:1670902065
INFO 12/13-11:27:45.763 /root/go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/hooks.go:28 livekit的webhook: event:"participant_joined"  room:{sid:"RM_Z2ibi6k7thc4"  name:"177816318836737_152770007072769"  empty_timeout:300  creation_time:1670902065  turn_password:"t4FC6aB9LWd2QxYtJajE0flOfl91LLKPh3uLPRBZiN5"  enabled_codecs:{mime:"audio/opus"}  enabled_codecs:{mime:"audio/red"}  enabled_codecs:{mime:"video/VP8"}  enabled_codecs:{mime:"video/H264"}  num_participants:1}  participant:{sid:"PA_dHwax7HG74Ko"  identity:"177816318836737"  state:ACTIVE  joined_at:1670902065  version:9  permission:{can_subscribe:true  can_publish:true  can_publish_data:true}  is_publisher:true}  id:"EV_d7JE47mwJLFW"  created_at:1670902065
INFO 12/13-11:27:46.123 /root/go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/hooks.go:28 livekit的webhook: event:"track_published"  room:{sid:"RM_Z2ibi6k7thc4"  name:"177816318836737_152770007072769"}  participant:{sid:"PA_dHwax7HG74Ko"  identity:"177816318836737"}  track:{sid:"TR_AMnUyEyuWyZ8eC"  source:MICROPHONE  mime_type:"audio/red"}  id:"EV_juKAzBjvHCwt"  created_at:1670902066
INFO 12/13-11:27:49.596 /root/go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/hooks.go:28 livekit的webhook: event:"egress_started"  egress_info:{egress_id:"EG_7u2nYaurU5k4"  room_id:"RM_Z2ibi6k7thc4"  room_name:"177816318836737_152770007072769"  room_composite:{room_name:"177816318836737_152770007072769"  layout:"speaker-dark"  file:{file_type:MP4  filepath:"live/177816318836737_152770007072769/2022-12-13/1052185049105313792.mp4"}}  file:{filename:"live/177816318836737_152770007072769/2022-12-13/1052185049105313792.mp4"}}  id:"EV_TXjpLXXkC3Pa"  created_at:1670902069```


do you have any livekit-server logs for when participant `EG_7u2nYaurU5k4` joined the room? That would be the egress instance, I’m wondering if it took a long time to connect or if it was some other issue

Here is the livekit-server logs at the same time:
```2022-12-13T11:27:45.835+0800	INFO	livekit	rtc/participant.go:410	received offer	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "177816318836737", "pID": "PA_dHwax7HG74Ko", "remote": false, "transport": "PUBLISHER"}
2022-12-13T11:27:45.836+0800	INFO	livekit	rtc/participant.go:438	sending answer	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "177816318836737", "pID": "PA_dHwax7HG74Ko", "remote": false, "transport": "PUBLISHER"}
2022-12-13T11:27:46.118+0800	INFO	livekit	rtc/participant.go:1160	mediaTrack published	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "177816318836737", "pID": "PA_dHwax7HG74Ko", "remote": false, "kind": "audio", "trackID": "TR_AMnUyEyuWyZ8eC", "rid": "", "SSRC": 267650044, "mime": "audio/red"}
2022-12-13T11:27:49.593+0800	INFO	livekit	service/twirp.go:111	API Egress.StartRoomCompositeEgress	{"service": "Egress", "method": "StartRoomCompositeEgress", "duration": "3.960484ms", "status": "200"}
2022-12-13T11:27:52.305+0800	INFO	livekit	service/roommanager.go:261	starting RTC session	{"room": "177816318836737_152770007072769", "nodeID": "ND_KU22jczb", "participant": "152770007072769", "sdk": "ANDROID", "sdkVersion": "1.1.4", "protocol": 8}
2022-12-13T11:27:52.306+0800	INFO	livekit	rtc/room.go:297	new participant joined	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "pID": "PA_QZ5ifpSAyfmr", "participant": "152770007072769", "protocol": 8, "options": {"AutoSubscribe":true}}
2022-12-13T11:27:52.308+0800	INFO	livekit	service/rtcservice.go:242	new client WS connected	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "152770007072769", "remote": false, "connID": "CO_m69XoVPPLx2Y"}
2022-12-13T11:27:52.308+0800	INFO	livekit	rtc/participant.go:1138	sending offer	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "152770007072769", "pID": "PA_QZ5ifpSAyfmr", "remote": false, "transport": "SUBSCRIBER"}
2022-12-13T11:27:52.309+0800	INFO	livekit	rtc/participant.go:880	added subscribedTrack	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "152770007072769", "pID": "PA_QZ5ifpSAyfmr", "remote": false, "publisherID": "PA_dHwax7HG74Ko", "publisherIdentity": "177816318836737", "trackID": "TR_AMnUyEyuWyZ8eC"}
2022-12-13T11:27:52.435+0800	INFO	livekit	rtc/participant.go:424	received answer	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "152770007072769", "pID": "PA_QZ5ifpSAyfmr", "remote": false, "transport": "SUBSCRIBER"}
2022-12-13T11:27:52.460+0800	INFO	livekit	rtc/participant.go:1138	sending offer	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "152770007072769", "pID": "PA_QZ5ifpSAyfmr", "remote": false, "transport": "SUBSCRIBER"}
2022-12-13T11:27:52.573+0800	INFO	livekit	rtc/transport.go:479	selected ICE candidate pair	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "152770007072769", "pID": "PA_QZ5ifpSAyfmr", "remote": false, "transport": "SUBSCRIBER", "pair": "(local) udp4 host 14.125.52.152:56501 &lt;-&gt; (remote) udp4 host 172.16.11.53:50469"}
2022-12-13T11:27:52.638+0800	INFO	livekit	rtc/transport.go:1366	ice candidates	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "152770007072769", "pID": "PA_QZ5ifpSAyfmr", "remote": false, "transport": "SUBSCRIBER", "lc": ["udp4 host 14.125.52.152:56501", "tcp4 host 14.125.52.152:7881"], "rc": ["candidate:2783312442 1 udp 2122260223 172.16.11.53 50469 typ host generation 0 ufrag AHdp network-id 3 network-cost 10"], "lc (filtered)": [], "rc (filtered)": []}
2022-12-13T11:27:52.775+0800	INFO	livekit	rtc/participant.go:424	received answer	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "152770007072769", "pID": "PA_QZ5ifpSAyfmr", "remote": false, "transport": "SUBSCRIBER"}
2022-12-13T11:27:52.807+0800	INFO	livekit	rtc/participant.go:410	received offer	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "152770007072769", "pID": "PA_QZ5ifpSAyfmr", "remote": false, "transport": "PUBLISHER"}
2022-12-13T11:27:52.810+0800	INFO	livekit	rtc/participant.go:438	sending answer	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "152770007072769", "pID": "PA_QZ5ifpSAyfmr", "remote": false, "transport": "PUBLISHER"}
2022-12-13T11:27:52.849+0800	INFO	livekit	rtc/transport.go:479	selected ICE candidate pair	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "152770007072769", "pID": "PA_QZ5ifpSAyfmr", "remote": false, "transport": "PUBLISHER", "pair": "(local) udp4 host 14.125.52.152:54334 &lt;-&gt; (remote) udp4 host 172.16.11.53:44827"}
2022-12-13T11:27:52.917+0800	INFO	livekit	rtc/transport.go:1366	ice candidates	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "152770007072769", "pID": "PA_QZ5ifpSAyfmr", "remote": false, "transport": "PUBLISHER", "lc": ["tcp4 host 14.125.52.152:7881", "udp4 host 14.125.52.152:54334"], "rc": ["candidate:2783312442 1 udp 2122260223 172.16.11.53 44827 typ host generation 0 ufrag 6bpC network-id 3 network-cost 10", "candidate:2783312442 1 udp 2122260223 172.16.11.53 42478 typ host generation 0 ufrag 6bpC network-id 3 network-cost 10"], "lc (filtered)": [], "rc (filtered)": []}
2022-12-13T11:27:53.150+0800	INFO	livekit	rtc/participant.go:1160	mediaTrack published	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "152770007072769", "pID": "PA_QZ5ifpSAyfmr", "remote": false, "kind": "audio", "trackID": "TR_AMVf22QrmLHUaS", "rid": "", "SSRC": 1791860272, "mime": "audio/red"}
2022-12-13T11:27:53.151+0800	INFO	livekit	rtc/participant.go:880	added subscribedTrack	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "177816318836737", "pID": "PA_dHwax7HG74Ko", "remote": false, "publisherID": "PA_QZ5ifpSAyfmr", "publisherIdentity": "152770007072769", "trackID": "TR_AMVf22QrmLHUaS"}
2022-12-13T11:27:53.302+0800	INFO	livekit	rtc/participant.go:1138	sending offer	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "177816318836737", "pID": "PA_dHwax7HG74Ko", "remote": false, "transport": "SUBSCRIBER"}
2022-12-13T11:27:53.426+0800	INFO	livekit	rtc/participant.go:424	received answer	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "177816318836737", "pID": "PA_dHwax7HG74Ko", "remote": false, "transport": "SUBSCRIBER"}
2022-12-13T11:28:01.793+0800	INFO	livekit	service/roommanager.go:261	starting RTC session	{"room": "177816318836737_152770007072769", "nodeID": "ND_KU22jczb", "participant": "EG_7u2nYaurU5k4", "sdk": "JS", "sdkVersion": "1.3.2", "protocol": 8}
2022-12-13T11:28:01.794+0800	INFO	livekit	rtc/room.go:297	new participant joined	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "pID": "PA_5aspDYPGcujR", "participant": "EG_7u2nYaurU5k4", "protocol": 8, "options": {"AutoSubscribe":true}}
2022-12-13T11:28:01.796+0800	INFO	livekit	rtc/participant.go:1138	sending offer	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "EG_7u2nYaurU5k4", "pID": "PA_5aspDYPGcujR", "remote": false, "transport": "SUBSCRIBER"}
2022-12-13T11:28:01.796+0800	INFO	livekit	service/rtcservice.go:242	new client WS connected	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "EG_7u2nYaurU5k4", "remote": false, "connID": "CO_5DZ24TnW9axQ"}
2022-12-13T11:28:01.799+0800	INFO	livekit	rtc/participant.go:880	added subscribedTrack	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "EG_7u2nYaurU5k4", "pID": "PA_5aspDYPGcujR", "remote": false, "publisherID": "PA_QZ5ifpSAyfmr", "publisherIdentity": "152770007072769", "trackID": "TR_AMVf22QrmLHUaS"}
2022-12-13T11:28:01.799+0800	INFO	livekit	rtc/participant.go:880	added subscribedTrack	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "EG_7u2nYaurU5k4", "pID": "PA_5aspDYPGcujR", "remote": false, "publisherID": "PA_dHwax7HG74Ko", "publisherIdentity": "177816318836737", "trackID": "TR_AMnUyEyuWyZ8eC"}
2022-12-13T11:28:01.840+0800	INFO	livekit	rtc/participant.go:424	received answer	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "EG_7u2nYaurU5k4", "pID": "PA_5aspDYPGcujR", "remote": false, "transport": "SUBSCRIBER"}
2022-12-13T11:28:01.950+0800	INFO	livekit	rtc/participant.go:1138	sending offer	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "EG_7u2nYaurU5k4", "pID": "PA_5aspDYPGcujR", "remote": false, "transport": "SUBSCRIBER"}
2022-12-13T11:28:01.965+0800	INFO	livekit	rtc/participant.go:424	received answer	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "EG_7u2nYaurU5k4", "pID": "PA_5aspDYPGcujR", "remote": false, "transport": "SUBSCRIBER"}
2022-12-13T11:28:08.928+0800	INFO	livekit	service/twirp.go:111	API Egress.StopEgress	{"service": "Egress", "method": "StopEgress", "duration": "2.988226ms", "status": "200"}```

<@U01UB0LHTTQ> Yes, It seems the egress client takes a long time to join the meeting:
```2022-12-13T11:27:49.593+0800	INFO	livekit	service/twirp.go:111	API Egress.StartRoomCompositeEgress	{"service": "Egress", "method": "StartRoomCompositeEgress", "duration": "3.960484ms", "status": "200"}
...
2022-12-13T11:27:49.644+0800
{"level":"debug","ts":1670902069.6443985,"logger":"egress","caller":"web/source.go:83","msg":"launching chrome","nodeID":"NE_dyJzJvKPqBFn","egressID":"EG_7u2nYaurU5k4","url":"<https://egress-composite.livekit.io?layout=speaker-dark&amp;token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NzA5ODg0NjksImlzcyI6IkFQSUNGRWJ1RjVoSHo3QyIsIm5iZiI6MTY3MDkwMjA2OSwic3ViIjoiRUdfN3UybllhdXJVNWs0IiwidmlkZW8iOnsiY2FuUHVibGlzaCI6ZmFsc2UsImNhblB1Ymxpc2hEYXRhIjpmYWxzZSwiY2FuU3Vic2NyaWJlIjp0cnVlLCJoaWRkZW4iOnRydWUsInJlY29yZGVyIjp0cnVlLCJyb29tIjoiMTc3ODE2MzE4ODM2NzM3XzE1Mjc3MDAwNzA3Mjc2OSIsInJvb21Kb2luIjp0cnVlfX0.450A9cNv1RrwGtVdJlaas6NUyIc2rbhcmAD5SdLQLYs&amp;url=wss%3A%2F%2Fqm-live.xiangxiang.com.cn>"}
2022-12-13T11:28:01.770+0800
{"level":"debug","ts":1670902081.769755,"logger":"egress","caller":"web/source.go:172","msg":"chrome log: START_RECORDING","nodeID":"NE_dyJzJvKPqBFn","egressID":"EG_7u2nYaurU5k4"} 
...
2022-12-13T11:28:01.793+0800	INFO	livekit	service/roommanager.go:261	starting RTC session	{"room": "177816318836737_152770007072769", "nodeID": "ND_KU22jczb", "participant": "EG_7u2nYaurU5k4", "sdk": "JS", "sdkVersion": "1.3.2", "protocol": 8}
2022-12-13T11:28:01.794+0800	INFO	livekit	rtc/room.go:297	new participant joined	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "pID": "PA_5aspDYPGcujR", "participant": "EG_7u2nYaurU5k4", "protocol": 8, "options": {"AutoSubscribe":true}}
2022-12-13T11:28:01.796+0800	INFO	livekit	rtc/participant.go:1138	sending offer	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "EG_7u2nYaurU5k4", "pID": "PA_5aspDYPGcujR", "remote": false, "transport": "SUBSCRIBER"}
2022-12-13T11:28:01.796+0800	INFO	livekit	service/rtcservice.go:242	new client WS connected	{"room": "177816318836737_152770007072769", "roomID": "RM_Z2ibi6k7thc4", "participant": "EG_7u2nYaurU5k4", "remote": false, "connID": "CO_5DZ24TnW9axQ"}```
It trigger egress start at 2022-12-13T11:27:49.593+0800, the chrome start at 2022-12-13T11:27:49.644+0800, and JS client start recording at 2022-12-13T11:28:01.770+0800, The egress server got the request from js client at 2022-12-13T11:28:01.793+0800 and the egress client as a participant join the meeting at 2022-12-13T11:28:01.794+0800.
It seems the chrome start take about 12 seconds.

Are your egress and livekit server in the same data center? I’m unable to reproduce that, but that is a long time for the connection

Hey <@U02PDJ8R2A1>, they indeed can, but as of right now you have to build that functionality yourself. Here’s a high-level overview of how we did it for LiveKit Cloud:

<https://blog.livekit.io/scaling-webrtc-with-distributed-mesh/|https://blog.livekit.io/scaling-webrtc-with-distributed-mesh/>

we’ve recently run into the same issue - I’m currently updating the entire pulse+chrome stack. For now, the only solution is to restart the service

<@U01UB0LHTTQ>, After do some research, I found the root cause and make a pull request: <https://github.com/livekit/egress/pull/197>, please help to review it. TKS.

<@U045CJPACG1> , What version of livekit server? The latest release version may not support cluster configuration. The commit of livekit server support cluster is <https://github.com/livekit/livekit/commit/e5dabd466ec58676f52a393f4471ab472fbd2f28> which not include the latest version. The latest release version is 1.3.1 and it released at Nov 10.

<@U03PARRPHS4> noted with thanks, we are using latest 1.3.1 version

So you have to wait for the next release or build yourself version from master.
BTW, I also build a docker image base on master. <https://hub.docker.com/repository/docker/tomxiong/livekit-server>

got it, so based on code commit redis sentinel also not supported in current version right ? will check building docker image and confirm. Thanks for the tip.

Any idea when will have the next release for this month ?

we will have a new release cut today or tomorrow!

<@U01KZ61GV5Z> <@U03PARRPHS4> After livekit-server v1.3.2 update we connected to redis cluster mode and we get following error and pod getting restarted non-stop
```could not get rooms: MOVED 4321 10.2.1.4:6379```
livekit-server log:
```2022-12-21T09:39:33.753Z	INFO	livekit	redis/redis.go:73	connecting to redis	{"cluster": true, "addr": ["redis-cluster.redis-cluster.svc.cluster.local:6379"]}
2022-12-21T09:39:34.038Z	INFO	livekit	rtc/config.go:303	external ip already solved, ignore duplicate	{"external": "X.X.X.X", "local": "10.2.5.1", "ignore": "X.X.X.X"}
2022-12-21T09:39:34.038Z	INFO	livekit	rtc/config.go:303	external ip already solved, ignore duplicate	{"external": "X.X.X.X", "local": "10.2.5.1", "ignore": "172.17.0.1"}
2022-12-21T09:39:35.034Z	DEBUG	livekit	rtc/config.go:95	using external IPs	{"ips": ["X.X.X.X/X.X.X.X", "X.X.X.X/X.X.X.X", "172.17.0.1/172.17.0.1", "X.X.X.X/X.X.X.X", "10.2.5.1/10.2.5.1"]}
2022-12-21T09:39:35.035Z	INFO	livekit	service/turn.go:127	Starting TURN server	{"turn.relay_range_start": 30000, "turn.relay_range_end": 40000, "turn.portTLS": 3478, "turn.externalTLS": false}
could not get rooms: MOVED 4321 10.2.1.4:6379```
livekit helm chart override.yaml

```replicaCount: 1

# Refer to <https://docs.livekit.io/deploy/kubernetes/> for instructions

image:
  tag: v1.3.2

livekit:
  log_level: info
  rtc:
    use_external_ip: true
  redis:
    #address: redis-master.redis.svc.cluster.local:6379
    password: XXXXXXX
    
    # redis-cluster enabled 
    cluster_addresses:
    - redis-cluster.redis-cluster.svc.cluster.local:6379
      
  keys:
    XXXXX: XXXXXXXXXXXXXXXXXXXXX
  turn:
    enabled: true 
    domain: &lt;turn-domain&gt;
    tls_port: 3478
    secretName: lvkt-turn-ssl-secret
...```
Any insights to into sort the redis issue.

Ashan, it looks like Redis cluster mode support isn't configured correctly. it appears we'd have to set MaxRedirects in Redis options.

Got it. Thank you for the confirmation.

So should that need to handle from our end in redis settings/ livekit server settings or is there going be a patch in livekit-server code ( for redis redirect handling in cluster mode) ?

Got it. Thank you for the confirmation.
is there going be a patch in livekit-server code ( for redis redirect handling in cluster mode) soon ?

<@U01KZ61GV5Z> ^^

<@U045CJPACG1> this was contributed by the community and currently not a priority at the moment. If you are interested in contributing, I'd welcome it!

Noted with thanks. Will look into it and confirm.

do you mean to use CRF and have it automatically adapt bitrate? that seems like a param we should support.

want to open an issue for this?

thank you! use CRF and have it automatically adapt bitrate, yes, exactly, do you mean open an issue by me?

yup, I filed an issue here: <https://github.com/livekit/egress/issues/193>

Not sure if this is helpful, but LiveKit now supports ingress.

what do you consider "ingress" ? :slightly_smiling_face:

I dont think they have the proper doc page up &amp; running just yet

<@U047TQP3550> what source generated this RTP? Is it just raw packets? Wondering how one would handle things like transmission loss and rate control.

Ingress will support WHIP soon, so if your source can spit out webrtc, that'd be a better path.

It shouldn't be too hard to ingest raw RTP either. I think the best path is probably to use an adapter that pushes samples via our Go SDK.

without going into too much detail.... it wouldnt be RTP over the internet....... would be from a controlled internal to livekit network....

so i dont expect transmission loss / rate control to really be an issue

ingest rtp... egress via livekit sfu webrtc

I see. We currently don't have a way to do this. I think the RTP adapter to livekit SDK is probably the best path to get it to work.

its actually one of the features ive wanted for a while for say consuming from a near by gstreamer process (that isnt under appsink/appsrc) (wanted from other SFUs)

ah, yeah, if you have GStreamer, you can feed it into our CLI with a TCP server and `--publish` flags. <https://github.com/livekit/livekit-cli#publish-from-tcp-ie-gstreamer>

I need to whip up a full example pipeline for it

Haven’t heard any other reports of this problem.
Would you want to open an issue on github with some steps on how to reproduce it and the error logs you are seeing? That’d be very helpful!

you could set `videoCodec` to ‘h264’ on the `TrackPublishDefaults` when creating your room

Hey all, we've just released <https://github.com/livekit/livekit/releases/tag/v1.3.2|v1.3.2>. It contains a few bug fixes including a workaround for a bug in Firefox when connecting over TURN.

This release also contains an internal protocol change. This change is made in a backwards-compatible manner. We recommend upgrading to this version as we'll eventually deprecate the old protocol.

Hey all, I am one of the co-founders of <http://unping.com|unping.com>
Unping is a virtual office &amp; video call app fully developed in Flutter for Desktop by using LiveKit :heart: .
You can use it as an alternative to around, teamflow or even teams/hangout.
You can do internal and external meetings or just have informal talks with your team mates.
We just released an *early access version for Mac* (Windows coming in some days).
Feel free to check the website and<https://app.unping.com/signin| create a workspace> + download the app and try it with your team. If you need any help, please let me know :slightly_smiling_face:
Thanks a lot to <@U01KZ61GV5Z> <@U02GAC58A79> and the rest of the team to make it possible!

to be honest, if you’re asking questions at this level it might make more sense for you to use our hosted cloud service: <https://livekit.io/cloud>

Hey Denis, this should be supported today via <https://docs.livekit.io/client-sdk-swift/LocalVideoTrack/#localvideotrack.createbuffertrack(name:source:options:)|createBufferTrack>. We use this internally to handle screen share publishing.

Hey! I saw this function, maybe I missed something, but I didn't find how to use it for this purpose

Yeah, we don't have any examples right now (and need to make a couple). Basically you would do:

```let track = LocalVideoTrack.createBufferTrack(name: "", source: .camera)
guard let bufferCapturer = track.capturer as? BufferCapturer else { return }

// set up your process to get buffers from camera device and process
// then feed it in to BufferCapturer
// buffer is either a CMSampleBuffer or CVPixelBuffer
bufferCapturer.capture(buffer)```
CC: <@U02MAJLGN65>

Thanks again <@U01KZ61GV5Z>. I checked it really works.  (cc: <@U046B0LNLD7>)

hi <@U04G0ETEG8P>,
this seems related to a recent change! 
Is this a new project or did a previously working project suddenly stop working with this error?

and could you post the output of `npx envinfo --npmPackages "{livekit*,@livekit/*}"` when run within your project?

Do you have any processes running operations on your redis instance? this would happen if some of the keys we write gets deleted

Thanks <@U01KZ61GV5Z>
I don't think but I'll recheck.
Also is there a Redis resources requirements (CPU/Memory) in relation to the number of participants and rooms (You don't think this may fire this issue) .

no, it uses very few resources on redis, any instance would do

Disregard I was able to get it to work if I used a Vp8 encoded video

Not yet, when flutter-webrtc supports insertable streams, it should be able to support file playback

<@U04GJGHGACS> there are other ways of achieving playback of audio files from the server. I would recommend checking out the <https://github.com/livekit/livekit-cli|CLI>

Supporting this from Flutter SDK is not high on our list of priorities.

<@U01KZ61GV5Z> I want to be able to control the video/audio track that I am publishing (i.e play, pause and seek) which is not supported by the cli afaik

Hi <@U04G6JNFR0X>, I’m not familiar with the term, “oc”? Could you tell me a bit more about what you’re looking for?

we do not provide a webpack config. typically the code is bundled into your application and it's up to the application developer to choose the packaging tool.

<@U01KZ61GV5Z> noted. 

In our observations, when build the app usuing turbo ( npm build), we observe that in source code there is some source code js and css code compress build is happening and we assume this proceed from webpack service ( reside in within react example folder or source code root directories). 
 Can you kindly confirm whether if above assumption is correct or incorrect?

On side note: we atrying to implement cache busting (<https://dev.to/flexdinesh/cache-busting-a-react-app-22lk|https://dev.to/flexdinesh/cache-busting-a-react-app-22lk> ) in our react app,  so we can serve user the latest changes without asking to clear cache everytime irrespective of browser and/ or server caching implementation

hi <@U045CJPACG1>
To answer your question, the react sdk is being packaged (minimized) and written to `dist`. you can see the library used for that by inspecting the `build` script in pgk/components/package.json.

cache busting strategies might be a bit out of scope of LiveKit support. As david has mentioned this usually happens in the last build step (not the one of the library, but the one that you are writing/the web app). A framework like nextjs will take care of this for you <https://nextjs.org/docs/going-to-production#caching|automatically> if you are in doubt!

Thank you for the insights <@U02EU2ZPTQU>. Will look into those.

Hey <@U04GN75GRPX>, we’re close to starting a closed beta for a Unity multi-platform SDK. If you’re interested in testing an early version, let <@U022KB72NFJ> know. :slightly_smiling_face:

Hi <@U022KB72NFJ>, we want to participate beta phase, can you include me ? Thank you very much.

Hi again, is there any news about beta phase ?

Hi <@U04GN75GRPX>, happy new year (!), we’re aiming to have a first cut within the next couple weeks.

Thank you. By the way, I'm sorry if it feels like i'm forcing it. We're really excited about our new project.

Hi there <@U04H064JTB5>, I think we need some more information from you to be able to help you. cc <@U01NZKTNE1H>

E/AndroidRuntime(32228): java.lang.SecurityException: Need BLUETOOTH permission
E/AndroidRuntime(32228): 	at android.bluetooth.BluetoothHeadset.&lt;init&gt;(BluetoothHeadset.java:426)
E/AndroidRuntime(32228): 	at android.bluetooth.BluetoothAdapter.getProfileProxy(BluetoothAdapter.java:3144)
E/AndroidRuntime(32228): 	at com.twilio.audioswitch.bluetooth.BluetoothHeadsetManager.start(BluetoothHeadsetManager.kt:156)
E/AndroidRuntime(32228): 	at com.twilio.audioswitch.AudioSwitch.start(AudioSwitch.kt:160)
E/AndroidRuntime(32228): 	at com.cloudwebrtc.webrtc.audio.AudioSwitchManager.lambda$start$2$com-cloudwebrtc-webrtc-audio-AudioSwitchManager(AudioSwitchManager.java:70)
E/AndroidRuntime(32228): 	at com.cloudwebrtc.webrtc.audio.AudioSwitchManager$$ExternalSyntheticLambda1.run(Unknown Source:2)
E/AndroidRuntime(32228): 	at android.os.Handler.handleCallback(Handler.java:938)
E/AndroidRuntime(32228): 	at android.os.Handler.dispatchMessage(Handler.java:99)
E/AndroidRuntime(32228): 	at android.os.Looper.loopOnce(Looper.java:210)
E/AndroidRuntime(32228): 	at android.os.Looper.loop(Looper.java:299)
E/AndroidRuntime(32228): 	at android.app.ActivityThread.main(ActivityThread.java:8118)
E/AndroidRuntime(32228): 	at java.lang.reflect.Method.invoke(Native Method)
E/AndroidRuntime(32228): 	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:556)
E/AndroidRuntime(32228): 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1045)

I believe this is the issue tracking bluetooth permissions: <https://github.com/livekit/client-sdk-android/issues/168>

I believe we are working on it. Please subscribe to the issue for updates.

Related, it'd be great if you can type in english since very few on the team/community could understand Chinese. Thank you!

Yes, we'll need to improve bluetooth handling so it doesn't require that permission to be declared. A workaround is to ask for `BLUETOOTH_CONNECT` for now.

Do you dynamically request BLUETOOTH CONNECT permissions before entering the room?

The target version of our project is Android11 running on Android 12 phones. I don't know why this error occurs

we don’t currently dynamically request BLUETOOTH_CONNECT permissions

we will be removing the need for these permissions soon though, so this should fix the need for requesting those

Do these happen together? It seems like we should change the egress participant to ignore the room participant limit as you suggested, and that should also solve the first issue (websocket timeout means it could not connect to the LiveKit room, but the connection to the template website itself seems to be ok)

<@U01UB0LHTTQ> Thank you for your insights. to our observations we see the issues occurrences in separate instances.
Now that you mention, above 2 issues may relate to each other. Will do some testing further and share if we find any further observation.

As mentioned, are you guys able to issue a fix (the solution you mentioned - change the egress participant to ignore the room participant limit) ?
I assume this is not the first encounter in above behavior (specially for Issue-2).
In cloud, in free tier, after loading 1k users in a session and start recording, will the recording get failed ?

Yes, I'll be adding that fix with the next LiveKit server release. I believe it will currently fail when hitting the user limit in either OSS or cloud

<@U04GJGHGACS> If you are using `Android/Windows/macOS` you can select audioInput/audioOutput by the following methods
<https://github.com/livekit/client-sdk-flutter/blob/main/lib/src/hardware/hardware.dart#L78>
<https://github.com/livekit/client-sdk-flutter/blob/main/lib/src/hardware/hardware.dart#L86>

Hey all.  I’m running under docker locally.  Is it possible to get livekit to log to a file rather than the default?  I couldn’t see in the config-sample.yaml on how to do this.  Thanks

These two PRs are mainly used for End to End Encryption, and InsertableStreams are not yet supported.

image.png

got some clue from networktrace: ip seems error, need change values.yaml?

hi, this generally means your server isn't set up correct. please use the <https://livekit.io/connection-test|connection tester >to verify

Thanks <@U01KZ61GV5Z> , error shows below. in k8s system. the udp port of pod is not open to the internet. How can it setup?

log:   livekit▸rtc/config.go:292▸  external ip already solved, ignore duplicate▸   {“external”: “54.245.193.242", “local”: “10.0.4.104", “ignore”: “10.0.4.55"}

I added the firewall ingress rule. finally ok. Thanks!

Where is the missing ingress yaml example file ? I want to setup an ingress + lk server in k8s

No ingress helm ?
``` helm install ingress  livekit/ingress --namespace lk1  --values ./ingress.yaml
Error: INSTALLATION FAILED: failed to download "livekit/ingress"```
