LiveKit Community

all the port in the deployment guid are also open

yup, websocket is over 7880, all of our signaling is over that channel. Browsers require TLS in front of it.

media goes over UDP (ideally) or TCP.

one tricky thing with kubernetes.. is you'd need to kill the pod to restart it. merely updating the config file would not trigger a reload

okay I was able to restart the pods by uninstalling and reinstalling the deployment. now livekit-server is giving me the error “api_key is required to use webhooks”. When I look in my deployment’s generated configmap I can see the webhook section and it has an api_key field. I’m not sure what’s going on

<@U02T8UUS3J8> you'd want to tell us *which* <https://github.com/livekit/livekit-server/blob/master/config-sample.yaml#L124|API key> should be used to sign webhook requests

it currently doesn't default it to the first one. we'll make that change in the future

Okay I think I misunderstood. I was putting a secret value in place of &lt;api_key&gt;.

If I understand correctly, I need to replace &lt;api_key&gt; with the name of a key that’s defined elsewhere in the config file?

okay the webhooks are making requests to my endpoint now. thanks!

and by the way, this is not the case if I unpublish the track when pausing, if I unpublish the track and then upon resuming I create and publish it again, the video resumes fine

I think this is probably a bug on our end, looks like we need to stop/restart the video capturer accordingly

<https://github.com/livekit/client-sdk-android/issues/71>

We don't store any data, so I'm not sure how that could happen after an app restart

Can you please provide an example of the issue

No, not with the livekit . Actually i want to display livedata from firebase then display it to the ui, but even after restarting the app, Mutable Shared Flow holds the old values . Just need some idea about this from you to how to fix.

sorry, can't help you with that, have you tried stackoverflow?

oh.. it's been updated to `<http://github.com/livekit/protocol/livekit|github.com/livekit/protocol/livekit>`  Let me fix the docs.

Hi Andrius,

There's a few things here. The first is to set `autoSubscribe` to `false` when connecting from a client. That will allow you to select which tracks (audio or video) you'd like to explicitly subscribe to, for each participant.

The second thing is to set `dynacast` (dynamic broadcast), to `true`. The way that works is, if there are no subscribers to a publisher’s video, it will automatically stop sending down that information to participants.

Also note: `autoSubscribe` is a `RoomConnectOption` and `dynacast` is a `RoomOption`.

<https://github.com/livekit/client-sdk-js/blob/main/src/options.ts|https://github.com/livekit/client-sdk-js/blob/main/src/options.ts>

thank you for explanation :pray: . Is there some documentation on how to manually handle subscriptions once you turn off `autoSubscribe` ?

Actually never mind I looked at slack history and I found some answers for this question already. Thank you again :pray:

There’s no official documentation on this yet, but we’re in the process of revamping our docs, so I’ll add this to my list of examples/recipes. :white_check_mark:

Haha, glad you found some answers though in the meantime. Let me know if you need more help/guidance.

Hey Rey, have you taken a look at this yet:
<https://docs.livekit.io/guides/webhooks/>

yes , but to be honest I'm kinda new to webhooks and don't understand it properly , can you provide me a sample code ?

There is sample code on that page. Do you have LiveKit server successfully running? What are you having trouble with understanding with regards to webhooks? Perhaps I can help clarify.

The way webhoohs work is that livekit will call your endpoint every time an event occurs, for example a user joins a room, through these notifications you can log different activities, HTH

Yes, you can do this, but what’s not currently possible is for this participant to share the same identity. You'd have to use two or more different identities for this user.

It is also possible to add tracks from different sources for the same user.

We handle this by sending a composite identity. UId:deviceId

awesome, that’s essentially like the rest of livekit …. infinite possibilities :wink:

Hello,
I've problem with WebRTC bitcode. I added broadcast extension. When I build my app it crashes

DEV/BuildProductsPath/Debug-iphoneos/WebRTC.framework/WebRTC' does not contain bitcode. You must rebuild it with bitcode enabled (Xcode setting ENABLE_BITCODE), obtain an updated library from the vendor, or disable bitcode for this target.

iOS

Or someone had same issue with WebRTC, when broadcast extension created
FBundleIdentifier Collision. There is more than one bundle with the CFBundleIdentifier value 'org.webrtc.WebRTC' under the iOS application '<http://Dev.app|Dev.app>'. With error code STATE_ERROR.VALIDATION_ERROR.90685

Are you wanting to update `node-ip` after the container has been started? Currently, LiveKit Server takes NodeIP either as a command line parameter, or environment variable (see <https://github.com/livekit/livekit-server/blob/master/cmd/server/main.go#L59>)

`node-ip` should not rely on caddy. also, if you are trying to use the public IP of the server, you could enable <https://github.com/livekit/livekit-server/blob/master/config-sample.yaml#L27|use_external_ip> option, that automatically finds your external IP address.

My understanding of the react library is that you would either use the display components or use the hooks and do your own thing.

I personally decided to just integrate livekit sdk directly in my react app so I can manage state however I like.

I managed to fix the issue. Problem was caused by components re-mounting due to some sloppiness in how I was building up the react component tree. Re-renders aren’t actually a problem because I can prevent side effects with useEffect and appropriate dependencies.

Hi David, would you mind sharing an example of how you mounted useRoom inside a context provider. I believe I have a similar use case than yours. Thanks!

I want to make sure my EKS can scale properly.
I simply don’t know how to do a test with hundreds/thousands of rooms.

you should be able to use the load tester, but it’ll add your participants to a single room

you could write a script that would run multiple load testers with different room names

I thought about using multiprocessing in python to do it, not sure if it can be done for 100~1000 processes.

python should work too! but you’ll probably need a pretty beefy machine depending on how many publishers and subscribers you’re configuring the tester with

you’ll probably need to split your test across a few machines

you can also launch it with something like:

```#!/bin/bash

for i in {1..1000}
do
   livekit-load-tester --url your_url --room "room$i" --publishers 10 --subscribers 10 &amp;
   sleep 1m
done```

Here is the result of the load test.
Most of the summary looks like below:
```Summary | Tester | Tracks | Bitrate                  | Latency  | Total Dropped
        | Sub 0  | 5/8    | 145.1kbps                | 60.759ms | 0 (0%)
        | Sub 1  | 0/8    | NaNmbps                  |  -       |  - 
        | Sub 2  | 0/8    | NaNmbps                  |  -       |  - 
        | Sub 3  | 0/8    | NaNmbps                  |  -       |  - 
        | Total  | 5/32   | 145.1kbps (36.3kbps avg) | 60.759ms | 0 (0%)```
which indicate tracks are missing.

The test is run in my local PC with
• 50 rooms
• 4 pubs, 4 subs in each room
• 20kps audio only
My server is an EKS with 3 nodes of  m5n.xlarge. The CPU and memory is lightly used and don’t exceed 20%,
the network is boosted maximum around 50mb/s.

I don’t know if this is because I tested on a local machine. Worried about real-life scene when accesses boost.

you probably don't want to run the tester from your dev machine. you'll saturate the bandwidth/processors on it long before you'd be able to saturate the server's

I see, it would be quite difficult to test in a real-life close situation.
I thought about using testRTC, but didn’t figure out how to.

It would be very kind of you to have a brief instruction to help us test the server using testRTC.

I think it's pretty straight forward.. you can copy the load tester to a linux VM on the server side and run it there

if you’re comfortable with Kubernetes, here’s a Deployment template that you could use. I’m planning on committing this to the `livekit-cli` repo soon.
```apiVersion: apps/v1
kind: Deployment
metadata:
  name: livekit-load-tester
spec:
  replicas: 2
  selector:
    matchLabels:
      app: livekit-load-tester
  template:
    metadata:
      labels:
        app: livekit-load-tester
    spec:
      containers:
        - name: container
          image: livekit/load-tester:latest
          args:
            - --url=wss://
            - --api-key=
            - --api-secret=
            - --subscribers=200
            - --num-per-second=1
      restartPolicy: Always
      affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - livekit-load-tester
                topologyKey: "<http://kubernetes.io/hostname|kubernetes.io/hostname>"```

I deployed the livekit server using namespace `kube-system`.
And I add a PDB for it using the following command.
`kubectl create poddisruptionbudget livekit-pdb --namespace=kube-system --selector app=livekit-server --max-unavailable 1`

I also annotated the pod I want to delete.
`kubectl annotate -n kube-system pods livekit-server-k8s-6cd9d76647-9pbsq "<http://cluster-autoscaler.kubernetes.io/safe-to-evict|cluster-autoscaler.kubernetes.io/safe-to-evict>"="true"`

Do you have live sessions on the server when scaling down? We prevent TERM signals from killing active user sessions

No, I don’t have.
I think I might have solved this problem.

I set the number of replicas to 4 (and I hoped the autoscaler to scale down below 4), my misunderstand of k8s.

To be sure, the replicas set by `kubectl scale --replicas=3` is the minimum number of nodes for k8s.
Is the above correct ?

image.png

Screen Shot 2022-04-14 at 5.30.51 PM.png

<@U036D0UCRBJ> that's really odd. which browser are you using? I'm able to publish just fine to 0.15.6.

FWIW, I've tested on the latest version of Chrome/Firefox/Safari

I see it’s really odd) I’m using Chrome 99 and Safari 15.4 on Mac.
Ok, thank you. I just wanted to know if it’s a some already known issue.

I’ll try different networks and clean server installation. ILYK

Hi <@U01KZ61GV5Z> and folks!

I did an investigation and it’s really  :exploding_head:. If anyone knows what can I check else, please help me)

The “breaking” change is in this commit in pion/dtls —
<https://github.com/pion/dtls/commit/bfbe15ef34f393f864a8caacc26f5199eef8b236>. This change increased DTLS flight1 handshake UDP packet from 131 bytes to 137 bytes (6 CipherSuiteIDs instead of 3 before).

This packet looks like (nothing special):
```10:56:25.282656 IP (tos 0x0, ttl 64, id 15428, offset 0, flags [DF], proto UDP (17), length 165)
  [hidden server].60855 &gt; [hidden client].63868: [bad udp cksum 0xec5a -&gt; 0x0618!] UDP, length 137
        0x0000: 4500 00a5 3c44 4000 4011 124c 0a80 000e E...&lt;D@.@..L....
        0x0010: 6dc2 7368 edb7 f97c 0091 ec5a 16fe fd00 m.sh...|...Z....
        0x0020: 0000 0000 0000 0000 7c01 0000 7000 0000 ........|...p...
        0x0030: 0000 0000 70fe fd62 6138 59eb da6e bf2b ....p..ba8Y..n.+
        0x0040: c968 5fe5 617d 26a7 c2bd 94e7 977c 8e4a .h_.a}&amp;......|.J
        0x0050: f66c 2c92 a88e 1a00 0000 0cc0 2bc0 2fc0 .l,.........+./.
        0x0060: 14c0 14c0 14c0 1401 0000 3a00 0d00 1000 ..........:.....
        0x0070: 0e04 0305 0306 0304 0105 0106 0108 07ff ................
        0x0080: 0100 0100 000a 0008 0006 001d 0017 0018 ................
        0x0090: 000b 0002 0100 000e 0007 0004 0007 0001 ................
        0x00a0: 0000 1700 00               ```
(tcpdump from server)

And the trouble is that this and only this packet is dropped by the network every time. I checked it in tcpdump on client machine. Pion/dtls uses retransmission flight1 packet several times, but none of these packets are received (if size is “wrong”).

When I modified CipherSuiteIDs in flight1 packet by hardcoding 5 (instead of 6) int16 in <https://github.com/pion/dtls/blob/0fbf8d103737d247a36377c739eb34bea6d790a6/flight1handler.go#L130|flight1handler.go>, packet size got smaller (135 bytes) and everything started working well.
```CipherSuiteIDs:    []uint16{49195, 49199, 49172, 49172, 49172}```
I also checked different packet sizes:
• 3 CipherSuiteIDs (131 bytes) - works (as with pion/dtls v2.1.2)
• 4 CipherSuiteIDs (133 bytes) - doesn’t work 
• 5 CipherSuiteIDs (135 bytes) - works
• 6 CipherSuiteIDs (137 bytes) - doesn’t work (as with pion/dtls v2.1.3)
• 7 CipherSuiteIDs (139 bytes) - works
• 8 CipherSuiteIDs (141 bytes) - works
I can’t figure out which network conditions are causing it. I know I have NAT on my home router. Thats why I tried to check UDP hole punching manually and it works. I even checked transmission of exactly this packet through the hole, and it worked! But it doesn’t work in dynamic WebRTC process.

I suppose there are some issue in my network, but it would be nice if the LiveKit JS SDK at least showed an error.

whoa.. this is really strange... It's not clear to me why the packet would be dropped by your network. To be clear, the server sent this packet but was dropped by something along the way to the client?

Yes, the server sent this packet and I saw it in tcpdump on server shell. 

But the client didn't receive this packet. I can't find it by packet Id or by payload in tcpdump on client machine shell. But I see another neighborhood packets from the sever. 

Hi! I'm finally managed it. 

I should confess I’m temporary in Russia now. And the reason is in fucking russian government censorship :rage:. Almost every russian internet provider was forced to setup some filtering device (which can be controlled only by government). And now (for several months) it tries to block Tor Snowflake, which uses Pion/tls. So LiveKit’s and Snowflake’s DTLS handshake packets have similar signatures.

I changed dtls packed size, it works now and ok for me . 

Thank you for your attention) 

wow! I definitely didn't expect that. I'm glad you got around it.

Livekit -&gt; Push Gateway &lt;- Prometheus.

Libekit -&gt; NodeExporter &lt;- Prometheus ?

helm install livekit livekit/livekit-server -f values.yaml.
This is my values.yaml file.

If you could setup for prometheus, advice me...  :pray:

Hey Daichi, you need to set up Prometheus separately from LiveKit and point to the livekit Prometheus port to scrape it's metrics. This is how the Prometheus system works, where it goes out to fetch metrics from each app that exposes metrics via an endpoint. 

Yes, I set up prometheus separately.

It means, need to change prometheus.yaml for fetching metrics of Livekit?

I couldn’t find documents for prometheus.yaml.
Could you show me sample.yaml??

Prometheus and Livekit on docker, it works.
Metrics path is ‘/’.
```  - job_name: 'livekit'
    scrape_interval: 5s
    metrics_path: /
    static_configs:
      - targets: ['livekit:6789']```
I think config-sample.yaml is wrong.
<https://github.com/livekit/livekit-server/blob/7a3e446a65e0ffea3591056a3e85e77367e0781b/config-sample.yaml#L72>

And server.go.
```Handler: promhttp.Handler(),```
<https://github.com/livekit/livekit-server/blob/7a3e446a65e0ffea3591056a3e85e77367e0781b/pkg/service/server.go#L107>

I did it !

Helm charts doesn’t work.

1. kubectl edit deployment/livekit 
```        - containerPort: 6789
          hostPort: 6789
          name: metrics
          protocol: TCP```
2. kubectl edit service/livekit
```  - name: metrics
    nodePort: 31789
    port: 6789
    protocol: TCP
    targetPort: 6789```
3. configmap for prometheus
```    - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      job_name: 'livekit'
      metrics_path: /
      static_configs:
        - targets: ['livekit-server-k8s.livekit.svc:6789']```
Then prometheus starts to collect metrics.

You can’t monitor prometheus on k8s without this.

If I have time, I would pull req.

oh I see what you mean, the helm chart currently doesn't expose the port even if it's enabled.

docker-compose.yaml
```version: "3.9"
services:
  caddy:
    image: lucaslorentz/caddy-docker-proxy:2.3-alpine
    ports:
      - 80:80
      - 443:443
    environment:
      - CADDY_INGRESS_NETWORKS=caddy
    networks:
      - caddy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./data/caddy:/data
      - /etc/localtime:/etc/localtime
    restart: unless-stopped
  livekit:
    image: livekit/livekit-server:latest
    command: --config /etc/livekit.yaml
    restart: unless-stopped
    network_mode: "default"
    container_name: livekit_server
    volumes:
      - ./livekit.yaml:/etc/livekit.yaml
      - /etc/localtime:/etc/localtime
    networks:
      - caddy
    expose:
      - 7880
      - 6789
      - 8000-10000
    ports:
      - "7880:7880"
      - "6789:6789"
    depends_on:
      - redis
    labels:
      caddy: app.localhost
      caddy.reverse_proxy: "{{upstreams 7880 }}"
      caddy.tls: "internal"
    environment:
      - VIRTUAL_HOST=app.localhost
  redis:
    image: redis:6-alpine
    ports:
      - "6379:6379"
    volumes:
      - "./data/redis:/data"
      - /etc/localtime:/etc/localtime
    networks:
      - caddy

networks:
  caddy:
    external: true```
livekit.yaml
```port: 7880
rtc:
    tcp_port: 7881
    port_range_start: 50000
    port_range_end: 60000
    use_external_ip: true
redis:
    address: redis:6379
    username: ''
    password: ''
    db: 0
turn:
    enabled: true
    domain: app.localhost
    cert_file: ''
    key_file: ''
    tls_port: 5349
    udp_port: 443
    external_tls: true
keys:
    YOURAPI: YOURAPIKEY
prometheus_port: 6789```

This is on a build from master from a few days ago. I'm pushing a new version live now from the current mainline.

oh, actually it seems to be intermittent. every so often I'll get a value, but most of the time they are 0. Same with current mainline.

My bigger concern is if that value isn't accurate, it wouldn't be valid to use it as a "sort by" metric.

interesting, we haven't seen this during node assignment, but will check it out. is the repro just with `list-nodes` ?

are you passing in your configuration file btw?without it it'll not be reading the right values from redis

i.e. `livekit-server --config config.yaml list-nodes`

<@U01KZ61GV5Z> Yeah, I am. And I'm getting valid values for everything else, just those two rates seem to be off. One sec, let me sanitize some output and share an example.

ah gotcha.. ok I see what you are seeing as well.

will file a bug, not sure if it's a display issue vs data collection issue

the other fields don't appear to be updating on the same cycle either..

other fields being the ones relating to bytes/packet counts

Based on my quick glance at the code, I'm betting if it isn't showing properly there it probably isn't valid to use it during node selection either :grimacing:

I bet it's because some of the values are updated from a thread without proper locking

FWIW, Prometheus metrics seem to be valid. `livekit_packet_bytes` matches `node_network_transmit_bytes_total` from node exporter, as expected.

filed: <https://github.com/livekit/livekit-server/issues/617>

I commented on the ticket, but this issue still seems to be happening on the current mainline branch. Less frequent, but still showing zeros quite often.

you are right.. this still happens whenever there's packet loss

fixed in a <https://github.com/livekit/livekit-server/pull/639|new PR>

Great! Sorry, I had missed that one. Thanks.

hah you didn't. I just pushed it after your report

Ah! That does remind me that I should check in-flight PRs before doing a build though. :slightly_smiling_face:

the issue is you are not managing state changes when the track is subscribed. To do so you would want to listen to `TrackSubscribed` and then trigger state changes to cause the `useEffect` to fire.

honestly it'd be much easier to use our react-sdk.. the `useParticipant` hook handles this for you

Thanks David, I had a look at the react sdk but had no time to integrate it, will do it next week mostly :+1:

Hey <@U039W6Y4BK4>, well the source code is available through: <https://github.com/livekit/livekit-react/tree/master/example>. Try using the code on your server and bring up any code change that you want/need.

Which platform/SDK version are you using?

also is `adaptiveStream` enabled? Muted could be due to adaptiveStream pausing the stream due to visibility detection.

this is reproducible with an example app?

I haven’t tried the example app but I’ll try it, and this is on JS SDK, and yes, `adaptiveStream` is enabled

sounds good.. is your app using React or some sort? I wonder if it's state issues as you are attaching/detaching from tracks.

When using adaptiveStream, it's important to use the provided `.attach()` to bind to Video elements. Otherwise it will not know the stream is still being used.

the app is React but we’re not really using the React SDK, we’re using the vanilla JS library, and yes, we use `attach` to playback the video feeds

setting `adaptiveStream` to `false` does solve the issue

I think it probably has to do with state management and adaptive stream. unfortunately it's kind of tricky with React. Our SDK doesn't have this issue so that's worth taking a look

I’ll check it out, but I do see `muted` being set to `true` when someone leaves a group call and I’m not calling `setCameraEnabled` with `false` anywhere 

`setCameraEnabled` happens on the publisher side. the state issues are likely happening on the subscriber side itself. `muted` on the MediaStreamTrack is set to true when the feed runs dry.. it doesn't indicate an explicit user action at all. I think what's happening is:

1. adaptiveStream disabled the stream because it thinks no elements are attached to it
2. server stopped sending data for that track
3. `muted` became true
The key question is why does 1 happen at all.

ah I see.. expiring credentials isn't something that we support right now. I think it'll probably have to plug into a different system than a simple config file. perhaps providing the credentials from a DB or something

Thanks for the reply david. Is there a workaround you can suggest until livekit supports expiring credentials?

Unfortunately I don't.. does the embedded TURN server not work for your needs?

I haven't tried that yet but I'll give that a try today. Thanks.

Hey David, I tried integrating the embedded TURN server but it doesn't seem to work. Not sure what I am missing here.

My setup has a livekit running in a docker container on an ubuntu machine on premise that is behind a reverse proxy through Cloudflare argo tunnels.

I've proxied everything TCP on 5349 to the device and I can confirm that the proxy is correctly forwarding all traffic. Here's my config
```port: 2053
rtc:
  tcp_port: 7881
  udp_port: 7882
  use_external_ip: yes
  stun_servers:
  - <http://global.stun.twilio.com:3478|global.stun.twilio.com:3478>
  congestion_control:
    enabled: true
    allow_pause: false
logging:
  json: false
  level: debug
turn:
 enabled: true
 tls_port: 5349
 external_tls: yes
 domain: <http://turn.device.abc.com|turn.device.abc.com>```
I do see livekitserver starting the turn server and listening on the right port
```2022-04-20T10:52:37.497Z	INFO	livekit	logger/logger.go:26	Starting TURN server	{"turn.portTLS": 5349, "turn.externalTLS": true}

$ netstat -tulpn  | grep 5349
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6       0      0 :::5349                 :::*                    LISTEN      -```
And I can also see that the livekit server is able to get and send candidates but the connection never gets established over TURN. It works perfectly on local network/STUN

To workaround this, I created a new TURN server which has fixed credentials. This works perfectly with native pion implementation but doesn't seem to be working when I add this as a TURN server on livekit.
```port: 2053
rtc:
  tcp_port: 7881
  udp_port: 7882
  use_external_ip: "yes"
  stun_servers:
  - <http://global.stun.twilio.com:3478|global.stun.twilio.com:3478>
  turn_servers:
  - host: <http://turn.spot.ai|turn.spot.ai>
    port: 50000
    protocol: udp
    username: 09d76a934b40f03ba140f8b7731b6fcaacbd5512174b4bda8fdc81d4f5fb5720a3fd12023cf3620879ee4622ea5b697e80e34eb822ef751b2f7f966e9d9bdc29ba361e
    credential: 2eda6fa8076a86b777493379fb765d9f0b97adf57c810df5015e2b02353a1fe0```


for the embedded server, ensure the reverse proxy is listening on `:443` this is currently a fixed port that we advertise to maximize firewall compatibility.

the external TURN configuration looks right to me. though your `use_external_ip` value should be set to `true`

The best way to accomplish this is to give users very short tokens (say 1min). Then use <https://docs.livekit.io/guides/server-api#removeparticipant|RemoveParticipant> server API to remove the user from the room.

They will not be able to connect again with the original token.

Internally, LiveKit will re-issue tokens to the connected client in case it becomes interrupted and can then reconnect. So issuing tokens with short TTL would not impact existing clients

Screenshot_2022-04-20-21-21-12-296_com.android.chrome.jpg

go with GCP, linode doesn’t support UDP

Ok thanks ,

Can you answer about also what is thos value

values are here <https://github.com/livekit/livekit-helm/blob/master/examples/server-gke.yaml>

For example, there are actually more Video and Audio tracks published during the second screenshot than the first, but bandwidth usage on the first is about 8x. Theoretically, stream sizes should be the same between these.

I don't expect anyone from LiveKit to dig in on this - I'm still trying to find info on my end and see if I can repro it happening. But figured I'd check to see if others have had it happen and have more insight :slightly_smiling_face:

Hey <@U02C10ZEP8R>, I believe this is a bug that we had <https://github.com/livekit/livekit-server/pull/620|just fixed> (but have yet to release).

In short, if the publisher has experienced a reconnection (that is typically transparent to the user), dynacast settings would become lost and starts publishing all layers.

Oh... That actually seems to fit what I was seeing in my logs. Excellent! I'm already running a self-build from mainline so I can have my sort-by change in place. I'll cut a new build and deploy that ASAP :slightly_smiling_face:

nice! lemme know how that ends up working for you!

There may have many reason for this error. You can check more details from `about:webrtc` . If this is only happening with firefox then you can try to setup a turn server. It seems to me that firefox can't established connection.

aboutWebrtc.html

This is my `about:webrtc` page. I couldn't understand what went wrong here. By the way, how can I start a turn server and use it alongside my livekit deployment?

As far I know apache isn't proxy server. You'll require a proxy server or load balancer to handle requests, if you want to use same server for both website &amp; livekit. Or you can follow this article: <https://docs.livekit.io/deploy/vm> &amp; setup a new server for livekit itself only. If you just need a ready to use web conferencing system using livekit then you can have a look: <https://www.plugnmeet.org/>. It is based on livekit.

is the track really unpublished? (i.e. does it have a new track sid)?
your publisher should have logs if that's actually happening. there is nothing in livekit that would unpublish track on your behalf.

having logs here would be helpful to pinpoint what had happeneed

We haven’t done logging for the track lifecycle. That’ll be for next sprint. But thanks for the pointer, I highly doubt LiveKit does it. I’ll collect more data and revisit this if I can’t figure it out. Thanks <@U01KZ61GV5Z> !