LiveKit Community

when it's muted, we pause the stream to all subscribers. it doesn't use any bandwidth/incur load at that point

So for TURN UDP, you shouldn't have any conflicts on port 443 because NGINX SSL termination uses TCP on port 443. TURN TLS however runs on TCP and that's where you will have a conflict. In this case, you can use an external load balancer. You should also be able to use a different port, but it may not be as effective at getting around firewalls.

We use HAProxy as a load balancer. For low traffic systems (dev, tests, QA) where the deployment of multiple machines and external load balancers would be overkill, we configure HAProxy in a kind of loopback to itself. This allows us to receive all TCP traffic on port 443, divert TURN requests based on SNI (e.g. <http://turn.example.com|turn.example.com>) to the LiveKit backend without SSL termination and pass all other requests back to HAProxy for another round. On the second pass we terminate the SSL encryption and then pass the requests to either the LiveKit API (7880) for requests to <http://livekit.example.com|livekit.example.com> or to our frontend at <http://example.com|example.com>. So far, this has worked very well for us (in the above-mentioned cases). There is some overhead because of the loopback, but the overall simplicity and portability of the system make it worthwhile to us.

Hi, thanks Luis I am not so familiar with HAProxy... And I don't think I understand your solution fully:sweat:. If this setup creates troubles... What is the easiest way to enable the TURN server according to the LiveKit team? I mean we are reverse proxying all the requests from 443 to 7880 so that LiveKit Server can work so I am just confused in any setup how can we configure 443 reverse proxy or load balancing to 7880 and the tls_port of the TURN server at the same time?

<@U02BBTRGF98> I tried the UDP port there are no errors when starting the server but on <https://livekit.io/connection-test> it fails to connect with TURN, here are the <https://gist.github.com/Thunder80/52c6e0709c2e374985bc3260c1227bb4|logs>. I am not sure what I am doing wrong?

What you are addressing (that both protocols should listen on port 443, which is not usually a thing) is indeed the core of the matter. The problem is, that – even though that's highly unusual – both SHOULD be served on port 443, because a) you really want to serve HTTPS over 443 for all public-facing services (because that's what browsers, users and firewalls expect you to do) and b) TURN TLS over any other port than 443 will create a lot of problems with a lot of firewalls (especially in corporate environments, where TURN is mostly needed in the first place).
I can think of three ways to approach this:
1) Use two different network interfaces for HTTPS and TURN and have both listen on port 443. Point your domains to the separate ip addresses of those interfaces.
2) Use an external load balancer that forwards TURN requests on port 443 to some other port on your local machine.
3) Use an internal load balancer like I described to "share" port 443 between multiple local processes.
If your external load balancer uses only one interface to serve multiple domains (e.g. <http://turn.xyz.com|turn.xyz.com> AND <http://livekit.xyz.com|livekit.xyz.com>) on port 443 it will have to perform SNI just like Option 3 – with the only difference that it's abstracted away for you.. :slightly_smiling_face:

O thanks for the explanation <@U028SSAKLQ1> I think I got you and why everything needs to be on 443 and I will try out the suggestions you mentioned.

In case you want to try out the HAProxy loopback route, here's the relevant snippet of our config. It's slightly stripped down but should work as is if you change your domains and configure the ssl certs in the `global` block:
```frontend ft_ssl
    bind *:443
    mode tcp

    tcp-request inspect-delay 5s
    tcp-request content accept if { req_ssl_hello_type 1 }

    # Pass TURN requests to LiveKit Turn server without SSL termination
    use_backend     bk_livekit_turn  if { req_ssl_sni -i <http://turn.conference-domain.com|turn.conference-domain.com> }
    
    # Indirectly pass all other requests to the next frontend to handle SSL termination
    default_backend bk_tcp_to_https

backend bk_livekit_turn
    mode tcp
    server live_turn_local 127.0.0.1:5349 check

backend bk_tcp_to_https
    mode tcp
    server haproxy-https 127.0.0.1:8443 check send-proxy

frontend ft_https
    mode http

    # HAProxy will take the fitting certificate from the available ones
    bind *:8443 ssl crt /etc/haproxy/cert/ accept-proxy
    
    # Spread the requests between backends
    use_backend     bk_web if { hdr(host) -i <http://conference-domain.com|conference-domain.com> }
    use_backend     bk_livekit_api  if { hdr(host) -i <http://livekit.conference-domain.com|livekit.conference-domain.com> }

backend bk_web
    mode http
    http-request set-header X-Forwarded-For %[src]
   
    # Pass the requests to the local web server for frontend assets and Non-LiveKit API
    server local 127.0.0.1:8080 check

backend bk_livekit_api
    mode http
    http-request set-header X-Forwarded-For %[src]
    
    # Pass the requests to the LiveKit HTTP API
    server local 127.0.0.1:7880 check```

so you're using the requested Host name to direct the request to either the LiveKit websocket port or TURN TCP/TLS port on the same host. nice solution here <@U028SSAKLQ1>.

I may have to adapt this solution for our cloud-init deployment method

Please let me know if I can help with that

I think we can use NGINX <https://www.digitalocean.com/community/tutorials/how-to-set-up-nginx-server-blocks-virtual-hosts-on-ubuntu-14-04-lts|Server Blocks> to achieve the same result as your HAProxy set up without too many changes. It does add a little more complexity for the user, but maybe it could be an optional step.

That's a solution we were initially exploring, too. IIRC you have to run NGINX as root and configure the firewall to be able to use transparent proxy_bind for client IP "passthrough". Is that what you're referring to with the added complexity? There might be a way to use proxy_protocol with yet another loopback (or by handling proxy_protocol on the TURN server) but I can't really see how that would be going to reduce complexity.
The decisive point why we sticked to HAProxy though was that we're already using it as an actual (i.e. external) load balancer and I really prefer the simplicity of having as few (different) moving parts as possible.
Still, I'd love to find the best way to tackle this with NGINX. So if you need a rubber duck or want to discuss technical details, count me in!

I appreciate it! I was more thinking complexity from a user’s point of view in setting up a LiveKit server that requires two different domains and SSL certs. Maybe some don’t care about TURN and would want it left off? the goal here is to make it as easy as possible to run and maintain a LiveKit server, and if HAProxy makes it easy for the user, then maybe it is worth the switch.

I see. Making TURN optional could work. Maybe just turn the feature off if the user does not provide a TURN_HOSTNAME config (or similar)? The deployment could also be further simplified by (also optionally) automating certificate issuance and renewal through an ACME provider.

At the end of the day, when the whole process is automated through a cloud-init script, it won‘t really matter if HAP or NGINX are deployed, right?

automatic cert issuance would be sweet too!

As for usage of server blocks on nginx level: we actually tried it on our side and it even worked, but then I realised that it can’t work because the server block examples from the article above rely on `location` and `server_name`, however if my understanding is correct, both of those only have any meaning when we’re talking about HTTP(s) traffic. The TURN allocation requests are not an HTTP traffic, it’s just some binary protocol, meaning that it can’t work like this.
It seems like in order to make it work, the one have to use the SNI on nginx level to route the traffic between web, livekit-server and TURN.

I could not spot this issue for a long time, because the connectivity test for TURN was always green. This seems like a bug. Perhaps the TURN test shows the checkmark even if TURN worked over UDP without testing all the stuff like TURN-over-TLS.

<@U028SSAKLQ1> Did you test the TCP TURN forwarding with UDP TURN disabled using the connection tester? I can't seem to get it to work using your example.

If I put the API on a different port and put livekit's TURN directly on TCP:443, I can get the connection tester to report good in Firefox but not Chrome. If I put TURN behind haproxy I can't get it to report good with either.

<@U02C10ZEP8R> you are right! I have originally tested the clients with UDP blocked – but on the firewall in front of the test clients. It looks like this firewall was incorrectly configured and that's why we could not see that TURN TCP was indeed not working. I have just rerun the connection test with TURN UDP blocked on the server and the test failed. Thank you so much for pointing this out!
Unfortunately I can still see no problem with the config above. When debugging in HAP, it looks like all the traffic is correctly routed to TURN TCP. <@U02BBTRGF98> do you have an idea how to trace/debug this on LK's end?

What <@U02AQRN3DKK> said makes sense. TURN on TCP is not HTTP, thus it won’t have any `HOST` header information for HAProxy/Nginx to be able to route requests for a specific hostname. As it is, I’m not sure there’s much we can do from LK’s end on this, but there may be an option with some code changes:
Theoretically speaking, you could have two separate network interfaces on your host, where it has two IP addresses. There would need to be some changes on LK to allow you to specify which interface TURN TCP/TLS listens on. Then Nginx/HAProxy would listen on the other interface so the two don’t conflict. I’m not sure exactly how big of a change this would be, or what logic changes would be needed for this to work.

It may also be possible to configure Nginx/HAProxy to route all non-http traffic to a specific port. Websocket being HTTP and TURN TCP/TLS being non-HTTP.

What we’re currently trying to do in our case to workaround this issue is to rely on SNI on nginx level to route different types of traffic to `livekit-server`/`turn`, i.e. we have 2 hostnames pointing to the same machine (CNAME) which allows us to make use of the SNI. Not sure if it would work, but that’s something we’re experimenting with :slightly_smiling_face:

This PR might help too: <https://github.com/livekit/livekit-server/pull/168>
Allowing Nginx to terminate TURN TLS just might allow hostname routing to work.

<@U02C10ZEP8R> <@U02BBTRGF98> false alarm! After an embarrassingly long debugging session, I've just found the reason why the connection test didn't work yesterday: We haven't fully automated the certificate issuance in staging (where I tested) and the certificate for the LK TURN server has expired a week ago :face_palm: . I can confirm that our HAP setup works with TURN TCP – tested in two ways: 1) denying the TURN UDP port on the firewall and 2) setting `turn.udp_port: 0`  in the LK config.

<@U02BBTRGF98>: SNI is an extension of TLS and does not rely on the HTTP `Host` header but the `server_name` parameter of the ClientHello message. AFAIK that's why TCP-routing in HAProxy works for TLS connections with non-HTTP protocols.

I really look forward to pull request <https://github.com/livekit/livekit-server/pull/168|168> as it makes the whole setup a lot simpler. I'll post an updated HAP config with full TLS termination when I had the chance to properly test it. This will also bring us a big step closer to certificate automation, as we won't have to restart multiple services after cert rotation.

Amazing! Glad to hear there isn’t anything more complicated that needs to be done. So it sounds like SNI does still work from the HAProxy layer, even when SSL termination happens on your LK server?

Using that "Turn with external ssl termination" change, this is the simplified version of an HAProxy config that works:

```frontend ft_ssl
  mode tcp
  bind *:443 ssl crt /etc/haproxy/cert/

  tcp-request inspect-delay 5s
  tcp-request content accept if { req_ssl_hello_type 1 }

  use_backend     bk_livekit_turn  if { ssl_fc_sni -i <http://turn.livekit.example.com|turn.livekit.example.com> }
  default_backend bk_livekit_api


backend bk_livekit_turn
  mode tcp

  # Pass the request to LiveKit TURN
  server livekit_turn_local 127.0.0.1:5349 check


backend bk_livekit_api
  mode http
  http-request set-header X-Forwarded-For %[src]

  # Pass the requests to the LiveKit HTTP API
  server livekit_api_local 127.0.0.1:7880 check```
Then in `/etc/haproxy/cert/` I have certs for `<http://turn.livekit.example.com|turn.livekit.example.com>` and `<http://api.livekit.example.com|api.livekit.example.com>`.  This could easily be extended to also have a web page in front of it.

One issue I'm seeing is that Firefox fails the LiveKit connection tester on the TURN step. However, this was also true when terminating SSL at the LK layer and using HAProxy with <@U028SSAKLQ1>'s previous HAProxy example. Both work fine in chromium browsers. It seems there's something with the HAProxy front end that is making Firefox not like it. I thought that was the reason for the `inspect-delay`  and `accept if` lines, but I've tried that and several other changes and have never been able to make Firefox happy except when I have it go straight to the LK server with no proxy.

It's weird, but looking at the connection log in FF about:webrtc I cannot find any attempt of the browser even trying to connect to TURN over 443/TCP. <@U02C10ZEP8R> unfortunately I have no easy way to set up an instance of LK without a proxy in front of it right now. Do you think you could maybe catch the logs in about:webrtc for both a HA-proxied and a direct connection (with UDP blocked to force TCP)? Being able to compare the two would be quite helpful in understanding what FF is doing differently.

Yeah, I still have my test instances running. I can flip it back to that mode for a bit.

Perfect! Do you want to capture the logs or can I help with that?

<@U02BBTRGF98> yes, the SNI part is working as intended – I think the only remaining question is what‘s going on with FF+HAP. 

I'll take a stab at it in a bit. I'll let you know if I have trouble getting the data and am happy to give you access to to the test box. It's just a linode instance and separate from everything else.

Thanks to <@U02C10ZEP8R>'s logs I could confirm that the first suspicious log entry I've found yesterday actually doesn't occur in Firefox when connecting directly. Compare this (direct connection):
```(turn/INFO) TURN(relay(IP4:10.0.42.6:0/TLS|IP4:0.0.0.0:443/TLS)): Succesfully allocated addr IP4:[REDACTED_TURN_SERVER_IP]:11217/UDP lifetime=600```
with this (HAProxied connection):
```(stun/WARNING) STUN-CLIENT(relay(IP4:10.0.42.6:0/TLS|IP4:0.0.0.0:443/TLS)::TURN): XOR-MAPPED-ADDRESS is bogus
(stun/WARNING) STUN-CLIENT(relay(IP4:10.0.42.6:0/TLS|IP4:0.0.0.0:443/TLS)::TURN): Error processing response: Invalid data, stun error code 0.```
It looks like there is a <https://community.jitsi.org/t/coturn-chronicles/73099/22|known issue> with Firefox and proxied TURN. Interestingly it's a JITSI/coturn issue, so probably nothing Pion-specific – and they used NGINX as a reverse proxy, so probably also nothing HAP-specific. This makes me wonder if maybe the official Kubernetes deployment (with NGING for k8s-Ingress as a reverse proxy) suffers from the same (not-so-easily discoverable) issue. <@U02BBTRGF98> do you have a K8S-deployment running that could be tested the same way? Because if that setup works, we can start comparing the differences between the two proxy setups.

I don’t have k8s deployment that I can provide as our internal clusters aren’t a standard setup, but I can spin up a fresh cluster with the latest release for you if you like. Though, I’m pretty sure the issue exists with the k8s deployment as well, as I was not able to connect with TURN on Firefox last time I tried. I just assumed it was a Firefox limitation.

<@U02BBTRGF98> well, if the k8s setup (probably) won‘t work anyway, I think we can save the time for now. I‘ve been trying to make it work using proxy protocol with a modified version of LK, but so far no luck. I would love to take another stab at it, but unfortunately will be afk for the next 3 weeks. I‘ll certainly be back to this thread and hope we will find a solution to make this work properly in FF.

That would be amazing! Have a fantastic break in the meantime. 

hey guys, I have tested nginx as lb and both ff and chrome is ok either terminal tls in nginx or lk, and watch packets through wireshark that all traffic is over tls/443. Does this problem is limited to ha? I'm not using k8s, just run nginx and lkserver on one machine

that's weird publish pc's ice connection will disconnect after 4s since begin, and subscriber's pc is ok so that ff can get chrome's media stream but chrome can't get ff's. log shows ice timeout

<@U02GP9AENTH> this is TURN/TLS and Nginx?

webrtc internals shows a relay-tls candidate

In my case, the publish ice interrupt after connection is because there is a bug in pion/turn, open a pr to fix this Fix process partial stun message by cnderrauber · Pull Request #237 · pion/turn (<http://github.com|github.com><https://github.com/pion/turn/pull/237|)>
with this pr, test on my machine lk can work with turn behind nginx both ff and chrome, and either terminate tls on nginx or lk. also without check with k8s

great work <@U02GP9AENTH> :rocket: we'll cut another release shortly.

Nice. I'll have to test that out with my haproxy setup and see if it fixes it. Probably won't get to that today, though.

So to sum up, how to set up turn server, while having the error that port 443 is already used by nginx?

Is there a recommended way to debug the webhook feature? We added the webhook config based on the docs, but haven't been able to find anything in the logs

can you point me to the confusing spot in the docs? I looked but wasn't seeing it.

Might be a good idea to have the code sample areas for key/secret substitutions cause compile/runtime errors :thinking_face: 

<https://docs.livekit.io/guides/webhooks#receiving-webhooks>

```webook:
  api_key: 'api-key-to-sign-with'
  urls:
    - '<https://yourhost>'```

<https://docs.livekit.io/guides/server-api>

that would be something you could implement in your own app. i.e. delay joining until approval.

currently the webhook is posted before the participant is added to the db - we can update that for the next release to make sure the participant is saved to the db first

you can use docker-compose.yml and pass the LIVEKIT_CONFIG like this:
```services:
  livekit:
    image: livekit/livekit-server
    environment:
      LIVEKIT_CONFIG: |
            port: ${PORT}
            log_level: ${LOG_LEVEL}
            rtc:
              tcp_port: ${TCP_PORT}
              port_range_start: ${PORT_RANGE_START}
              port_range_end: ${PORT_RANGE_END}```
and so on, then put all those vars in .env file

If this is reasonable, I can create a github issue for it as an enhancement request. I wanted to check to see if it would even be in scope for livekit, though.

hey <@U02C10ZEP8R>, I think that's a bit of out of scope for the moment. doing multi-tenant right will be quite an endeavor. For the near term we will focus on making sure we have the best SFU and client capabilities for a single tenant.

That's what I figured :slightly_smiling_face: My use case is a little unique. I can probably find some workarounds.

I think the issue is connecting to
```ws_url: <ws://192.168.26.55:7880>```
from inside the docker container

I wondered about that.  I connected in, installed `ping` and was able to ping it.

I don’t have a solution yet, but I also haven’t been able to record a room running on `localhost`

```% docker exec -it demo bash
root@49c208c88190:/# apt update
...
root@49c208c88190:/# apt install -y iputils-ping
...
root@49c208c88190:/# ping 192.168.26.55
PING 192.168.26.55 <tel:(192.168.26.55|(192.168.26.55>) 56(84) bytes of data.
64 bytes from 192.168.26.55: icmp_seq=1 ttl=37 time=8.52 ms
64 bytes from 192.168.26.55: icmp_seq=2 ttl=37 time=7.76 ms
64 bytes from 192.168.26.55: icmp_seq=3 ttl=37 time=22.5 ms
64 bytes from 192.168.26.55: icmp_seq=4 ttl=37 time=161 ms
64 bytes from 192.168.26.55: icmp_seq=5 ttl=37 time=2.33 ms
^C
--- 192.168.26.55 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 2.330/40.466/161.190/60.729 ms```

I believe it’s because the templates are hosted at <https://recorder.livekit.io/>

and it has problems connecting from a secure site to an insecure websocket

Hmm...I can secure it.  In fact, my proper setup is using signed certs behind nginx as a reverse proxy, but I bypassed all that in the troubleshooting.

when you use a template, under the hood it just creates the url <https://recorder.livekit.io/#/{template-name}?url={ws_url}&amp;token={token}> and opens it in chrome

try securing it and let me know if you still have issues getting it to connect

Yikes...so it expects to access the VM from the outside world?

yeah, if you want to keep this all internal, the alternative would be to host the recorder site yourself (found at <https://github.com/livekit/livekit-recorder/tree/main/web>)

...and I should explain that my certs rely on a privately signed hierarchy...so I suspect I'd need to do something to the trust store in the docker container.

Screen Shot 2021-11-22 at 10.39.39 AM.png

ah, yeah I don’t believe self-signed certs will work, as they aren’t trusted by chrome

Makes sense (a problem for later then).  May I ask what you mean by "host the recorder site yourself"?  I'm running the container on my own machine - will eventually go on another VM.

Same behaviour when I go through the reverse-proxy, of course.
```% docker run --rm --name demo \
    -e LIVEKIT_RECORDER_CONFIG="$(cat config.yaml)" \
    -e RECORDING_REQUEST="$(cat request.json)" \
    -v ~/src/livekit-recorder/recordings:/out livekit/livekit-recorder
+ rm -rf /var/run/pulse /var/lib/pulse /root/.config/pulse
+ pulseaudio -D --verbose --exit-idle-time=-1 --system --disallow-exit
W: [pulseaudio] main.c: Running in system mode, but --disallow-module-loading not set.
N: [pulseaudio] main.c: Running in system mode, forcibly disabling SHM mode.
I: [pulseaudio] main.c: Daemon startup successful.
+ pactl load-module module-null-sink sink_name=grab sink_properties=device.description=monitorOUT
10
+ pid=0
+ trap 'kill ${!}; term_handler' SIGTERM
+ pid=14
+ wait 14
+ XDG_RUNTIME_DIR=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/root/.cache/xdgr
+ ./livekit-recorder

(gst-plugin-scanner:207): GStreamer-WARNING **: 18:38:40.987: Failed to load plugin '/usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstvaapi.so': libva-wayland.so.2: cannot open shared object file: No such file or directory
0:00:00.090829675   207 0x55e314089240 ERROR            msdkcontext gstmsdkcontext.c:140:gst_msdk_context_use_vaapi: Couldn't find a drm device node to open```
and then nothing else.

I've read that last error is to be ignored - nothing else happens in the log (nor in the server log).  So, I'm guessing it's a fundamental failure to connect.

I _think_ you could host the recording site locally, and instead of using the template, use a url to `localhost:port/#/{template-name}?url={ws_url}&amp;token={token}` . I haven’t tried doing this to connect to a local `ws_url` though

the wayland and vaapi errors are because we have an extra dependency that I eventually need to remove

Yeah, saw the dependency elsewhere in the slack.  Okay - so we're thinking run the URL test, but craft the URL.  Two questions:
1. I presume this means I need to generate my own token (not a problem)
2. I presume recorder is running a chrome instance in there that it'll use (I think I read elsewhere it was)

yes, you can generate a recorder token using livekit-server (or in code)
```./bin/livekit-server --key-file keys.yaml create-join-token --room roomName --identity recorder --recorder true```
recorder is running a chrome instance
port would be whatever port you host it at, I think it’s usually 3000

Ah, do I need to be running my own client?  Does `livekit-recorder` not bring one?

This is all just for if you want to host the recorder template yourself

Otherwise, using our templates it'll require signed ssl certs and an outside connection

Okay.  Apologies, I think I've missed the significance of "host the recorder template yourself".  I thought that was just a UI presentation setting - but sounds like there's an agent at play here.

it’s the UI, but it also acts as a client to join the room using the provided (or generated) token

Thanks <@U01UB0LHTTQ>.  I was missing that the template was a running process, apologies for being thick.  ;&gt;.  Getting one running that my recorder can connect to, and I'll try and do it via URL.  Really appreciate your sticking with me!

<@U01UB0LHTTQ> Progress.  I've got the template code running in a local container, and exposed the http port.  By running a version of the URL command, I can see interesting stuff in the logs, I see the recorder connected as a participant in the call, and a zero-length file representing the room recording.  So, not working, but a big leap ahead.

I had hoped I could revert back to the original docker call (not crafting a URL), and pass in the location for the local template process, but trying:
```{
    "url": "localhost:3003",
    "template": {
        "layout": "speaker-light",
        "room_name": "classroom"
    },
    "filepath": "/out/room-recording.mp4"
}```
only yielded
```proto: (line 3:5): error parsing "template", oneof livekit.StartRecordingRequest.input is already set```
in the logs.

Do you know if it's possible to do what I'm trying, or would I need to try overriding the code?

you can’t use both url and template - your url should be `localhost:3003/#/speaker-light?url={ws_url}&amp;token={token}`

note that `ws_url` there needs to be url encoded

Understood - they're separate tests.  The URL  approach appears to connect to the room and produces a file, but the file is zero-length.  Playing with the `onParticipantDisconnected` function to see if I can get the template to end the recording where there is 1 participant, or even on anyone disconnecting, but no luck.  I'm not sure if any of those console messages matter in URL mode.

_Alternatively_, I was wondering whether I could use the template approach and simply point at my running instance instead of <http://recorder.livekit.io|recorder.livekit.io>.  I think that would be preferable

I could be approaching this all wrong, though.

ah, I think the console messages only matter if it connects to <http://recorder.livekit.io|recorder.livekit.io>

I checked out the `livekit-recorder/web/` code, put it in a container, and launched it with yarn from in there.  Would that be a moral equivalent?

<https://github.com/livekit/livekit-recorder/blob/main/pkg/recorder/request.go#L100>

the template will always direct to <http://recorder.livekit.io|recorder.livekit.io>

Okay...new to `go`...wasn't sure if that was a default rather than a hardcoded value.

<https://github.com/livekit/livekit-recorder/blob/main/pkg/recorder/recorder.go#L65>

<https://github.com/livekit/livekit-recorder/blob/main/pkg/recorder/request.go#L81> was making me wonder if there was a way to pass in a val

the `url` and `template` fields are a `oneof` in the protobuf that translate to `*livekit.StartRecordingRequest_Url`  and `*livekit.StartRecordingRequest_Template`

Perhaps <https://github.com/livekit/livekit-recorder/blob/main/pkg/recorder/recorder.go#L62|this> locks me in place?  Does that suggest it won't _start_ recording unless I'm on the hosted template?

that means it won’t wait for the `START_RECORDING` message, and will instead start as soon as it can

I'm thinking the next thing to try might be to replace the hardcoded recorder.livekit address with my own, build the container and try to run that.  wdyt?

that should work, it would be the same as using `"url": "localhost:3003/#/speaker-light?url={ws_url}&amp;token={token}"`

Okay.  I was hoping that it might help me with the `END_RECORDING` problem, but  yeah - I can see that this whole thing seems separate to that.

I would first try sending an `EndRecordingRequest` to your server

to figure out if the issue is in fact with the `END_RECORDING` message, or the recording itself

Trying a few things, but not getting far.  How do I send an `EndRecordingRequest`?

Okay, a spot of joy!

I was reading the server logs to ensure the recorder was actually joining the room (template method), it looks like it was...but being the only one there (I launched it first) it closed down very quickly.  I did a second test where I joined one participant, then added the recorder, then the second participant, and then my two participants left.  The room then closed, and the recorder cleanly shut down and produced a file I could play back with VLC.

I'll toy with this, and no doubt will have follow ups later.  <@U01UB0LHTTQ> I can't thank you enough for your help today!

you can send the end request with either cli or server sdk
<https://docs.livekit.io/guides/recording/#end-a-recording>

awesome, glad to hear it! It seems that’s the problem with it not waiting for `START_RECORDING`

You could mute the track and then show/hide it in your view based on the tracks <https://docs.livekit.io/client-sdk-js/classes/track.html#ismuted|isMuted> property.

I am showing the stream to the local user. Once I mute the stream, the display becomes black. I would like to avoid that.

Why does my proposed solution not work for your use case?

If I mute the stream, the local stream display turns black. I want to avoid that..

As I said, you can hide the video element in your UI if the track is muted. Or is it another video stream that turns black?

I dont want to hide.. I would like to keep showing the stream locally... Basically keeping the stream open/close different from publish/ubpublish

I see, please excuse the confusion. In that case, you could maybe keep publishing but unsubscribe the stream on the other clients?

But that consumes the bandwidth... Ideal is unpublishTrack(), if it can take an optional closeTrack : boolean option to control this behaviour. Not sure whether this is good for for the sdk.

i had the same problem when disconnecting from a room (which by default stops all tracks).
so for disconnect() there is the option already to keep the tracks running locally.
here's the commit: <https://github.com/livekit/client-sdk-js/commit/9e03ef5528d63e2c43b7868af4ee6feb27443094>

I was wondering, whether similar option can be provided for unPublishTrack() as well.

<@U02LPQVF6QH> I think that's a great idea! your use case makes a lot of sense. lmk if you wanna create an issue and/or PR for this.

<https://github.com/livekit/client-sdk-js/issues/81>

PR should be coordinated across client sdks (js, android, ios) or just in js SDK?

I think we can make the change for JS right to make sure the API is well designed, then we'll create issues to implement for mobile SDKs

<@U01KZ61GV5Z> I am not a typescript/javascript dev. I've tried to make the change.. assuming it is a simple boolean condition.. Please check and merge if it is fine.. Thanks..

can you share the request you were sending to freeze it?

For e.g.,
With the below request body:
`{"layout":"speaker-dark","roomName":"Krishna_1"}`

The error is as below.
`{"code":"internal","msg":"input url or template required","meta":{"cause":"*errors.errorString"}}`

After this with correct input, I get as below.
`{"code":"internal","msg":"no recorders available","meta":{"cause":"*errors.errorString"}}`

After restarting the recorder, it works properly.

Were you able to reproduce this issue? Should I raise an issue in GitHub?

I haven't yet, I've been out of office since Wednesday. I should have a fix on Monday though

<https://github.com/livekit/livekit-recorder/pull/30>

<https://github.com/livekit/client-sdk-ios/issues/26>

Hi <@U02L86FPQ66>, this is already implemented in v0.9.0. :slightly_smiling_face:
You can see the example app here <https://github.com/livekit/client-example-swift>

Have a look at <https://livekit-users.slack.com/archives/C01KVTJH6BX/p1637174009227500|this thread> where this issue is discussed in detail.

Great discussion thread there! Thanks for pointing me out.

server, livekit doesn't support p2p transport

any plans to add that? 1.to.1 communication? <@U01KZ61GV5Z> Can you help with the second part? If a user has a published stream, but no user is subscribed to that stream, will it consume bandwidth on server or client side?

currently it'll still send the stream to the server. in the future it's possible to detect that no one's consuming the track and pause publishing

In that case, client is sending tracks to the server. So, it is consuming upstream bandwidth.

<@U01KZ61GV5Z> "_in the future it's possible to detect that no one's consuming the track and pause publishing_" Is this going to get implemented soon? This would help me in my application design.

<https://github.com/livekit/server-sdk-js/blob/main/src/RecordingServiceClient.ts|https://github.com/livekit/server-sdk-js/blob/main/src/RecordingServiceClient.ts>

There's no documentation about RecordingServiceClient but it's implemented in the server-sdk-js
the constructor is : `RecordingServiceClient(_host_: string, _apiKey_?: string, _secret_?: string)`
more info in <https://github.com/livekit/server-sdk-js/blob/main/src/RecordingServiceClient.ts|source code (line #27)>

anyway, I wouldn't use that api (server-sdk-js) in the client, instead I'd build a backend service that implements livekit webhooks to listen for livekit events, and trigger the recording from there

If you want it triggered specifically by the client, you could also add an API to your backend service that kicks off the recording from client requests

Thanks for the follow-up comments, fellas.  I missed those yesterday.  Trying to trigger it from the client just as a POC (I see the twirp call being made but getting a 404 back)...but the proper solution would likely rely on a backend service of mine.  I'll likely ask more about this next week, if you don't mind.  Happy Thanksgiving!

If I may follow up on this thread (happy to move to the main room, if that's preferred... <@U02HN8GPUBY> <@U01UB0LHTTQ> How do you see the backend service kicking off the recording request, since this seems to be recommended over using the <https://docs.livekit.io/guides/recording/#start-a-recording|service client> ?

I have got the webhooks working in that they're calling an endpoint on my own service.  Is it as crude as launching a recorder executable (I also understand that only one recorder instance can run a recording at a time...and I'd have multiple concurrent rooms that need to be recorded).

right now the best way is to deploy in service mode and autoscale based on CPU <https://docs.livekit.io/guides/deploy/recorder/#ensuring-availability>

In the future we plan on adding custom autoscaling to ensure X recorders are available at all times

Yeah, but that's a K8s resource config, yes?  I haven't got Kubernetes on this platform (will in the future).

yeah, that example is using k8s. If you don’t have any way to autoscale then you’d need to either deploy the max that you would need, or have your backend service manage it somehow. It’s fairly CPU intensive, so having your service launch the executable on its own machine will only scale so far

Understood - I plan to profile it, and then set up some hardware guidelines.  In this scenario, would you say launching the executable is the way to go, then?

I think running it with Docker is going to be the way to go

launching the executable requires all the gstreamer dependencies, pulseaudio to be running, and you can only launch a single executable per machine (or the audio from every recorder service would all get mixed together, as they listen to system audio)

Makes sense.

If you'll indulge me further...the other week you were helping me set up my own template - which I have done (single instance).  Trying to run simultaneous recording through it seems be creating some interference.  Would you expect that I'd need one template instance per recording as well?

what kind of interference? I don’t think you need one per recording

I need to run some more tests to properly isolate (they're multiple clients with the camera on me, so it can be confusing), but in one instance:
• I created "Room A" with one participant, started recording, and added two more participants.
• With that running, I created "Room B" with one participant, started recording, and added one more participant.
• Reviewing the recordings - my recording of Room B opened with three participants...and it didn't change when I added the "second" participant to Room B. 

Glad to hear that you'd only expect needing one, though - it's what I thought too (and simply must be, somehow - given you're directing the world to your site by default).  ;&gt;

No, passing the API key and secret via config.

it sounds like your room B recording is actually joining room A

Very much doubt it.
• In the clients - the room participants are who they should be, eg. charlie, snoopy, woodstock, and lucy are in one room - I see them and only them.  In the other room; gobo, wembly, and mokey - I see them and only them.
• In addition (just tested now), when I play back the second recording, it shows four participants (there were four in the first room).  But as new people join the second room during the recording, no new participants are shown
• Finally (just noticed).  I ended the second recording and let the first continue for a bit.  The second file stayed at zero until I ended the first recording...then both files were filled
(Hope I'm making sense.)

So, definitely streams are getting crossed somewhere.

I built the grid-dark layout into a Docker container, and I'm using that.  Only change to it was to the CSS file to support some extra layouts (3x2, 4x3, etc).

file staying at zero until recording stops is normal

(And apologies...I should've read your note more carefully...you said _B recording_ was joining A...yes, looks like it.

both being at zero until only one of the recorders stops definitely isn’t normal

Okay...running "left" and "right".  Everyone has left the right room.  right.mp4 is still 0.  Will give it a minute

Still 0...will end "left" now...

Boom.  Both files

Okay...right continued recording.  So yes - right recording is definitely joining left room.

your recording request is using template with room name?

Yes...request.json is being generated by a script where I pass in the room name as an arg.  I clean up the files, but I'll leave them there so I can inspect them.

Okay...so the zero thing was definitely a mistake I was making.  Just did a re-test; reviewing the videos

I tested on our deployed template and can’t reproduce the interference, it’s hard for me to say whether it’s an issue with the token or the template

Thanks.  I'll try and dig.  I've definitely recreated the cross, though.
• Start left channel and recording, grows from one participant to four
• Start right channel and recording...it grows from one participant to two
• Left channel drops from four to three
• Right channel grows from two to three
• Everyone drops from the right channel
• Everyone drops from the left channel
The right playback starts with four participants, doesn't increase as people join the right channel, drops to three when the fourth person drops from left, and the playback ends when everyone has dropped from the right channel.

So the right recording starts and ends based on my instructions, the template is pointed to the right room, but it's recording the feed for the left room.  Bizarre.

does left recording end when everyone drops from left channel?

Left recording does everything I expect it to.

Watched left again...there's a visual glitch when right recording starts.  The screen goes grey for a moment.

that is bizarre. The recording ends when chrome sends an `END_RECORDING` message, which happens when `room.participants.size ===  0`  (<https://github.com/livekit/livekit-recorder/blob/main/web/src/common.ts#L23>)

are you running the recorder via executable currently (is it possible they’re sharing a screen)?

Yes...and that's "fine", in it's way.  Each recording is ending when it's supposed to...all participants have left the room in question...but the rooms are both showing the first feed within the time window.

it sounds like recording B is connected to room B if it’s ending properly, but then it’s getting the feed from room A

I'm running `livekit-react` locally - each person that joins one of the rooms is just another tab.  The `livekit-server` is deployed to a VM.  The livekit-recorder and template are on _another_ VM.

or both recorders being launched on the same VM

No, each recorder running on the same VM, its own docker instance.  Same config - using `template_address` to point to the co-located template container, different request.jsons (generated based on room name).

Each recorder has the same local volume mounted into `/output`, and they're both using host networking (because I'm using unsecured http for now, so connecting to the template with `<http://localhost>...` (in case any of this might be relevant).

one thing you could try is printing the url <https://github.com/livekit/livekit-recorder/blob/main/pkg/display/display_prod.go#L39|here>, navigating to the url that recorder B prints, and checking if you still see the participants from A but get the `END_RECORDING` message in the chrome console from B

Thanks, I'll try that.  Also going to try spinning up a second template instance to ensure they don't cross (I can't imagine why they would...but I can't figure this out either).

Recorder B recorded channel A - even when pointed at different template instances.  Building with the `display_prod` change now.

<@U01UB0LHTTQ> I may have misunderstood, but when I navigated to the URL, I got "room closed".  Only thing in the console was an AudioContext warning and
```Uncaught TypeError: Cannot read properties of undefined (reading 'addEventListener')
    at l.&lt;anonymous&gt; (Room.ts:201)
    at Object.f (events.js:242)
    at l.s.emit (events.js:153)
    at RTCPeerConnection.i.oniceconnectionstatechange (RTCEngine.ts:172)```
_Re: the URL...I had to change the server to the hostname since it was_ localhost _and I was connecting from another VM._

you got room close when the participants were all still connected?

Yes...and the calls stayed up, and I could add more participants...but the recordings never saved after everyone left.  Files are 0...recorders never shut down.

I added a <https://github.com/livekit/livekit-recorder/blob/main/pkg/display/display_prod.go#L117|println> to log the console messages (looks like it's just START and END RECORDING messages).  Each recorder starts and ends where it should.  One interesting note; I tried stopping the first channel first, to see how the second would react.

In the first recorder's logs I saw
```msg [ END_RECORDING ]  args [ [END_RECORDING] ]  // my log message
{"level":"info","ts":1638248653.003385,"logger":"livekit-recorder","caller":"service/handler.go:135","msg":"recording complete","recordingId":"standalone","duration":202}```
The second recorder immediately wrote
```X connection to :99 broken (explicit kill or server shutdown).```
and terminated.

<https://github.com/SeleniumHQ/docker-selenium/issues/370>

```However, it seems that separate chrome-node dockers on the same host share the same 'screen', meaning even with multiple separate chrome-node dockers, the maximum amount of started chrome instances is still limited to ~128 (256 screen 'clients')```

seems like the two recorders are sharing a single xvfb display

Makes sense...as soon as I got the same feed from separate recorders and separate templates - I figure it must be a leak into a host resource...it's outside my expertise, though.

for now, you can try changing `const Display = ":99"` to `var Display = fmt.Sprintf(":%d", 10+rand.Intn(990))`

<https://github.com/livekit/livekit-recorder/blob/main/pkg/config/config.go#L16>

Nice.  Will try that in the morning.  Thanks!

Made a <https://github.com/ct7t/livekit-recorder/commit/2f8a531fe4cd79806d1d6107cb7c5ca9f47583ee|quick-and-dirty change> so I could pass in a `DISPLAY` through env, and it will fall back on `:99` - which solved the problem!   I noticed that the value was already set to the env (which should be harmless here) and you're accessing the `Display` symbol directly elsewhere.

I suspect a proper fix may involve threading it through so it can be passed into the xvfb launch...seemed more tangled than I should take on; but if you want me to open a PR on this code, I'm happy to.  Let me know how we should proceed.

I have one more question about your setup - are you running the recorders with `--network host` in your docker command?

I’d like to find a way around adding it to the config, either by using a random value if collision probability is low enough, or somehow checking which displays are already in use