LiveKit Community

```unable to connect to redis: dial tcp: address redis://:p8b9985e...<http://zonaws.com:13810|zonaws.com:13810>: too many colons in address```

You can deploy it anywhere using the docker file or by building from source

My team uses terraform to deploy as a Google cloud compute instance

if you run it in development mode, you can hit `/debug/goroutine` or `/debug/rooms` to get a bunch of debug info on the server state

I'm not currently, but maybe I will for a while to try to track down this bug.

I'm also currently writing a small go app that uses the server sdk to perform a healthcheck.

we do handle the callback at `/`, though it currently doesn't reflect the health of the system. It'd be great to evolve that into a health check function.

The issue turned out being esbuild not being compatible with the publish dist of the lib. Was able to remap the imports esbuild had an issue with and all good now.

Screen Shot 2021-09-09 at 11.58.00 AM.png

I think it and `dev` are suppose to be env variables but i’m not seeing them show up when I pass them in through a docker-compose file.

On compose up the `nodeIp` and `logger`  level are different

Asking because I think it might be the reason i’ve been running into a `"WS connection closed"` whenever I try to connect to livekit.

On the client i’m using the react library with the example code with a token generated by the livekit server.

If I copy the token and use the example.livekit site it connects

you can find all the configurable env variables in <https://github.com/livekit/livekit-server/blob/master/cmd/server/main.go#L55>

looks like `dev` is not configurable via env, but NodeIP is, with `NODE_IP` instead of `node-ip`

The issue to my problem ended up being I was not exposing the right ports

fix
```
    expose:
      - 7880
      - 7881
      - 7882```

haha yeah and a great way to sink a few hours

do you have a backend? or do you mean adding this functionality into livekit?

I see. I think it should be straight forward to attach to the Room protobuf object. There are a couple of ways of doing it. We've talked about using a key/value pair based system that could allow us to attach various attributes to the Room itself

Someone just added an issue <https://github.com/livekit/livekit-server/issues/115|https://github.com/livekit/livekit-server/issues/115>

I would add a metadata variable of type bytes in the Room proto, so we can serialize whatever we want in it. I’m just wondering how we can implement an OnChange event ? 

yeah, I'm certainly open to that! the onChange event can be pretty similar to how participant metadata is handled

<@U022KB72NFJ> are you already working on this?
I'm the one who posted the issue on github and would be happy to contribute some groundwork, with the biggest stepping stone being my lack of go experience...

edit: opened a draft PR on the server repo by copying how participant metadata is handled

can't run it clustered yet due to limitations with the <http://fly.io|fly.io> load balancer setup

you can run a single instance easily enough

<@U02C10ZEP8R> has shared a config <https://github.com/bekriebel/livekit-flydotio>

You can do it clustered, but you need to add some extra stuff. The thread of Fly talks a bit about how I currently have mine set up. However, I think I may move my cluster to another host since it really feels like trying to fit a square peg it a round hole with running a LiveKit cluster on Fly. For a single instance, that example config works really well, though.

I followed your guide and was able to get everything deployed! Only thing I’m having trouble with now is I can’t seem to connect to any rooms. User is able to find the room, and it starts connecting, but just kicks user after about 2 seconds

```2021-09-17T01:43:23.710122124Z app[d0a66ba6] iad [info] {"level":"info","ts":1631843003.7073815,"logger":"livekit","caller":"rtc/room.go:191","msg":"new participant joined","pID":"PA_iN9GAGFSSHCe","participant":"FqbtoPdfBRX8dfuOy61k9KHjZTy2","room":"014b588e-6329-4810-b22c-2efa6bda108a","roomID":"RM_LvDYdja6caex"}```
2021-09-17T01:43:28.763127241Z app[d0a66ba6] iad [info] {“level”:“info”,“ts”:1631843008.7607243,“logger”:“livekit”,“caller”:“service/rtcservice.go:124",“msg”:“WS connection closed”,“participant”:“FqbtoPdfBRX8dfuOy61k9KHjZTy2",“connID”:“CO_zNaP7wupWpU7"}
2021-09-17T01:44:36.341262593Z app[d0a66ba6] iad [info] {“level”:“info”,“ts”:1631843076.3389888,“logger”:“livekit”,“caller”:“rtc/room.go:332”,“msg”:“closing room”,“roomID”:“RM_LvDYdja6caex”,“room”:“014b588e-6329-4810-b22c-2efa6bda108a”}
2021-09-17T01:44:36.343003481Z app[d0a66ba6] iad [info] {“level”:“info”,“ts”:1631843076.3390553,“logger”:“livekit”,“caller”:“service/roommanager.go:145",“msg”:“deleting room state”,“room”:“014b588e-6329-4810-b22c-2efa6bda108a”}
2021-09-17T01:44:36.346803258Z app[d0a66ba6] iad [info] {“level”:“info”,“ts”:1631843076.3438182,“logger”:“livekit”,“caller”:“service/roommanager.go:359”,“msg”:“room closed”,“incomingStats”:{“packetBytes”:0,“packetTotal”:0,“nackTotal”:0,“pliTotal”:0,“firTotal”:0},“outgoingStats”:{“packetBytes”:0,“packetTotal”:0,“nackTotal”:0,“pliTotal”:0,“firTotal”:0}}

logs for reference in case you’ve come across this before

We’re in the exact same position as <@U0295KSGK3J> . You can add our vote for SIP call-in which would be the only missing feature for us to make the switch :slightly_smiling_face:

hey <@U02EN4Y8DCM>, it still looks like some sort of firewall issue.

if you look at the Security Group used by the elasticache cluster, does it allow inbound connections from the EKS nodes?

Note: the tricky part about this is EKS may use a couple of different security groups depends on how you set it up. See if your redis security group allows for incoming connections from the listed EKS security groups.

Screen Shot 2021-09-16 at 5.09.47 PM.png

See EKS -&gt; Clusters -&gt; your cluster -&gt; Configuration

yeah I have explicitly allowed traffic to the Redis port from all sources, and allowed it particularly from the primary EKS security group. I’ll try putting the ‘additional’ one in the list too :confused:

CleanShot 2021-09-17 at 10.14.10@2x.png

here’s the redis cluster security group. they’re in the same region (double checked). I’ve done a reachability analysis from the EC2 nodes powering the EKS cluster and it said they’re reachable.

as I mentioned I was getting a failure to dial tcp at first, adjusted the cluster security group, and now it’s a timeout on read instead

I’m new to everything AWS, sort of filling in as a fullstack currently so we can test self-hosted video, so I’m learning on the go a bit

notably I did have a `dial tcp` error before I opened the correct security group ports, so I’m not sure it’s a networking error in my config anymore…

alternatively if there’s a good guide to deploying Redis alongside LiveKit in k8s I’d love to see that too

```2021-09-17T01:43:23.710122124Z app[d0a66ba6] iad [info] {"level":"info","ts":1631843003.7073815,"logger":"livekit","caller":"rtc/room.go:191","msg":"new participant joined","pID":"PA_iN9GAGFSSHCe","participant":"FqbtoPdfBRX8dfuOy61k9KHjZTy2","room":"014b588e-6329-4810-b22c-2efa6bda108a","roomID":"RM_LvDYdja6caex"}
2021-09-17T01:43:28.763127241Z app[d0a66ba6] iad [info] {"level":"info","ts":1631843008.7607243,"logger":"livekit","caller":"service/rtcservice.go:124","msg":"WS connection closed","participant":"FqbtoPdfBRX8dfuOy61k9KHjZTy2","connID":"CO_zNaP7wupWpU7"}
2021-09-17T01:44:36.341262593Z app[d0a66ba6] iad [info] {"level":"info","ts":1631843076.3389888,"logger":"livekit","caller":"rtc/room.go:332","msg":"closing room","roomID":"RM_LvDYdja6caex","room":"014b588e-6329-4810-b22c-2efa6bda108a"}
2021-09-17T01:44:36.343003481Z app[d0a66ba6] iad [info] {"level":"info","ts":1631843076.3390553,"logger":"livekit","caller":"service/roommanager.go:145","msg":"deleting room state","room":"014b588e-6329-4810-b22c-2efa6bda108a"}
2021-09-17T01:44:36.346803258Z app[d0a66ba6] iad [info] {"level":"info","ts":1631843076.3438182,"logger":"livekit","caller":"service/roommanager.go:359","msg":"room closed","incomingStats":{"packetBytes":0,"packetTotal":0,"nackTotal":0,"pliTotal":0,"firTotal":0},"outgoingStats":{"packetBytes":0,"packetTotal":0,"nackTotal":0,"pliTotal":0,"firTotal":0}}```

<@U028837NZMW> Can you try changing your region to `sea`? I think Fly may have changed something in some of the regions that broke UDP. My `ord` instance isn't working, but my `sea` and `fra` ones still are. I'll try `iad` next.

Yep, I confirmed that `iad` is failing in the same way `ord` started to recently. I posted a message to Fly's forum: <https://community.fly.io/t/udp-changes-in-some-regions/2566>

interesting, appreciate all you’ve done to help out!

They just did the constraint mentioned in the forum post and `ord` is working for me now. I'd recommend messaging Fly there if you are still having issues.

Yeah unfortunately I switched to ord and it is not working for me either

are you looking for a scalable cluster? or a single node?

single node is pretty easy.. I would just deploy it on a single VM somewhere and have a process watcher to monitor it.

for a cluster, I would recommend going with GKE, it's fairly simple to set up

good question. The helm chart stores them in a config map. You could use access control in kube (RBAC) to control who should be able to see that.

FWIW, we can probably move the entire configuration as a config map in a later version of helm chart.

I’m using gitops deployment, so I’m mostly trying to figure out whether I need to avoid committing them to our repo as part of the deployment configuration

it would be simpler for me to just have them in a secret, but I’m too novice at kubernetes to know how I’d supply a key name from a secret if such a thing is possible

hey Peter, we haven't done much with cloud formation. It'd be super helpful if you could share your recipes. I'm planning to create a repo with pointers to various deployment guides that folks have. Happy to link to it.

at one point we did have <https://github.com/livekit/livekit-server/tree/d38fc43b895d4b7c714748c65ebc06a9e157d96b/deploy/aws-ecs|terraforms deploying to ECS> in our repo, in case that's helpful

Thank you <@U01KZ61GV5Z>. Once I come up with something I will share it.

(And, if not, then I am happy to create mine and share it)

it seems that your DNS isn't responding with it.

I just launched a redis container in the namespace too and tried executing redis-cli to ping - seems to work fine

```➜ kubectl exec -it redis-cli -n livekit -- redis-cli -h redis-master.redis -p 6379 ping
PONG```

at a bit of a loss here since the container is in the same namespace, same node, connecting to same host… what else could be misconfigured between this container and the livekit-server?

have you tried by putting in the IP directly? just to see what the issue might be?

<https://stackoverflow.com/questions/59740104/can-kubernetes-pods-that-use-host-networking-access-service-resources-by-the-ser>

might be related

confirmed, changing the pod’s dnsPolicy to ClusterFirstWithHostNet got past that error

patched it manually, perhaps this could be changed in the helm chart?

ah great find.. we'll fix it in the helm chart

might have just generated them wrong… if anyone has a reference resource file for a certificate I’d be grateful

yeah, looks like there’s supposed to be a .crt … hm

might be confused here, it’s possible the certs never generated and I’m assuming these secrets are what I was after … the names are wrong lol

yup, don’t mind me, I haven’t even routed my DNS to the cluster so it can validate the certs :man-facepalming:

Hey <@U02EN4Y8DCM>! It seems like you got LiveKit set up on GKE, including the turn? I'm also trying to do that, but having trouble generating the turn SSL certificate using cert-manager. Did you use the HTTP or DNS resolver yourself?

here’s the stuff I currently have deployed: <https://gist.github.com/a-type/a48f3ec3d472b8329c190d7685d7318c>

so, two certs for cert-manager, then I used the `gke` load balancer in the livekit helm chart values file and provided the names of the secrets

deploying that, it created the external load balancer for the turn service in GKE automatically and properly connected the cert once it was generated by cert-manager

might help to know if the cert is actually being generated or of it’s just not being connected to the service / load balancer, or what

Thanks Grant! I think the problem probably is somehow related to the fact that we use the nginx ingress instead of the GKE defaults (cert-manager is configured to use nginx as well). It seems that there was some recent update livekit related to this so I'll check that out.

that sounds great! You should be able to generate a token using the configured api key/secret to connect to the server.

<https://docs.livekit.io/guides/getting-started/#generate-access-token>

then navigate to <https://example.livekit.io>, enter <wss://yourhost.com> into the URL, and paste in the token

ok, tried that - the socket did connect and upgrade, some binary messages were passed (looks like ICE candidates, etc). But the page reports “could not connect after timeout”

would be happy to provide the server / token and more network details in DM

that might sound like the RTC ports aren't <https://docs.livekit.io/guides/deploy/#firewall|opened on the firewall>. can you double check there?

oh yup, I forgot to redo that step when moving to GCP

success! I can finally see my face! :smile:

the sample command just runs `"cd example &amp;&amp; webpack serve --host 0.0.0.0 --public localhost:8080",`

perhaps try running it manually and/or without the `--public` flag?

looks like a webpack update has changed a few of the command line params. fixed now!

Ok it opens fine now but when I try to connect it says: 
could not connect: Cannot read properties of undefined (reading 'getUserMedia')
With the Android sample and <http://example.livekit.io|example.livekit.io> works fine.

Which browser are you using? Some browsers block access to getUserMedia on non-secure webpages, which this will be one of. Chrome should allow for this.

chrome running on a Xiaomi note 9 pro smartphone (Android 11)

probably a limitation of the mobile browser.. you'd need HTTPS

Don't you think it's a bit heavy to call ListRooms every time I want to find an available room ? Wouldn't it be better to have a synchronized list of rooms with the webhooks ? I just wonder what will happen if one of the nodes doesn't answer anymore and RoomFinished is not called ?

I'm not following.. all of the state are persisted into Redis, it's used as a database

I mean I have a service that calls ListRooms every time someone wants to find a room ( According to certain criteria ). And I wonder if it's not a bit slow to have to call ListRooms every time

not at all. it's backed by Redis, uses HGETALL. you can throw 100k qps at it and it'll be fine

Ok nice, me who thought I was going to have to keep a synchronized list of rooms using webhooks :sweat_smile:

hmm, if you tried disabling CORS on your browser, did it pass?

trying to boil it down to purely a CORS thing as opposed to some issue with the request to the endpoint

if I run chrome by disabling web-security options, then it by-passes CORS and works fine. But it doesn't work in normal browser.

I would say this is something <@U01KZ61GV5Z> can probably help with, since he works on the main Server component. Something about the CORS settings in  server being spun up through Go is blocking even localhost requests...

yes, I was wondering how does everyone uses livekit-server-sdk. does it not not give cors error to them?

When I used it to test my local client, it worked fine...

I have added livekit-server-sdk as a dependency in livekit client and calling the room service apis from there. What is the recommended setup?

<@U027T5G3KQE> the server SDK isn't designed to be used in a client application. It should be used by your node.js server

other than CORS, you also don't want to embed your api key/secret inside the client app

Yeah I thought it sounded like you might not have set up a server for yourself. You'll want to have a server (or I suppose theoretically a serverless function/set of functions could work as well?)

thanks <@U01KZ61GV5Z>, <@U01TC9Z6ELD>, <@U0291E7JEP3>. created a separate server and it worked!

AFAIK there is no built-in support for SSL certificates in livekit.
The docs recommend either a load balancer that handles termination or a reverse proxy. (<https://docs.livekit.io/guides/deploy#domain-ssl-certificates-and-load-balancer>)

I don't know what your setup looks like, but assuming you are running a single instance it might be easiest to use a reverse proxy that automatically handles the certificate creation/update for you. You could use Caddy for that: <https://caddyserver.com/docs/quick-starts/reverse-proxy#caddyfile>

Thanks <@U02EU2ZPTQU>
Can you help me a litte?
I created two reverse-proxy for https and wss as below:
```sudo caddy reverse-proxy --from <https://domain.x:8880> --to <http://127.0.0.1:7880>```
And
```sudo caddy reverse-proxy --from <wss://domain.x:8880> --to <ws://127.0.0.1:7880>```
but client throws an error:
```could not connect: Error: could not connect after timeout,```
And livekit debug:
```2021-09-29T07:00:15.967Z        DEBUG   livekit service/roommanager.go:226      starting RTC session    {"room": "RM_AavtozPgz2XX", "nodeID": "ND_QfPJ2iRu", "participant": "<mailto:namlongxx@gmail.com|namlongxx@gmail.com>", "sdk": "JS", "sdkVersion": "0.12.2", "protocol": 3}
2021-09-29T07:00:15.969Z        INFO    livekit rtc/room.go:192 new participant joined  {"pID": "PA_h3mpapkNqbLi", "participant": "<mailto:namlongxx@gmail.com|namlongxx@gmail.com>", "protocol": 3, "room": "333", "roomID": "RM_AavtozPgz2XX"}
2021-09-29T07:00:15.970Z        INFO    livekit service/rtcservice.go:139       new client WS connected {"connID": "<mailto:namlongxx@gmail.com|namlongxx@gmail.com>", "roomID": "RM_AavtozPgz2XX", "room": "333", "participant": "<mailto:namlongxx@gmail.com|namlongxx@gmail.com>"}
2021-09-29T07:00:16.121Z        DEBUG   livekit rtc/participant.go:760  sending server offer to participant  {"participant": "<mailto:namlongxx@gmail.com|namlongxx@gmail.com>", "pID": "PA_h3mpapkNqbLi"}
2021-09-29T07:00:16.121Z        DEBUG   livekit rtc/participant.go:705  sending ice candidates  {"participant": "<mailto:namlongxx@gmail.com|namlongxx@gmail.com>", "pID": "PA_h3mpapkNqbLi", "candidate": "udp4 host 10.148.0.10:7882"}
2021-09-29T07:00:16.121Z        DEBUG   livekit rtc/participant.go:705  sending ice candidates  {"participant": "<mailto:namlongxx@gmail.com|namlongxx@gmail.com>", "pID": "PA_h3mpapkNqbLi", "candidate": "tcp4 host 10.148.0.10:7881"}
2021-09-29T07:00:16.170Z        DEBUG   livekit rtc/participant.go:366  setting subPC answer    {"participant": "<mailto:namlongxx@gmail.com|namlongxx@gmail.com>", "pID": "PA_h3mpapkNqbLi"}
2021-09-29T07:00:21.021Z        INFO    livekit service/rtcservice.go:122       WS connection closed    {"participant": "<mailto:namlongxx@gmail.com|namlongxx@gmail.com>", "connID": "<mailto:namlongxx@gmail.com|namlongxx@gmail.com>"}
2021-09-29T07:00:21.022Z        DEBUG   livekit service/roommanager.go:333      RTC session finishing   {"participant": "<mailto:namlongxx@gmail.com|namlongxx@gmail.com>", "pID": "PA_h3mpapkNqbLi", "room": "333", "roomID": "RM_AavtozPgz2XX"}
2021-09-29T07:00:21.022Z        DEBUG   livekit rtc/participant.go:724  updating participant state      {"state": "DISCONNECTED", "participant": "<mailto:namlongxx@gmail.com|namlongxx@gmail.com>", "pID": "PA_h3mpapkNqbLi"}
2021-09-29T07:00:21.022Z        DEBUG   livekit rtc/room.go:165 participant state changed       {"state": "DISCONNECTED", "participant": "<mailto:namlongxx@gmail.com|namlongxx@gmail.com>", "pID": "PA_h3mpapkNqbLi", "oldState": "JOINING"}
2021-09-29T07:00:42.589Z        INFO    livekit rtc/room.go:343 closing room    {"roomID": "RM_AavtozPgz2XX", "room": "333"}
2021-09-29T07:00:42.589Z        INFO    livekit service/roommanager.go:74       deleting room state     {"room": "RM_AavtozPgz2XX"}
2021-09-29T07:00:42.590Z        INFO    livekit service/roommanager.go:302      room closed     {"incomingStats": {"packetBytes":0,"packetTotal":0,"nackTotal":0,"pliTotal":0,"firTotal":0}, "outgoingStats": {"packetBytes":0,"packetTotal":0,"nackTotal":0,"pliTotal":0,"firTotal":0}}```
I wonder, changing the port livekit-server has any effect? I deploy livekit-server on gcp and open ports:
```tcp: 7881,7880,12000,9090,4200,8880
udp: 7882,50000-60000,30000-40000```
any wrong?
Thanks for any suggection.

try omitting the protocol in the caddy command
```caddy reverse-proxy --from domain.x:8880 --to 127.0.0.1:7880```
also check the logs of caddy to see if this side of things is working

when u mean "refresh the page", do you mean having to wait a bit of time before ListRoom will reflect the deletion? 

if so.  that is correct. we currently return the call before the operation is fully done. it should be a simple fix. feel free to make a PR if u need this soon

ah great to hear it. alternatively, you could set <https://github.com/livekit/livekit-server/blob/master/config-sample.yaml#L30|rtc.use_external_ip> to true, and also use host networking (if running via docker)

Hi <@U01KZ61GV5Z> ,
How can I set use_external_ip= true ? We are using Docker images about that livekit-server.

<@U02FZ0Z744F> You could mount the config.yaml while starting the docker image

There is a sample config file hosted <https://github.com/livekit/livekit-server/blob/master/config-sample.yaml|here>

```docker run --rm \
  -p 7880:7880 \
  -p 7881:7881 \
  -p 7882:7882/udp \
  -e LIVEKIT_KEYS="&lt;key&gt;: &lt;secret&gt;" \
  -v ${PWD}/config.yaml:/config.yaml \
  livekit/livekit-server \
  --dev \
  --config config.yaml \
  --node-ip=&lt;machine-ip&gt;```