LiveKit Community

Hey Guru, how is this getting into LiveKit? we don't support raw RTP packets as a form of ingestion. You should try feeding the encoded stream via a unix socket to livekit-cli

The rtp packets are being read in golang from the UDP port and then written using a `TrackLocalStaticRTP` object. Could there be an internal timeout that is rejecting the packets? I am seeing a tone of PLI requests even though the packets are being pushed.

it's possible, we did notice that GStreamer's tcpsink isn't able to keep up with certain streams. <https://github.com/livekit/livekit-cli/issues/81>.

Oh. I will try using a unix socket and see if that improves. We are dependent on livekit for SFU. Thanks a lot for your response!

no problem! alternatively you can try to embed GStreamer like the user in the linked issue above

Hey <@U01KZ61GV5Z>, we made a change to the way keyframes are detected for H264 based on the RFC and now the gstreamer packers seem to be working. Here is a PR that <@U041FJD5WQJ> submitted for the fix.  <https://github.com/livekit/livekit/pull/1016>

*Background*
We found that for the same `TrackLocalStaticRTP`, the FFMPEG pipeline works well but the gstreamer pipeline didn't work. Digging deeper into livekit's code made us understand this was happening because the `checkSync`never locked the layer because the keyframe wasn't detected.

We want to get your thoughts on the PR before we proceed to create a patch for this.

<@U01KZ61GV5Z> To add to that livekit was dropping the packets that contains the keyframes. Checking for SPS as a keyframe indicator makes SFU forward the keyframes and fixes artifacts.

this is fantastic! I'll send this over to folks that understand H.264 better to have a look. appreciate how deep you guys had gone and patching it with the PR.

which client SDK are you using? and which version?

<@U01KZ61GV5Z> i am testing livekit running server on a <https://example.livekit.io/|https://example.livekit.io/> and got this error.

<@U01KZ61GV5Z> 
Is there any issue in my configuration? 

<@U01KZ61GV5Z>
We are using a flutter livekit client SDK version 1.1.1 .

Hey <@U0427KZNM7A>, can you share the config you've used to deploy?

<@U01KZ61GV5Z>
I had deployed this on AWS ubuntu VM .Using this reference <https://docs.livekit.io/deploy/vm/> .
AWS Configuration
• t2 medium instance 
• Deployed using cloud init generated script by using "docker run --rm -it -v$PWD:/output livekit/generate"

image.png

are you able to reproduce this consistently?

can you share what's inside your `config.yaml` file? it's in /opt/livekit/

<@U01KZ61GV5Z>
I am not able to see this config.yaml file

please note folks' working hours and avoid tagging multiple times.
In any case, your config looks fine. are you actually having issues connecting? or are you just concerned about the log line?

Ok <@U01KZ61GV5Z> 
Once attendee joined ,after some time left so in short connecting issue.

LiveKit React Example - Google Chrome 2022-09-14 12-43-38.mp4

<@U01KZ61GV5Z>
I have a captured a video.

I don't see any disconnected events.. however, your camera seems to be deactivated (the button changes from `Disable Video` to `Enable Video` )
this can happen when another application has acquired the camera, or if there are device level changes.

does it happen when you are only using a single browser window (and ensure no other applications are using the camera)?

<@U01KZ61GV5Z>
We are using different system browser and once attendee joins after some period of time video turn off and host side attendee tiles not visible.

I see you've comments on the GH issue, if you can provide repro instructions, we'd be happy to take a look. I'm not able to reproduce this on my end.

LiveKit React Example - Google Chrome 2022-09-15 12-02-25.mp4

Hi <@U0427KZNM7A>, are you able to solve this issue? If so, what's the step do I need to solve this?

Thanks.

Hi there!

what does your server logs state? It sounds like the URL might not be pointed the server correctly.

neither of the livekit-cli commands I pasted generate any logs

they did generate logs, please refer to the `invalid authorization token` messages in your screenshot

there are always logs printed when you attempt to join the room, even if the join attempt fails

what is the command line that you are using with `livekit-cli` ?

see if `wss` works? (also you might want to blank out the key/secret from slack)

hmm that's a head scratcher if the server isn't showing any logs.

they are both going over the same port/service, so if room service API can reach, it should be going through ok

because the container/pod is publishing ports

but they're being mapped to different ports on the host

HTTP and websockets are handled the same way on livekit

I'm noticing your certificate is issued to `<http://timu.life|timu.life>`, not `<http://livekit.timu.life|livekit.timu.life>`

It's a long shot, but it's possible Go's websocket dialer doesn't like that

I'm not seeing. the wildcard part in the common name

left is yours.. right is <https://encom.livekit.cloud>

I noticed that livekit has it's own contuor

perhaps it's because I'm piggybacking off the main site cert

and don't have the the livekit contuor configed with letsencrypt directly

I'm going to tinker around with this a bit more

no problem! hope that's it :slightly_smiling_face:

Hi David, thanks for the prior help.  As it turned out having a cert where the CN is the subdomain solved the issue.

We've gotten pretty far down the POC of livekit and we're looking at track egress recording.  At this point we can start non-composite track recording with "livekit-cli start-track-egress" successfully but what we really want to do is to record all individual tracks (separate audio and video) in a room without having to make additional calls to the API.  I noticed the "--recording-config value  path to json recording config file" on the create-room call but I haven't seen an example usage in the docs or code that shows how to use it.  I'm not even sure if it'll do individual track recording on all tracks in a room separately or if it was just intended for room-compositing.  Any info on this flag or intended direction would be helpful.  Thanks!

Awesome to hear!

Good timing re: egress. We are releasing an update in the next couple of weeks (1.3) that supports auto-egress with the CreateRoom API. It can automatically record all raw tracks that are published to the room.

Okay cool.  Is that also going to perhaps include a metadata file for all the tracks in a room?  Like for twilio if you have a room setup to record all tracks it'll make available a json file that has start and end times.  Participant SIDs on tracks etc..?

no, it'll post to your webhooks for each of the egress sessions

it's a good idea to store it as a manifest file too. we haven't thought of that :slightly_smiling_face:

for parity with twilio having a complete recorded session metadata file would be helpful otherwise we'd have to assemble the session state in post from numerous webhook callbacks.

I think the request is similar to this issue: <https://github.com/livekit/egress/issues/118|https://github.com/livekit/egress/issues/118>

lol no worries. it's a good feature request, we'll discuss and will keep you guys posted

As a suggestion the twilio manifest looks like this more or less...

Hey <@U03SN0JBTJT>, I use both in this sample project (needs better documentation, it's on my list of TODOs!):

<https://github.com/livekit/hq/blob/master/client/src/web/LiveKit.ts|https://github.com/livekit/hq/blob/master/client/src/web/LiveKit.ts>

Kubernetes would be the simplest way to go, take a look at <https://docs.livekit.io/deploy/kubernetes/|kube docs> and <https://docs.livekit.io/deploy/distributed/|multi-node considerations>.

Another question team!

our instances are being killed in k8s while there are egress services running. We checked and the terminationGracePeriodSeconds is not present in the helm chart for egress. This value should be added, no?

Edit: we added terminationGracePeriodSeconds manually and it seems it work for a moment. The pod was in Terminating state but when I stopped the recorder this happened: pod killed but no video uploaded and webhook sent.

thx!

Hey Albert,

You can use the <https://docs.livekit.io/deploy/egress/#ensuring-availability|prometheus availability metric> that we expose to autoscale. If it fails, then it's up to the client to retry when failures happen.

hmm based on what you are saying, there seems to be a bug with how we are draining then. it should have uploaded the files prior to shutting down

can you file an issue for this one? we'll look into it

Sounds good! We will try prometheus and open the issue

Thanks David :+1:

issue opened <https://github.com/livekit/egress/issues/106>

Hello <@U03Q1K8NXE3>, just a quick fyi and simply to keep the board cleaner, would you be able to thread your questions together, please. It's easier to follow also at the same time. :slightly_smiling_face:

when i run "livekit-server --dev" , it shows " no keys provided, using placeholder keys"

that's explained <https://docs.livekit.io/guides/getting-started/#starting-the-server|here>

thanks, but when i run "livekit-server --dev" it shows there is no "devkey" and "secret" , how can i generate those for my subdomain ?

you'll need a config file with your own key/secrets: <https://github.com/livekit/livekit/blob/master/config-sample.yaml>

thanks for the reply, can you please tell me how can i generate the secret key before running this user data script inside my EC2 ?


<https://github.com/livekit/livekit/blob/master/deploy/cloud-init.ubuntu.yaml>

I see what you are doing. Please refrain from using that template as it's out of date. I'll remove it from the repo.

Follow <https://docs.livekit.io/deploy/vm|docs here> to generate proper config files

Hello <@U041NKDC9B7>, I would suggest you to try the connection test tool available here : <https://livekit.io/connection-test>.

Please provide us with the details, hopefully this can be helpful.

when i add the url and the token in <https://example.livekit.io/> , it gives me that error

as far as our own internal testing goes, this should be fine and hasn't shown any issues

Great, thanks. I wasn't sure if it was even an intended mode of function!

<@U041H0KUB71> Like <@U01NZKTNE1H> mentioned this should work.

But, just wanted to highlight a couple of bits
• Acquiring media devices, this is always a bit of a challenge with browsers and OSes. Some times, there is exclusive access. I do not remember all the variations of this and if this is currently an issue with any browser/OS combination, but just something to watch for.
• If you are publishing to multiple rooms, the same stream (let's say you are publishing your camera video to two rooms) could be competing for upstream bandwidth and browser's bandwidth adaptation could adapt each stream differently.
• Same thing with down stream. If you are pulling media of same participant from different rooms, they will compete for downstream bandwidth.
I am guessing your application involves being present in multiple rooms, but probably media is active in only one of the rooms at any given time? Just wanted to check if you have seen this API and see if this is useful to you - <https://docs.livekit.io/client-sdk-js/classes/LocalParticipant.html#setTrackSubscriptionPermissions>. This will allow a participant to say who can see/hear their streams. Maybe, some form of this API is useful for your use case. Maybe, there are enhancements that will be useful to you. If you find this useful OR can think of ways to enhance this to make your use case implementation simpler, would love to hear it.

<@U02CM28PB4Y> <@U01NZKTNE1H> Indeed. The application is fundamentally an intercom system. say 60-100 people doing some common activity (a concert, a parade, ...). There are a number (say ~10'ish) of specialist groups within the team (maybe Security, Medical, Tickets, Catering, ... you get the idea). Each will have their channel (a Room is the plan) where they coordinate within their speciality. For maintaining awareness, many people will want to monitor one or more of the other channels (at reduced volume, usually). Occasionally, they will need to speak on one of the other channels.

Within each channel voice procedure discipline, and the UI, will ensure only one person speaking at any time.

This sort of system is what I have used for many years in my real-life job, but always implemented in very expensive and inflexible hardware ...

Now that I know I can use rooms for the channel grouping one of the bigger issues is clarified, but replaced by the pain of creating, tracking (and periodically renewing?) a large number of tokens (one per person/room pairing). The capability to have "multi-room" tokens would be a great help, for example ...

What a wonderful project Nigel. Thank you for the details. Please keep us posted on how it goes and any help we can provide:pray:

Just in case you missed the edit ...

The capability to have "multi-room" tokens would be a great help, for example

Token refresh will happen automatically from server side once you are connected. It should happen periodically before the current one expires

How does that work? If I have the following ....

```let at = new AccessToken(apikey, secretkey, {
        identity: participantName
    });
at.addGrant({ roomJoin: true, room: roomName, canPublish: true, canSubscribe: true });
let token1 = at.toJwt();```
I guess that the auto update will take place within 'at' and that I should convert to JWT at every point of use?
(Sorry, still fighting with slack formatting :zany_face:)

Mind filling a GH issue about multi-room token. We will discuss it.

A token is only needed at join time. A refresh is needed if there is a need to reconnect. Reconnect + token caching should happen within the SDK. There should be no need for you to do anything. Cc: <@U02EU2ZPTQU> for confirmation.

can confirm, that after you initially connected, the client sdk will take care of refreshing the token for a potential reconnect.

<@U02GAC58A79> Mate, additionally I am curious avout setting audio input and output on web and ios. I see android is fully supported with Hardware.instance singleton

Setting audio output isn't supported the same way on Web/iOS due to platform specific restrictions. iOS doesn't give application-level control to audio output (besides speaker or earpiece).
Web only allows audio output selection on <https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/setSinkId|certain browsers>.

<@U01KZ61GV5Z> thanks for getting back mate. Completely understood. Maybe for web thr can be helper function to chk support and then allow Hardware.instance singleton to try setting it or throw error

On ios do flutter livekit have helper function to toggle between speaker and earpiece?

Yes, there is a mobile-specific method
<https://github.com/livekit/client-sdk-flutter/blob/main/lib/src/hardware/hardware.dart#L81>

Hello <@U03T51VH7JS>,

If I recall correctly the JS version has those options. Feel free to have a look <https://github.com/livekit/client-sdk-js/blob/main/example/index.html>

Mate thats javascript client, I was asking about flutter web

Hi Nathan, yup, here are details about ports used (defaults) and ones that are exposed: <https://docs.livekit.io/deploy/ports-firewall/>

both server and client will support one major version of backwards compatibility.

i.e. version 1.x of server will work with 0.x and 1.x clients.

with that said, we recommend using the recent versions of both server/client as we are consistently making improvements to the entire stack

Hey Myron,

it should handle quite a bit. The exact number will depend on the size of machine that you run on. What audience size are you looking to host?

here. are <https://docs.livekit.io/deploy/benchmark/#video-room|some benchmarks> on a single publisher (publishing 720p stream).

We are looking to host 5 people with audio and video and up to 10k in the audience 

But if that is too much for our server we can set a limit of 7k in the audience 

We are just trying to find out how much it would costs to hosts these types of audiences with a dedicated server on AWS and what configuration is f memory and space would be necessary for this

for that sized audience, a single server is probably not going to handle it well. you might want to check out LiveKit Cloud. CC <@U02BH3ZHJ0Y>

But do you think it won't handle it well even with kubernetes autoscaling?

Also is LiveKit Cloud already live or open for use?

how can I get entry into the cloud community ?

Hey Myron, nice to connect with you. I'll send you a DM

hey <@U01KZ61GV5Z>  <@U02BH3ZHJ0Y> , If we do auto-scaling  on our vm or we use kubernatics , can we be able to achieve 5 pubs and 10k subs for multiple rooms?

suppose we're using 4 thread / vcpu  + 4gb ram server on digital ocen

and our sub max limit for the room is 10k. and pub max limit is 5. only 1 host can pub always.

<@U03B7M4QPRP> there are no limits on how many rooms you could have. The limit is how much traffic each server could handle (rooms must fit on a single server with self-hosted livekit)

take a look at benchmarking your install using the CLI. that should give you a better idea.

git bersion：07ac4da86bc001465caef664cec86bf7b7c40e65

sorry, we can only support official releases. individual commits may contain bugs/regressions. Please let us know if you can reproduce this on the latest version.

lastest version v1.2.4 has resolve this question  tks!

log.txt

Hi, does anyone encounter the problem that sound can be played but video cannot be played?

memoryerror 500.PNG

I think this is a bug with the load tester itself.

Could you file a bug report for livekit-cli? Please include as much details as possible in the report, including the operating system, CLI version, type of machine (cores/memory), and parameters used to trigger the issue

that's likely unrelated to how it's deployed, but more with the networking settings.

can you describe your set up a bit?

Now, I have run more testing. On flutter web it was connecting and reconnecting after some time

In terms of network setting, i used your default vm config on aws

With all ports open on ubuntu host instance

ah, that seems like a flutter-web issue. I see the comment on the issue that was reported. we'll look into it

<@U01KZ61GV5Z> <@U02GAC58A79> <@U01KZ0SCDKL> Any general comments on this?

it's a client-side setting, see <https://docs.livekit.io/guides/room/publish/#advanced-track-management|track management docs>

I’m checking this issue and it should be fixed soon.

Hey <@U03S4M53HEF>, you are right currently there isn't a good way to communicate rotation within livekit. this is something you can signal out of band with a data message.

we'll think about how we can make this more integrated within the system

Here's how you can send a data message on the device that's being rotated: <https://docs.livekit.io/guides/room/data/>

It's definitely possible. We know a few folks that are doing something like this to ingest content from other WebRTC systems to LiveKit. A few of things to note:
• There isn't a standard way to group tracks within plain WebRTC. without the metadata that we send, you'll get individual tracks
• Publishing tracks to livekit requires a signal message so that it has details about track metadata.
• You would have to use the `SignalClient` directly and intercept the SDPs.

What is your use case? and which webrtc systems are you trying to interface with?

I am thinking to integrate LiveKit with Freeswitch where LiveKit works as a SFU for webrtc participant and Freeswitch works as a MCU for SIP/telephone MCU where audio is mixed

basically the idea is that there is a middle man that sends SDP from Freeswitch to LiveKit and passes SDP from LiveKit to Freeswitch

Freeswitch publishes the mixed audio from SIP/telephone participants to LiveKit. For this side, Freeswitch sends SDP offer to LiveKit, and LiveKit replies with SDP answer to Freeswitch with the help of middle man

For reversed direction, middle man subscribes to the webrtc participants in LiveKit on behalf of Freeswitch. I expect LiveKit can send SDP offer to middle man that forwards to Freeswitch, and then Freeswitch replies SDP answer to LiveKit for the subscription

ah I see, that makes sense. I think you can do that pretty easily with ingestion. you'll need to send an <https://github.com/livekit/protocol/blob/main/livekit_rtc.proto#L17|AddTrackRequest> prior to offering tracks to LiveKit.

I'm not sure which platform you are building for, but others have had success with intercepting the offer via a proxy with our Go SDK.

thanks for the help, I will write the middle man in Golang

then on the subscription side, how to make LiveKit send out SDP offer to middle man?

I think AddTrackRequest is for publisher side

by default auto subscribe is enabled. you'd want to listen to <https://github.com/livekit/server-sdk-go/blob/main/signalclient.go#L32|OnOffer>.

LiveKit uses separate connections for publish and subscribe, so you'd want to implement `OnTrickle`  for both connections.

<https://docs.livekit.io/internals/client-protocol/|some docs> around the protocol

`LiveKit uses separate connections for publish and subscriber` , for connection do you mean different PeerConnections or connection for signalling?

found it on the documentation <https://docs.livekit.io/internals/client-protocol/>

assume  the following case. There are already two webrtc participants from browser, and then middle man joins the LiveKit room on behalf of Freeswitch. LiveKit creates two PeerConnections. PeerConnection 1 is for FS to publish its mixed audio to LiveKit, the second PeerConnection 2 is for LiveKit to send the audio of two webrtc participants to FS. So over PeerConnection 2, there are two audio streams from the two webrtc participants. Is my understanding correct?

yeah, it uses a single peer connection for all of the subscribed tracks.

per <https://docs.livekit.io/internals/client-protocol/#joining-a-room>, if auto_subscribe is not enabled when client makes websocket connection to LiveKit server, will LiveKit server initiate the subscriber PeerConnection and send offer to client?

yes, there's always a subscriber offer. that determines if the connection is established or not.

in this case, there is no SSRC in the offer because client does not subscribe anything, right?

then what will happen when the client starts subscribe to one publisher? Does LiveKit send a new offer with the SSRC of the publisher stream?

yup, it's just so we can deterministically report the client has a connection to the server

and yes, actual offer with tracks will come in when it's subscribed to them

thanks for the explanation, I am much clear now

based on my understanding, will the following solution work?

on the right side, there are three participants, each participant makes websocket connection to LiveKit server. The first participant only publishes, the second two only subscribes to one publisher

for the publisher-only connection, you'd want to create a token `canSubscribe: false`, that'll make it so that it wouldn't try to establish the subscribe peer connection

<@U01KZ61GV5Z> can your team to add an attribute in access token to disable data channel?

<@U043351G48Y> i think this is already there, do you need something different?
<https://docs.livekit.io/concepts/authentication/#video-grant>

basically I want a way to ask LiveKit to NOT add datachannel when create PeerConnection

especially for subscriber PeerConnection, so the SDP offer for subscriber PeerConnection does not have data channel part

Ok, yeah that’s different. I’m not the person to comment on that. I’ll defer to <@U01KZ61GV5Z>.

are you thinking for publish or subscriber connection?

for the subscriber connection, this isn't a simple change unfortunately

I hacked the code to disable data channel for subscriber connection. <https://github.com/kaiduanx/livekit/tree/livekitfs>

you can check the change, it is a hack :slightly_smiling_face:

the main reason why this is difficult to do well is it changes the connection lifecycle.

we use the placeholder datachannels as a way to test the peer connection before clients would report that they are successfully connected

<@U01KZ61GV5Z> please take a look at the above change. The idea is to treat user prefixed with FreeSwitch as special to disable data channel for it. With this way, I wrote a POC to integrate LiveKit with FreeSwitch where LiveKit works as webrtc SFU and FreeSwitch works as SIP/PSTN audio MCU. I will open source the POC soon.

will do! assuming the webrtc side is connected, how would signaling work here?

I wrote a middle man that uses signal client to connect to LiveKit and uses Verto over websocket to connect to FreeSwitch. The middle man passes the SDP between LiveKit and FreeSwitch

ohh, that's really clever! are you open sourcing the solution? there are a number of folks that would be quite interested.

talking with internal legal team to open source it, will share the update

I walked through the documentation, and can not find the answer to my question

Hey, I'm not sure if that's LiveKit related or Zeet. (Those guys are awesome!)

I've dropped them a note about this.