LiveKit Community

Wow, this looks super cool <@U047WL5LPNE>!

How are you pulling in the video into Quest Pro? Is it using the Android SDK?

<https://glitch.com/edit/#!/livekit-demos|https://glitch.com/edit/#!/livekit-demos>

ah, so it's an embedded browser within the quest app?

Here’s a glitch page - just set the server url and access tokens in livekitConfig.js

Hi <@U03S00K6BD5>
thanks for your thoughts on this.
We have some video-processors-like stuff in the works. I’m not sure what twilio’s implementation looks like, but we basically only define a pipeline of stream transforms without any other assumptions. those should be easily extensible/adaptable as long as stream transforms/insertable streams are supported.
The idea of being able to override what `restart` should do is interesting however.
Could you add a feature request for it on github? then we could discuss further there and explore options.
Also would be great if you could open a bugreport for the attachElement bug you’re seeing.
thanks!

haven't tried running on expo go yet, but the webrtc library we rely on specifies some extra setup required for expo go:

<https://github.com/react-native-webrtc/react-native-webrtc#feature-overview>

ah ok thanks, i will check this out. I assume it works with EAS builds?

can’t give any hard guarantees since I don’t have any experience with it, but there shouldn’t be issues as far as I can tell since it seems to handle native modules fine

I have managed to setup a React Native Expo app with all dependencies, runs fine on expo build. Most of the demos you have are in Typescript. Do you have any JS examples (not typescript)? as there is no documentation?

afaik, the typescript examples should largely be usable in javascript, minus any type declarations

Thanks mark, official announcement coming soon! :slightly_smiling_face:

I'm not aware anyone posted this here, but I think <@U01KZ61GV5Z> (I hope you're the right David :smile: ) did a great job explaining what the difficulties of WebRTC are and how LiveKit solves them.
It's something you can even show managers and they'll understand their share of it :smile::+1:
<https://webrtc.ventures/2022/10/webrtclive-livekit/>

Hey Vlad, thanks! For audio on mobile, you don't have to attach anything. The WebRTC stack handles mixing &amp; playback automatically.

This means you can directly ingest streams from OBS studio and any production workflow that supports RTMP!

Another thing to note. In order to use Ingress, you'll need LiveKit server 1.3.1 or higher.

I’ve got one question to that. Does the ingress “participant” count towards the `maxParticipants` limit of the room or not?

Would help if we get a sample.json for custom encoding types/stream sizes. Looks like default is using h264 with simulcast.

I can look into code, and figure out the format, but I’m really happy with how it looks.

Already updated my deployment and tested it, just a cpu usage of that Transcoding for simulcast is a bit more than I can afford.

<@U03L2F74CF7> yeah it does count towards `maxParticipants`

<@U03UHEVPUHW> we'll prepare a full example shortly. The full request is documented in the <https://github.com/livekit/protocol/blob/main/livekit_ingress.proto#L20|protocol here>. To disable simulcast, you could declare a single output layer in `IngressVideoOptions` .

<@U01KZ61GV5Z> already figured that out. Luckily I can read `.proto` files :slightly_smiling_face:

How can i use this functionality with cloud version and livekit-server-sdk ?

Very interesting, thanks for sharing this Jamie! :pray::skin-tone-4: 

Np. Atm ppl need to use osx apps etc with embedded srt players written in rust to use use these feeds real time, rather than just using webrtc for last mile.

+1 to SRT ingress. We work with the Hollywood industry and SRT is a key part of our media pipeline due to its robust security and error correction.

<https://github.com/livekit/protocol/blob/main/livekit_ingress.proto#L37> :wink:

For these workflows, how does it typically get set up? Is it usually programmatic?

Do folks generally use pull (media served on an srt:// endpoint where ingestion fetches) or push (media producer pushes data to an srt:// server) ?

Yea, our content authors create a new stream in our dashboard, which generates an endpoint for them.

We typically do push from the media producer to avoid firewall issues.

there are quite a bit of variance in speaker detection implemented by various providers. By default, ours models after Zoom in behavior, where the slight delay is to prevent false positives in detection.

you can tweak these in <https://github.com/livekit/livekit/blob/master/config-sample.yaml#L149|the config> to your liking

Got it! Thank you! Is it possible to change these values if I use LiveKit Cloud?

not today, eventually it'll be more customizable

image.png

do you have a prometheus instance running ?

<@U02EU2ZPTQU> no .Only the prometheus metrics are running . How to create that?

in order for grafana to use prometheus as a data source you will have to actually have prometheus running somewhere. prometheus is the service that you will want to point to your livekit server metrics. <https://prometheus.io/docs/prometheus/latest/getting_started/>

Hi Parrish, this should be totally doable if you run LiveKit Server with Redis. Check out our <https://docs.livekit.io/oss/deployment/|deployment docs>

2200 participants may equal 2200 streams published if you consider audio and video as a single stream but does not equal 2200 streams consumed

total streams consumed will be 2200 -1  streams per participant which equals 2200*2199 streams from the servers end which is a looot

<@U040XJTFZC4>
I'm sorry I was not clear earlier, each stream published just has 1 consumer, there is no audio.

<@U04BE2DEV0U> I would have expected it to scale roughly linearly for your use case of one publisher/one consumer.

With your packet rate (approx. 88K in and out, 300 kbps = @1000byte video packets = 40 pps) and bit rate (approx.  750 Mbps in and out), it should be doable on a large server.

For example, we have published load test results here - <https://docs.livekit.io/oss/benchmark/>. We have not run that recently. So, maybe there is some issue in the latest code. You can see that it is doing higher numbers on a 16-core (note the page shows MBps)

Also, interesting in understanding your need for static IP. Is DNS not an option?

Also, is it possible to run the load test in your environment to get a feel for what the limits are? The page I pointed to above shows how to run the load test.

The data rate was near about that, but cpu usage was very high. We have run the load  previously, but received results that it should have been possible. But in the production release it didn't seem to work out that well. According to you vertical scaling should not be an issue.

I'll give some context of our use case, our room size is fixed to a specific no of people, with seats of participants also fixed. So we cannot have the typical distributed setup as the seats may go empty if more rooms are created in the same instance then what we planned, as I understand room creation logic is based on CPU load.

In our design for a distributed system in which each SFU doesn't need to communicate with one other as they cater to fixed publishers and subscribers we disabled redis. We have external load balancer for signaling connection, but for turn servers we have to provide separate URLs as it is embedded  and does local relaying only, we use static IPs as it takes a long time to associate an IP to domain name and be accessible for all our users.

I'm unable to understand the TURN setup, and how would it would work in a distributed system with a single URL. That is why we went ahead scaling our serer vertically, and static IPs.

It would be really great if you could guide us on this, thanks.

TURN setup works via a load balancer, any TCP load balancer would do. If the load balancer supports TLS termination, then it could be done there instead of having to distribute the certificate everywhere.

Thanks David, we having a little problem understanding how TCP load balance would handle this.
For example:

• We have 2 EC2 instances each running a livekit instance (Kubernetes is not used)
• Both instances are running in the same region and the same VPC, and have of course a public IPV4 address
• Both livekit configurations have the turn server enabled, each one of them we have given a separate url. e.g. <http://turn1.example.com|turn1.example.com> and <http://turn2.example.com|turn2.example.com> 
So now if I understand you correctly we can put a TCP load balancer in the system e.g. listening to <http://turn.example.com|turn.example.com>.

Can we then simply define two upstream servers in the load balancer, and then it forwards the requests wit e.g. round robin to the two turn servers?

Will the turn server from <http://turn1.example.com|turn1.example.com> be able to route the traffic to the other livekit instance which is running on a different machine?
Is that how it would work?

&gt;  Can we then simply define two upstream servers in the load balancer, and then it forwards the requests wit e.g. round robin to the two turn servers?
yes and no..

typically this would be the case. However, I'm reading the thread further and seeing that you are not running livekit in cluster mode (i.e. sharing a Redis message bus). In your case, you cannot rely on a single load balancer across these TURN servers, because they are not aware of the rooms created on the other instances (and therefore cannot authenticate those users)

I've re-read carefully the docs - I understand the first question is  yes :slightly_smiling_face:
So only the second question has left

It’s pretty stable, the chrome egress types (web and room composite) have been much less troublesome/error prone than the other types

Is it possible to add api (in a fork) for egress server - to eliminate the need of launching livekit server?

I believe <@U01UB0LHTTQ> might be working on something to enable this already? (I might be wrong)

Yes it's possible, what I added recently was web egress, which records any website and is not related to a LiveKit room

You could modify the service to accept grpc requests instead of receiving requests through redis

this isn't possible. we are limited by what WebRTC supports here. 

can I select the channel to use on a virtual input device for a track using swift or js apis - for example BlackHole has 16 audio channels

hmm, alas not. <https://bugs.chromium.org/p/chromium/issues/detail?id=453876>

currently this is only possible with Cloud. Prometheus metrics do not expose per-room data

<@U01KZ61GV5Z> Other than cloud, any third party tools? To check bandwidth usage

I’m not sure if this helps, but you can check overall server bandwidth with prometheus (code <https://github.com/livekit/livekit/blob/master/pkg/telemetry/prometheus/packets.go|here>)

<@U02C10ZEP8R> this may be of interest re higher audio quality - noticed your issue on GH. I'm having the same difficulties with stereo (when using swift, js is working ok). <https://github.com/livekit/client-sdk-swift/pull/160>

Hi Federico! The intention is that YOUR server generates tokens, only after your server authenticates the client user.

(I know this is a little confusing; we plan to make the docs more clear on this point)

You could make a server that just spat out random access tokens but then you wouldn’t be able to identify the users when they connect

So the idea is to have a webhook to the livekit dahsboard, and then make requests to it, and get tokens that way?

I'm trying to use Livekit cloud, that's why I'm asking this

for context, are you already running a server for your project, or are you only writing unity code so far?

So far, unity code, and the livekit cloud dashboard setted up

So, how most LiveKit Cloud projects work is:

• there’s a custom server that handles login and other application state.
• when a user logs in, the client authenticates with the server
• when the user wants to join a room:
    a. the client makes a request to the server, which generates a token (using, for example, <https://github.com/livekit/server-sdk-go/blob/30a25d629a7aef3993a2220142f0019c668e1e3a/roomclient.go#L123|our Go server SDK’s CreateToken> method)
    b. the client uses that token from the server to connect to LiveKit

For your project, do you plan to add user logins?

at some point, yes, but at the moment for testing, we can generate a bunch of tokens, and just input them manually on the UI for joining then

In that case, I will point out that token generation is a pure function that takes your API key+secret as inputs

So we'll just skip the UnityServices call, and create a stub with the same structure, and feed that with a hardcoded token from a list

So, in case this is more convenient, you can <https://docs.livekit.io/references/server-sdks/|use one of the server SDKs> to generate tokens without hitting LiveKit servers. Just thought I’d point that out in case that’s easier than generating them by hand in the dashboard

but yeah when I’m developing with our test clients I just use the dashboard to manually generate tokens.

So, by replacing this string with the token, instead of getting a token from the server, if I sequentially select tokens from a list, then for testing that should be ok

and then switching the wss link on the network manager

no problem! let us know if you run into any more trouble

Does it capture stereo when you are using a microphone device? (not virtual). I have a sense the core issue is <https://github.com/webrtc-sdk/webrtc/blob/m104_release/modules/audio_device/mac/audio_device_mac.cc#L667|here>. SetStereo might not be called.. but <@U02MAJLGN65> is the expert here.

I forced stereo everywhere I could still didn’t work via swift (I think it’s at the Weber’s interface). I think all mics are mono at least the built in ones and most studio microphones - I ended up googling this yesterday as wasn’t sure either but they’re naturally mono devices 

I’ve been testing with blackhole the osx virtual device via Logic Pro 

yeah that makes sense.. this path might not be used.

will need to dig in further to see how it'd need to support stereo input.. in the mean time, can you open an issue under `client-sdk-swift` describing the request for stereo input? :pray:

スクリーンショット 2022-11-22 11.30.08.png

Hey, thanks for bringing this to my attention. I might need to patch WebRTC.
I think stereo recording can be enabled by default when the device supports it.
According to system report, my MacBook Air doesn't support stereo input.

<@U02MAJLGN65> the rest of us don’t speak Japanese :joy: but I trust that the screenshot says your MacBook Air doesn’t have a stereo mic :laughing:

I wonder whether the iPhone mic is stereo

<@U02MC5V3C2H> Almost forgot this world is dominated by English :joy:
Yea but the "1" is "number of input channels" .

all mics are mono guys, condenser mics or mac air mics it doesn't matter. that's fine - what I'm interested in doing is streaming a mix from a DAW like Logic with a screencast and having a studio quality mic record into that. End of day the mono mic is what matters, but a mono mix is rubbish for the artist recording it, they need to experience the mix as they're singing. All this works even using the sample app in client-js - but it "aw snap" crashes randomly in all browsers at higher bitrates. Whereas macOS is rock steady, but virtual devices are mono and mono at 300kbps sounds like stereo at 30kbps

(its stereo in js - just the crashing issue)

interestingly.... swift client as it is doesn't downmix it just discards the right channel on virtual devices.

Hiroshi check it out with <https://github.com/ExistentialAudio/BlackHole> as a virtual input - all the livekit sdks pick up the input (the first 1-2 chan).

WebAudio is limited to the first 2 chans but that's ok - it gets the stereo

I read "The newer models of the MacBook Pro actually have 3 microphones built-in", I supposed it's used for some sort of signal processing.
<https://www.macbookproslow.com/macbook-pro-microphone-location/>

I created an issue here to track stereo input:
<https://github.com/livekit/client-sdk-swift/issues/161>

Hey all,
We’re excited to announce some new egress modes - Auto Egress and Web Egress :tada:
• Auto egress: Instead of waiting for track_published web hooks to export every track, you can now set auto egress options as part of a CreateRoom request, cutting the number of round trips and total delay for egress to begin (<https://docs.livekit.io/server/egress/autoegress/|docs>)
• Web egress: similar to room composite, web egress uses chrome to record a website. However, with web egress, you are in full control of what and when to record, as it’s not tied to the lifecycle of a LiveKit room (<https://docs.livekit.io/server/egress/web/|docs>) 

Hey there, we're currently working on e2ee and plan to ship early next year. Dashboard/analytics for open source is a bit more ambiguous because we're making substantial changes to our Cloud Dashboard product as we test it with users.

Mind if I ask, what do you hope to learn from a dashboard/analytics for LiveKit? What decisions would you be able to make if you had a perfect analytics dashboard?

<@U02MC5V3C2H> More hopes are that in addition to a dashboard, there are also controls for managing the livekit delpoyment.

See my post from a Late October for more details about what I hope to be able to do with livekit <https://livekit-users.slack.com/archives/C01KZ61HZRR/p1666236899197359>

usually the token generation for an app has some constraints that only the server/db side is aware of (e.g. only supplying tokens for valid users).
that’s why there’s currently no general way to just request a token directly from the cloud dashboard via an API.
what people usually do is have a server side route somewhere that performs the necessary checks and then issues a token.
you could do this in any of the available server sdks.
For nodejs you could also run this inside a serverless function so that you don’t have to spin up a server instance just for your token generation.

for token generation that server script will only have to know the api key and secret. no other hooks are necessary, the token generation itself doesn’t have to communicate with the livekit (cloud) server

Ah understood! Thanks for the clarification!!

the server sdks each have the ability to generate tokens locally. for example: <https://github.com/livekit/server-sdk-go#token-creation>

Hey Gathsara, there isn't a way to revoke an access token. The best practices are to issue a token with a short ttl each time. so it'd be expired shortly after.

Hey <@U01KZ61GV5Z>, I'm curious about this one also, if we have a short TTL, while on the call, the token will automatically be refreshed when needed right?

Yup, it'll get <https://docs.livekit.io/concepts/authentication/#token-refresh|automatically refreshed>.

Awesome. Thats what i was thinking. Simply wanted a confirmation. Thanks <@U01KZ61GV5Z> :)

Hi, are you thinking about using Egress for this to process the tracks afterwards? Currently there isn't a way to do this.. What are you thinking about doing with the images generated?

I would need to store them in shared memory for immediate processing (analysis and composition) with as little overhead as possible.
So I was wondering if the tracks get transcoded there could be a way to extract the raw frames somewhere.

Gotcha. I think the best approach is to connect to the room with the Go SDK, and decode the tracks yourself. Egress does this as well, and you are welcome to use code from the project as a starting point

Egress can export tracks as HLS segments to S3.. so you could add a S3 listener to notify you when new segments are uploaded. This would be simpler to build, but does come with about some amount of latency (you can customize segment length, latency is segment length + ~200ms).

Hey Neal, we’re planning to add WHIP support before we promote “universal ingress” to GA. So, prob within the next ~couple months.

Yes, you’re correct. Someone in the audience would be forwarded media streams from your LK server instance. That’s egress bandwidth and you pay for that on AWS (or most/all other cloud providers).

Yup! We’re aiming for a developer preview of Unity Multiplatform by the end of the year. :crossed_fingers: 

Sounds great! So I assume it will possible to integrate it in a VR scene without webGL?

This isn't supported currently. It sounds useful though. We'd welcome a PR for it if you are up for it.

thanks for the confirmation <@U01KZ61GV5Z>

on further inspection of the source I can likely get bit by leveraging the `LIVEKIT_KEYS` and `REDIS_PASSWORD`  environment variables