LiveKit Community

hi <@U05B9CM1VDK>, yes, this will be handled automatically.

Hi <@U01KZ0SCDKL>, I have a few more questions. In the end-to-end encryption mechanism, can I customize the key rotation time like Telegram?

I think you can call e2ee rechetkey in the flutter sdk to derive the next key. the decryption side will automatically rotate to the next one, but the current sdk does not have a timer to rotate the key. You need to add it at the app level

I could be mistaken, but I believe it’s already available for the JS SDK. cc <@U02EU2ZPTQU>

the PR isn’t merged yet, but there aren’t any major road blocks ahead. Projecting that the PR will be merged with e2ee support marked as “experimental” within the next 1-2 weeks.

These are both new features in client-sdk-js. It should be usable within components as is. (If you publish tracks manually. We can probably wire up an example with it moving forward. CC: <@U03RM7G8Q6A> <@U02EU2ZPTQU>

aiming to tackle the integration in components some time next week! 

Thanks <@U01KZ61GV5Z> and <@U02EU2ZPTQU> for your prompt response.

It would be super useful. Thank you <@U02EU2ZPTQU>.

we are publishing tracks manually and not using components. We would be interested in the example too

the work for this is being done in <https://github.com/livekit/components-js/pull/534>

Hi <@U04862SMJU8> — <@U02BH3ZHJ0Y> is a great person to chat with about this

<@U02BH3ZHJ0Y> hey Brian, can you tell me about HIPAA on the cloud as well

<@U04CBAARXKP> It is needed for clients to be able to receive data. It is a design decision. All our client SDKs handle it internally. Application should not need to do anything special.

Which SDK are using?

Thanks for the answer <@U02CM28PB4Y>
I'm using my own python SDK:
<https://github.com/dguerizec/livekit-client-sdk-python>
It's still a work in progress, but I'm at the bugfix stage. For now, it's only been tested and used as a video track subscriber for a project in my company.

Got it. Ccing <@U022KB72NFJ> who has been working on our native SDK for thoughts on how he is handling this. He is planning to have Python bindings. Theo, can you please comment when you get a chance?

Those bindings/SDK are here: <https://github.com/livekit/client-sdk-python>

thanks, I didn't know this project existed, I'll have a look

yes, should I send my questions in <#C025KM0S1CK|> or <#C04G4GE0RGX|>?

I think I'll have question for the python layer as well as the rust lib

<#C025KM0S1CK|> for the former, <#C04G4GE0RGX|> is best for Rust specific questions

image.png

<@U03T55WRVM3> can you expound on (2) where you say “connecting to the cloud often causes interruption”? What interruption are you seeing?

This might be an issue with connecting to Cloud from China. we currently do not have any servers within China and it's possible connections could become blocked/interrupted by the Great Firewall.

Hey everyone! Just popping in for them monthly e2ee progress check? :slightly_smiling_face:

<@U05DV9T716J> also said (pls thread related messages :pray::skin-tone-4: ):

&gt; Or at least audio processor/sink, I can use video processor to simulate e2ee but need also audio proc/sink.

<@U05DV9T716J>, Just created an E2EE PR for Android, it will take a few days.:slightly_smiling_face:
<https://github.com/livekit/client-sdk-android/pull/238>

this was likely due to how we tagged the last release. will fix it in 1.4.4

It seems that once a room was created, the first to join would bind said room to a node. Others, no matter what endpoint they gave, seemingly connected to the initial node.

If that is the case, I assume there is a record of what room is hosted where in redis, making other nodes check first, and then return already existing ones.

Should I be right, the how would such a setup ever happen?

that last image is LiveKit Cloud, the open source version does not support mesh routing

<@U059VU51WNQ> yes, it’s the same SFU, here’s a snippet from a <https://blog.livekit.io/scaling-webrtc-with-distributed-mesh/|post> we wrote about how we built Cloud:
&gt; Maintaining compatibility with LiveKit’s Open Source stack (OSS) was important to us. We didn’t want any developer locked into using Cloud, or needing to integrate a different set of features, APIs or SDKs for their application to work with it. A developer should be able to switch between Cloud or self-hosted without changing a line of code.
&gt; 
&gt; This implied, at a minimum, our Cloud media server needed to support the same open-source signaling protocol. One option was to privately fork LiveKit’s SFU, but we decided to adopt it wholesale as fixes and improvements benefitted both OSS and Cloud.
we’ve added orchestration code around the SFU to handle mesh routing, but the core media server is the same codebase

<@U01KZ0SCDKL> thx for the quick reply.
Did you use psrpc to integrate the orchestration logic with the existing codebase? And if yes the blog also talked about an abstraction that lets you treat a relay node as a participant, how would you do this with psrpc?

PS: are there any plans for a psrpc documentation, that part of livekit is just a blur to me :(.

Yeah, psrpc is a core part of the overall orchestration system. I’m not the right person to answer your question about treating relay nodes as a participant (cc <@U01KZ61GV5Z>).

Re: psrpc documentation, it’s on our (long) list of todos! We have a nice, meaty blog post planned for the near future diving into psrpc details under the hood. We’ve primarily been waiting until it’s battle-tested at scale and we’re at that point now. :slightly_smiling_face:

Looking forward to read more about psrpc. <@U01KZ61GV5Z> any other input about the matter would be appreciated
Thanks :pray:

our relay stack does use psrpc for cross node communication. but the actual media traffic flows directly without going through any message buses.

this is what we've said about the relay protocol in the post Russ linked to above:
&gt; In the end, we decided to use a custom FlatBuffers-based protocol, but like WebRTC, we still use RTP for media packets. A custom protocol allows us to supplement packets with additional metadata, like track identifiers. It also let us uniformly deal with packet loss between sender and receiver. In WebRTC, video, audio, and data channels have different ways of handling packet loss.

I still intend to use webrtc livekit for most parts of my app, but I am exploring HLS as a cost saving solution for mass output to public where the latency is less of an issue. In a recent test a few months back, streaming multiple high quality video streams via webrtc did work great, but ended up using a lot of bandwidth and was more expensive than I can justify (24 hours, 3 video, 1 audio, ~30 viewers costed about $160). Some back of the envelope comparisons to HLS made me consider this as an alternative since static bandwidth via object storage is so much cheaper than livekit webrtc.

There are 2 intended use cases - mass live streaming, and also as a failure-resistant form of recording, where if the egress fails for some reason you would still have the recording up to that point (unlike with mp4 files). We don’t have any additional features planned right now

RTMP out is also an option to look into for live broadcasting

<@U01KZ0SCDKL> would appreciate your response here if you can

Hey, it seems like you didn’t add a livekit webhook?

<https://github.com/livekit-examples/kitt/blob/24ffc914e65de967b166af11f15e2650a8330860/lkgpt-service/pkg/service/server.go#L66>

<@U05C2NE9UTS> please only post an issue in one channel, in a single thread and refrain from @-mentioning particular people unless they’ve jumped into the conversation with you. We try to maintain good etiquette here and provide community support free of charge. :pray::skin-tone-4:  

<@U022KB72NFJ> instead of "/webhook" I'm supposed  to add my url?

no, you need to configure your livekit-server to call this URL inside the config.yaml or inside the settings page if you’re using Cloud

Exciting news, LiveKit community!

We’re launching our new Egress/Ingress cloud dashboard page! This tool provides near real-time listings of your project-wide egresses/ingresses. It’s intuitive and designed to give you a quick overview as well as more detailed insight into specific egresses/ingresses on the detail pages.

Take it for a spin today and feel free to report any issues or feedback.

Cheers,

The LiveKit Team :rocket:

Hey David! Is Simulcast supported with WHIP in LiveKit? A WHIP-enabled, Simulcast-enabled build of OBS is now available: <https://github.com/Sean-Der/obs-studio/pull/2>

Wondering if it'll work 'out of the box'.

Think I found the answer to my own question - "Simulcast is not supported when bypassing transcoding."

<https://docs.livekit.io/server/ingress/#bypass-transcoding-for-whip-sessions>

that's right. Unfortunately WHIP does not have any signaling mechanisms for simulcast. I've asked the author of WHIP spec to add in descriptor headers in order to better standardize the behavior, but I don't think he's thinking about it.

on the other hand, if OBS behaves a certain way with simulcast, I imagine that'll become the implicit standard. if/when OBS finalizes that support, we can certainly add proper handling for it.

Good to know! I just tested Sean DuBois' latest build of OBS with Simulcast support streaming to his 'Broadcast Box' (<https://b.siobud.com/>) and it works great. Will be interesting to see how this all unfolds. Interesting times!

yeah, I have no doubt it can work. the lack of a spec about how it should work is the main thing holding us back right now.

Such a great article to read! :fire: Thank you for sharing this!

スクリーンショット 2023-07-04 午後4.23.18.png

which version of client are you using? this looks like an old bug that had been fixed awhile back.

Thanks for the reply!
Here is the version I am using!
```"livekit-react": "^0.9.2",
"livekit-client": "1.2.9",
"livekit-server-sdk": "^1.0.1",```

yeah these are all dated. please upgrade to the latest to fix the issue

<https://github.com/livekit/client-sdk-js/pull/557>

Update: turns out it doesn't, I just wasn't reading the code right, it has a specific FrameCryptor for sending.

hi <@U05FJ228MEY>

yeah, you got it right. Each participant has their own cryptor. The only thing that’s currently not possible is to have different keys for different tracks of a participant. But that’s also a non-goal currently.

However the sender cryptor part still feels a little clunky to me, too.
Especially how the publisher keys get set with an `undefined` participantId is less than ideal. The problem I wanted to work around was to be able to set the key _before_ someone actively joined the room. But at that time we do not yet know the actual identity of the local participant… The earliest we will know that is the `connect` call.

I did notice the sender key has an identifier of 'publisher' so I wondered what would happen if a participant had that ID.

I agree that this doesn’t look right, but isn’t really a concern, as `encoding` and `decoding` are two different paths in the worker. in `encode` `getPublisherCryptor` is called directly, which accesses a cryptor dedicated to the publisher which doesn’t overlap with remote participant cryptors (they’re stored in separate lists). The hardcoded identity is still not a good look though, will likely iterate on that…

here’s a table with an overview comparing OSS to cloud <https://livekit.io/pricing>

ChatGPT is definitely hallucinating here. 

Also, fwiw its information cut-off is Sept 2021 and we launched (open source) in mid-July, 2021. 

Hi there, currently reconnectPolicy isn't there for Android. you could handle this on your own by looking for when `disconnected` events are fired and seeing if it should continue to retry.

If I understand correctly, the behavior you are observing from remote clients is expected (because the user did indeed leave the room after giving up on reconnects, and then connected again)

And how would we retry? Right now we would retry by calling connect until it passes. That would result in us entering as a new participant but now both participants would not be subscribed to each other. What are the steps to properly reconnect once you get the disconnected event on Android?

&gt; That would result in us entering as a new participant but now both participants would not be subscribed to each other
why is that? do you have autosubscribe turned off?

Can you describe the steps we need to take when we get the disconnected event on android?

Screenshot 2023-07-06 at 11.21.43.png

This message contains interactive elements.

To add to Jaspers comment. All we did on android side to do this is call connect once on disconnect event.

yes, I understand that. I'm asking how it would result in participants not be subscribed to each other? That part isn't clear to me.

the left + rejoin behavior is expected if one of the client became disconnected for a period of time that exceeds the automatic reconnect thresholds

Well that is the part we are reaching out for help. We use all default values on livekit android sdk, meaning autosubscribe should be on. My assumption is that we are not following the right steps to reconnect. All we do is call connect with serverURL and the same token we used in the start of the call. We are not republishing anything or reading the track from the remoteParticipant. Should we?

Here is a list of the livekit events:
```Call starts
11:09:41.947  ConnectionQualityChanged@57b245b
11:09:43.097  TrackPublished@7328196
11:09:43.102  ConnectionQualityChanged@4011117
11:09:43.249  TrackSubscribed@8bb8ce9
11:09:47.774  TrackPublished@d08f61c
11:09:47.936  TrackSubscribed@f714587
11:09:48.545  TrackStreamStateChanged@98eab4d
11:09:51.869  ConnectionQualityChanged@8a95258
11:09:51.870  ConnectionQualityChanged@f3f0fb1
11:09:56.888  ConnectionQualityChanged@b8e7633
11:09:56.889  ConnectionQualityChanged@21e0df0
Internet turned off
11:09:57.996  Reconnecting@c0f3b4e
11:09:59.372  TrackUnpublished@f2c486b
11:09:59.373  TrackUnpublished@1ad9fc8
11:09:59.373  TrackUnsubscribed@a4c2861
11:09:59.373  TrackUnpublished@2c0ef86
11:09:59.393  TrackUnsubscribed@3e0f074
11:09:59.394  TrackUnpublished@dc1e99d
11:09:59.395  ParticipantDisconnected@d2c7e12
11:10:36.506  Disconnected@bfc748c
11:11:16.341  ConnectionQualityChanged@87326cd
11:11:16.344  Reconnecting@e682b82
11:11:16.490  TrackSubscribed@4bb6285
11:11:16.595  TrackSubscribed@25810b
First reconnect
11:11:16.642  Reconnected@c829039
20 seconds later
11:11:37.237  TrackUnsubscribed@6deeb3d
11:11:37.243  TrackUnpublished@6c4ef32
11:11:37.282  TrackUnsubscribed@42a4600
11:11:37.283  TrackUnpublished@b53fe39
11:11:37.283  ParticipantDisconnected@34ecc7e
11:11:37.579  TrackPublished@4e42ac4
11:11:37.580  TrackPublished@1c61dad
11:11:37.634  ConnectionQualityChanged@553e7a9
11:11:37.747  TrackSubscribed@13a5806
11:11:37.751  TrackSubscribed@113ee92
11:11:38.101  TrackStreamStateChanged@2712042
11:11:41.878  ConnectionQualityChanged@551e957
11:11:41.879  ConnectionQualityChanged@9430c44```


thanks for confirming autoSubscribe is on.

Is the publisher reconnected? or is it the subscriber?

it is possible that the track is somehow not republished correctly after reconnection.. if the issue is with the publisher reconnecting

arent both participants both a publisher and subscriber? Its a video call.

I understand.. but I don't think both participants are disconnected at the same time?

No, in the case we are testing only one participant is disconnected.

ok, so when that participant (call it A) is disconnected, is the other participant (B) able to see A's video? and after A is reconnected, could it see B?

No, neither see each other for 20 seconds. once 20 seconds pass then they start seeing each other

You can see the 20s delay both in the event logs, as well as in the screenshot jasper sent, where you can see two participants joining. One 20 seconds later

So Android participant reconnect, web participant was always connected. They dont hear or see each other. 20 seconds pass. The events happen and then they see and hear each other.

understood now.. the issue here is a 20s delay after the manual reconnect. we'll take a look at reproducing it

And also that there is a an extra participant reported as connected but who never publishes tracks

hey sorry for lack of response on this, still looking into this

hey I think I have a fix for this, try out version 1.2.2-SNAPSHOT (you may need to add `maven { url '<https://s01.oss.sonatype.org/content/repositories/snapshots/>' }` as a repository in your build.gradle file access it)

Screenshot 2023-07-12 at 14.28.45.png

First of all thanks for looking into the issue :pray:
We just tested it and seems the same behavior is still present. There is one extra participant joining which never publishes any tracks. The participant is then disconnected and a new one joins which is publishing tracks as expected. (one improvement noticed may be that the android user now sometimes gets a chance to subscribe to the remote participant tracks when reconnected)

When you are testing your updates. It would be nice to know your exact flow for reconnecting. Do you use `connect` ? Do you connect with a new token? Or reuse same one? Do you manually cleaning up somethings when disconnected etc. Cause it might very well be that we are doing something in wrong order or missing out on something?

<@U05FZRAS1S5> for the testing, I was calling `room.disconnect()`  (even though it was already disconnected) and then `room.connect()` with the same token. This is more that the sample code was coded this way and I reused it for testing, rather than how it’s intended though. can you describe how your reconnect flow works or provide a code sample?

```is RoomEvent.Disconnected -&gt; {
    connectionChange(false)
    coroutine.launch {
        tryReconnect()
    }
}
suspend fun tryReconnect() {
    try {
        room?.connect(serverUrl, token)

        connectionChange(true)
    } catch (e: Exception) {
        delay(1000)
        tryReconnect()
    }
}```
I turn off wifi on my android device until i get disconnected event then I re enable wifi.

connectionChange function is just to update UI

gotcha, I’ll test this out a bit more. for reference, my reconnect test was registering a network change listener like so:

```private val networkCallback = object : ConnectivityManager.NetworkCallback() {
        override fun onLost(network: Network) {
            // lost connection, flip to reconnecting
            hasLostConnectivity = true
        }


        override fun onAvailable(network: Network) {
            // only actually reconnect after connection is re-established
            if (!hasLostConnectivity) {
                return
            }
            LKLog.i { "network connection available, reconnecting" }
            reconnect()
            hasLostConnectivity = false
        }
    }```
```        val cm = context.getSystemService(Context.CONNECTIVITY_SERVICE) as ConnectivityManager
        val networkRequest = NetworkRequest.Builder()
            .addCapability(NetworkCapabilities.NET_CAPABILITY_INTERNET)
            .build()
        cm.registerNetworkCallback(networkRequest, networkCallback)```
this should be more effective than trying to reconnect every x seconds

Can I see the content of reconnect function?

```fun reconnect() {
    room.disconnect()
    viewModelScope.launch {
        connectToRoom()
    }
}


    private suspend fun connectToRoom() {
        try {
            room.connect(
                url = url,
                token = token,
            ) 
            // ui update setup etc.
        }
    }```


<@U01NZKTNE1H> Have you had time to look into this?

<@U05FZRAS1S5> sorry, haven’t had time to check into this deeply, it’s still being tracked though

by services, we provide hosting of the APIs that you are referring to. <http://livekit.io/cloud|livekit.io/cloud>

thanks david , thats really useful.

Something else i'm trying to work out is if you can sport Okta CiC to provide MFA based logins for end users. I can see you have callbacks for your auth so i think it can be extended but i'm interested to know if you have tried this yet ?

sure! with our authentication model, you just need a backend component that can validate the okta authentication and then create a JWT access token with the identity of that user. you can read more about issuing access tokens tokens <https://docs.livekit.io/concepts/authentication/|here>

you can use <https://docs.livekit.io/client-sdk-js/classes/LocalTrack.html#getDeviceId|getDeviceId> to retrieve the current device on a LocalTrack

Hey Does livekit have any Flutter tutorials?

there was a regression with `--bind 0.0.0.0`, it's just fixed in `master`

Hello <@U05FUDL1QLX>, we kindly ask you to post your questions in threads on the <#C01KVTJH6BX|> channel. We will try to get back to you quickly, but sometimes it may take some time. Pinging us via PMs will not get you an answer faster, so please be patient.

Hi folks! :smile:

I want to share a small project I’m working on in my job.
It is simply a proxy to receive the webhooks from LiveKit and send it to a GCP PubSub Topic.
I think is a good add to any LiveKit deployment on GCP to allow integrate with other components or services.
Let me know what you think and PRs are more than welcome :smile:

<https://github.com/wolmi/livekit-webhook-proxy|https://github.com/wolmi/livekit-webhook-proxy>

before the Room state is `connected`, `ParticipantConnected` would not be emitted for any participant who's joined before the current user has joined. This behavior is documented <https://docs.livekit.io/client/events/#events|here>.

when the `connect` promise resolves, it'd be a good practice to look into existing participants in `room.participants`

Seems that I encounter the same issue, in my app all users join room at the same time, as a result, users that join BEFORE the current user will be listed in `room.participants` and handled properly, and users that join AFTER current user will be handled via ParticipantConnected event, but somehow users that join DURING the current user joining room are missing. Any idea?

that should not be happening.. can you share a code snippet where you can reproduce the issue?

They are mentioned here <https://docs.mux.com/guides/video/spaces-to-livekit#introduction-to-livekit>

It is true, we’ll say more publicly about this in a little while. :blush: 

<@U01KZ0SCDKL> Great :airplane_departure:

Image from iOS.jpg

The last time we did an analysis (I don’t think this has changed significantly):

Ok thanks.  That is similar to what we determined as well.