LiveKit Community

currently data messages aren't sent via webhooks (it'd be too noisy today)

however, you can get them by connecting to the server with our Go SDK as a client participant. you can create a hidden participant so it's not visible to others, and join the room. It'll receive all updates like any other client

thanks <@U046B0LNLD7>, we definitely not want to use webhooks to capture all message and do corresponding injection and moderation due to http post latency and delay.

also like using GoSDK  or other server SDK to join as hidden participant for capture message after Signaling/media server handling it coz other participants already received those messages.

I saw we have rtccomponent for handling message but it’s not a good place for me as well to decode message at media server stage which some chat message may not need to hit it.

i’m afraid we have to setup intermediate websocket services act as middleware layer to do the chat moderation in future (since chat message don’t need to add location metadata for location awareness to reduce the message size), but i guess *shoplive cloud* already provided such capability, may I know that as well?

no matter moderation or rich-able message types support in future, the capability to capture message at server side from all participants for live room is very nice to have, that’s why i’m looking for livekit/protocol to see i could setup GRPC connection by custom room manager to *add some handlers* which will be serialized when room deployed by SFU servers, then every server sdk could have such capability for integrating with their existing services base on language preference, e.g. Live bidding scenarios,  do you have any thought and idea?  sorry if i’m missing anything here, thank you so much!

for moderation, do you mean that each message needs to be reviewed before the other participants would receive it?

yes, not only for moderation purpose but also for the capability for capturing all messages on SFU servers side before sending to participants

e.g. if we have different message protos, client side could serde for corresponding actions, so does server side which could take action regarding different type messages, i also noticed ant media using rocket chat just now to see how it’s going on at that side.

i guess client side onMessage etc. could do client moderation, so if server side can’t achieve that, that’s still fine, but i’m trying to see i could get all message at server side by using server sdk instead of changing source code (i’m not a go developer yet ^ ^)

yeah the server SDK would be a better path.. we don't typically recommend changing the protocol mainly because it would be a maintenance nightmare (mostly for you :joy:).

though if there are patterns/suggestions that could make it easier to build what you are looking to do, please let me know. we can definitely improve the protocol to make it more useful for these use cases

for moderation, two ideas:
1. client sends to moderation server before posting to the data channel
2. client posts to your server, which uses our server API to broadcast messages to all clients
3. recipient of messages deciding if the message should be displayed.

yup, i kinder feel middleware layer for another WS socket server as moderation/chat server and map to the intiallized rtc conn data channel for broadcast message to all participants, that’s also i’m wondering i could use rocket.chat or any other opensource to act as the middleware layer

but not sure i have enough efforts and deadline to finish all stuffs, and thanks for your informative reply, i just look into realtime communication stuffs now, but I guess we definitely need to use livekit server API to broadcast msg

the reason is if we want to have custom template for capturing chat layout with egress, we need livekit SFU servers to get those messages for room composite, since i noticed someone mentioned the egress for including chat is in roadmap before.

ah yes, you can already do that today. as long as you put together a <https://docs.livekit.io/server/egress/custom-template/|custom Egress template>, it should capture an identical experience as what the end user would see in a browser.

so it should work regardless of the approach you end up taking with Chat.

<@U01KZ61GV5Z>
<@U0505UX26H3>
If we have an hidden client listening on the server side, will that hidden client be in the Participant List on another client in a way that we can find it by name?  We are trying to determine if we can set up defacto data paths by hidden users.  For example we would have a hidden user called "Chat Moderation" that we would send all chat messages to for processing.

<@U01KZ61GV5Z>
<@U0505UX26H3>
If the hidden participants are not in the client participant lists, can I still send a message to the hidden client if I have their sid?

<@U052PG0JJUT> that is a good question. I don't believe hidden participants are visible to anyone else. But you can still use the server APIs to list for them. So there is a way to retrieve their Sid, but the process isn't straight forward.

we do intend to make it so you could send data messages by participant identity as well. but have not yet implemented that change

<@U01KZ61GV5Z>
<@U0505UX26H3>
Do you have an idea when this work will be complete for the Android SDK?  Thanks in advance.

I don't have an estimate currently. juggling around a bunch of things, but it's definitely on our radar :slightly_smiling_face:

it'll subscribe to the highest resolution layers by default. are you seeing a different behavior?

yes, sometimes it takes MEDIUM also .....so i would like to understand what is the criteria to select resolution. Also in what condition Auto switching to video stream to high-resolution.

unless you change anything, it'll subscribe to the highest resolution assuming your bandwidth allows for it. if server detects network congestion, it'll have no choice but to fetch a lower resolution layer

so it all depends on network being used ? we can't do auto-switch or force-transition a video stream to high-resolution.I am correct ?

if your network can support 1Mbps, and you are trying to send 1.2Mbps of data.. something has to give.. you would either get choppy video or the bitrate would have to come down

Friendly reminder to thread your messages please, <@U03G3V85NHH> :pray::skin-tone-4:  

This isn’t quite enough information to help you figure out what might be causing a crash. Could you share some more specifics? Code and/or logs?

Hi <@U03G3V85NHH> that's not great, can I have the exact version you are using ?
Also what device can this be reproduced on ?

func switchCamera() {
        guard let track = room.localParticipant?.localVideoTracks.first?.track as? LocalVideoTrack,
              let cameraCapturer = track.capturer as? CameraCapturer else {
            return
        }
            cameraCapturer.switchCameraPosition()
    }
10:29

info LiveKitSDK : CameraCapturer.startCapture() starting camera capturer device: &lt;AVCaptureFigVideoDevice: 0x108274400 [Задня камера][com.apple.avfoundation.avcapturedevice.built-in_video:0]&gt;, format: (format: 'vide'/'420v' 1280x 720, { 1- 30 fps}, photo dims:{1280x720,4224x2376}, fov:70.291, supports vis (max strength:Low), max zoom:123.75 (upscales @3.00), AF System:2, ISO:33.0-3168.0, SS:0.000014-1.000000, supports HDR, supports multicam, supports high photo quality, dimensions: __C.CMVideoDimensions(width: 1280, height: 720)), fps: 30(1...30)

 warning LiveKitSDK : LocalTrackPublication.recomputeSenderParameters() Cannot re-compute sender parameters without dimensions

The devices that demonstrate this case: IPhone 12 (IOS 16.1), IPhone 13 (IOS 16.3), IPhone 14 (IOS 16.3), IPad 11 Pro (4Gen - IPadOS 16.0)

in the process of testing app using camera approximately 30 times (switching from front to back camera) the app crashed. Another way of getting rhis type of crash (in the faster way) is putting the phone on it's back (1-10 switches need).

looks like another switch camera request is executed while previous one is not completed. Switching camera or each API executes a complicated set of operations and you must always wait until they finish.

Can you update your code like:
```func onTapSwitchCam() {
  // disable UI for now
  cameraSwitchButton.enabled = false
  switchCameraPosition().then {
    // re-enable your switch camera button
    cameraSwitchButton.enabled = true
  }
}```

About SDK version - sorry can't understand - where to see it - in package README wrote so

```swift title="Package.swift"
let package = Package(
  ...
  dependencies: [
    .package(name: "LiveKit", url: "<https://github.com/livekit/client-sdk-swift.git>", .upToNextMajor("1.0.0")),
  ],
  targets: [
    .target(
      name: "MyApp",
      dependencies: ["LiveKit"]
    )
  ]
}
```

Can you try disabling the UI until the promise of `switchCameraPosition()` is complete ? like the sample I wrote above.

Please let me know if that works.

I believe it's 12.1 according to <https://github.com/livekit/client-sdk-flutter#ios|this section>

<@U01KZ61GV5Z> if i could get any input or plan or answer from you, that’s helpful, thank you!

Hey, we do not support plugins at the moment. Agreed as much as possible we'd like to keep user security outside of livekit itself so the application could have full control. The current approach is to use JWT tokens.

Thanks <@U01KZ61GV5Z>, i noticed GPT participant is using pion rtcclient to join the session for being as participant, may I know any plan for participant creation and joining the room for capture message at <https://docs.livekit.io/server/room-management/|Livekit Server SDK> side which helps to create wanted specific type participant (like GPTParticipant) for message handling, thank you!

another question is we still want to leverage livekit as more as possible though we are in design phase, and normally chat/message system is another vertical technologies compared with media stream distribution,
May I know livekit has labeled signaling servers which are responsible for message broadcast or not? then the middleware server might not be needed but instead scaling those “labeled” servers for using livekit pub-sub capability for message broadcasting still;

Or you suggest users to come up separate message system by integrating with livekit servers who will focus on media stream distribution more (e.g. ant media + rocket chat mode), appreciate for any insight and this also helps our plan more!

<@U0505UX26H3> you can indeed act as a participant on the server side with the <https://github.com/livekit/server-sdk-go/|Go SDK>.

I'm not sure what you mean by the second part. While LiveKit isn't designed to be a generic messaging solution, data channels should give you some functionality around message delivery to end users

hi <@U01KZ61GV5Z> thank you, i knew go sdk’s Participant sdk helps for that but it’s not exposed through by livekit/protocol as a common func regarding all servers sdk common func, may i know any plan of that; and yes, we will use livekit broadcast func by holding data channels;

it's not simple to expose this functionality to the server-side because it requires a webrtc engine underneath. We are working on a similar SDK in Rust. There are plans to create a C++ wrapper for the Rust SDK, and Python as well (down the road).

i think so too, the reason is team wanted me to see possibility for other sdks like kotlin/java which could save additional service for integrating with existing java based domain service, but actually go sdk is directly leveraging underneath webrtc engine for that like you mentioned instead of across languages’ protocol way. I guess we have to add additional go/python service and move some function there for that, i will let team know this info as well, thank you!

btw: if we have specific hidden participant for different message types handling like gpt participant, i’m wondering any bottle neck here by dispatching all specific type message to this participant and then broadcast to origin participants, like single point of failure, no easy to scale when massive viewers, etc. do you think my concern reasonable and livekit may have some ways/plans to overcome it already, may i know your insights, thank you!

SPOF remains a concern. Though I don't think scalability is an issue. While there's a limit at some point, it's going to be orders of magnitude higher than the number of messages human users will be able to consume :stuck_out_tongue:  (i.e. others have pushed multiple MB/s through the data channel)

sure, thanks david for late message, though im still have other concerns, it’s worth to try and see how things going on there, thank you!

nope, psrpc is not necessary. (it's also not necessary when deployed distributed today)

psrpc has a local mode where it simply passes commands through a Go channel.

in a future version of livekit, psrpc will be the default option for internal communications

I’ve investigated this and it turns out ios devices have a limit on hardware encoders - for most devices it’s 3. So 3 h264 encoders for simulcast video track + 1 encoder for screencast track exceeds the limit. Twilio had a similar issue <https://github.com/twilio/twilio-video-ios/issues/17> and they workaround it by using software fallback. We don’t have such option so we want to deactivate one layer of simulcast when screencasting. In `PeerConnection` we set `encoding.isActive` to `false` and it should work. It works when we initially have 2 layers.

However it doesn’t work when starting with 3 layers and then deactivating one. After disabling one layer, webrtc still probably uses 3 encoders. My guess is it’s because it doesn’t reset the encoder. There is a check <https://github.com/webrtc-sdk/webrtc/blob/a828846eb98e404bf3b00c1abf3787f93171a39c/video/video_stream_encoder.cc#L131|here> that if a layer is inactive the encoder doesn’t require the reset. This is just what I deduced from the code and the logs, I haven’t tested it.

Do you have any similar issues? Is my deduction correct and would you be interested in fixing this?

Thank you for the report Angelika, do you mind opening an issue for this in the repo? We would definitely like to fix this. One workaround for now is, you can publish camera video with H.264, and use vp8 for screenshare?

Hi, thanks! Unfortunately currently our backend doesn’t support vp8 (I don’t know why I’m not working on the backend side) so for now as a workaround we have 2 simulcast layers initially.
I opened the issue <https://github.com/webrtc-sdk/webrtc/issues/68>

Hey Gordie, mind providing a bit more details? Which SDK are you using? and which platform is this running on?

publish bitrate is a client-side setting. so it's not possible to set limits during room creation

Awesome work team! :partying_face::partying_face:

nice, when i saw twitch virtual streamer several months before, i’m considering any open source will join openAI services for such things, not only voice but also chat assistant, etc.

This is really cool! good job, will this work on an onprem deployment? or only on livekit cloud as of right now?

<@U04NZ1GD38B> yup it will! You just need to setup the webhook config when you run livekit-server to point at your lkgpt-service

Congratulations :tada: 
Really nice, awesome work guys:+1: 

I can already imagine what great things will come out of it in the next few months. 

Hi Zohaib, a good resource to check out would be the <https://github.com/livekit/client-sdk-flutter/tree/main/example|example app>

a6962b6e-0035-45ec-9515-df2070024824.jpg

thanks david i created the example app can you plz expalin how to gt server url and tokens

hi, you can refer to this steps to deploy the test server locally
<https://github.com/livekit/livekit#getting-started>

or in case you’re using cloud, the docs are <https://docs.livekit.io/cloud/project-management/creating-projects/|here>

Hey Quedo, it's not possible to use other audio codec than Opus.

In a WebRTC session, everyone would need to be able to decode the same codec and for the most part WebRTC has standardized on Opus

can i share the project ID, is that public?

<wss://livepeer-staging-tymhhfyx.livekit.cloud>

<@U0527GV4D6Y> can you DM me your hostname TLD that you’re trying to add for your webhook?

<@U02MC5V3C2H> Did you see my DM, could you help me out with this? thanks

<https://livekit-users.slack.com/archives/C01KZ61HZRR/p1681482706938239>

^^ Could someone help me out with this I still don't have a solution

Sorry for the delay here, pushing out a fix in ~10mins

fix deployed; let me know if you run into any more issues. Again, apologies for the delay.

Did you already read Kubernetes installation docs? <https://docs.livekit.io/oss/deployment/kubernetes/|https://docs.livekit.io/oss/deployment/kubernetes/>

We have not made the helm chart compatible with Azure yet. Unfortunately each vendor does things a bit differently so they have to be dealt with individually.

If you want to take a look, we'll gladly answer questions and review any PRs to this effect.

hi <@U053YJJNFK2>

thanks for the feedback!
the loop components are convenience components that serve the purpose of injecting a track (or participant) context into the jsx tree.
We’re aware that it’s not a very common react pattern, but it turned out to produce the most concise jsx tree for common usage patterns.
That said, the most common pattern these days for looping over stuff is probably the map approach, which is easily usable with how things work today and without relying on loop components
```const tracks = useTracks(...);

return &lt;&gt;{tracks.map(trackRef =&gt; &lt;RenderSomethingWithTrack {...trackRef}/&gt;) &lt;/&gt;```

Hello, I wrote the simple asr python server script using fasterwhisper after getting inspiration from KITT
<https://github.com/atyenoria/livekit-whisper-transcribe>

Hey <@U0527GV4D6Y>, it may need a note in the README itself (cc <@U03RM7G8Q6A>), but yeah, it’s Apache 2:
<https://github.com/livekit/meet/blob/58ce425c4eb68a234fd32a092d6f6d32aa952c45/LICENSE|https://github.com/livekit/meet/blob/58ce425c4eb68a234fd32a092d6f6d32aa952c45/LICENSE>

<@U0527GV4D6Y>. Thanks for pointing this out :+1: We added a license file with Apache 2 to the repo.

Hi there,
Is there some info how to implement ringing functionality, for sending/receiving calls?

either way works! it's pretty inexpensive to create each time

Hey Biswajit, we do not have a flutter token generator at the moment. we would discourage including keys on the client side because anyone with the keys could have full access to your project

Just my 2 cents, but you can revoke permissions through the token `canPublishData: false`

You are welcome <@U043351G48Y>. Just a FYI for future questions, there's a helpdesk channel for these kind of questions for quicker response.

This release version is not in our swift/flutter(iOS/macOS) sdk release, this is a frame encryption test version created for flutter sdk, sorry about this, I updated xframework 4 hours ago

6fcc775591aac5e806fda641d51d11f01fd124a573f3687fbb7b183ffc653670 WebRTC.xcframework.zip

I’ll try to post a new tag that covers this broken version,

new release
<https://github.com/webrtc-sdk/Specs/releases/tag/104.5112.15>

I’m assuming you are asking about LiveKit Cloud. LiveKit is _most likely_ cheaper, but it depends on what your application is doing. It’s certainly possible to construct a use case/usage pattern where LK costs more than an alternative using the participant-minute model. Dolby, in particular, has two pricing models — one for calls and one for streaming, while LK uses the same bandwidth/compute model across the board. Accurately estimating your costs (i.e. “doing the math”) is application-specific, so you’ll have to plug in some numbers to find out the magnitude difference in price.

We have swift sdk available on iOS native platform, currently there are flutter sdk that can support Windows and linux, but if you want to use native C++ API,  I think we need to export or generate a  C++ files from our `client-sdk-rust` cc <@U022KB72NFJ>

Just a suggestion, electron could maybe be an alternative way to tackle this. Just a different point of view.

Try bodypix  (<https://blog.tensorflow.org/2019/11/updated-bodypix-2.html?m=1|https://blog.tensorflow.org/2019/11/updated-bodypix-2.html?m=1>) to filterout required body part ( heads feed) from camera feed and fiterout

<@U04DUL22NF5> sounds pretty cool! I’ll let <@U022KB72NFJ> or <@U01UEMQ8C3F> or someone else who has done pre- or post- processing on audio/video chime in. But, I would do it either on the sender or receiver side, and not on the server. Probably on the sender (i.e. publisher) side, so you aren’t having to process potentially multiple video streams for each receiver (and also duplicating the work for 3+ user conversations).

The reason you don’t want to do it server side is it requires having to decode, process, and re-encode the stream(s) — not impossible to do, but not cheap either.

<@U04DUL22NF5> you can absolutely do this! Unity let’s you share textures between your website and game without copying the texture (via a native texture ptr).

I don’t know if our livekit webgl lib is flexible enough to let you do this BUT the code in there is super useful and will get you on the right track. Check out the texture related code here: <https://github.com/livekit/client-sdk-unity-web/blob/main/Runtime/Plugins/livekit-bridge.jslib#L290>

If you can get your “floating head” on a HTML5 canvas, you can publish/get it on Unity it with LiveKit

Thanks for the input everyone! Sounds like it should be possible, but the tricky part seems to be the actual face tracking, crop-in, and background removal to get a "square cutout" video feed of the face.

The BodyPix/TensorFlow thing <@U045CJPACG1> posted looks interesting! Is anyone aware of other alternatives or technologies that could achieve this? Since my client platform is Unity I wonder if it's even possible to do this local processing _inside_ Unity so that it's cross-platform for all the platforms Unity supports... Although I'd imagine there might be some sort of a performance cost doing it in the Unity layer vs in the browser/JS layer.

Anyways, just thinking out loud! I think it's a cool idea &amp; I'm just trying to figure out what various technologies could be combined in order to actually accomplish this. So, if any other ideas, would love to hear em' :slightly_smiling_face:

I’m not sure about doing it Unity _multiplatform_, but regarding Unity WebGL I have played with the image/video segmentation stuff in tensorflow.js a couple of years ago. I attempted to dig up the code for you but think it’s lost to the sands of time. :disappointed:

 You can probably do the segmentation, render onto a hidden canvas, and extract the image/frame all in a browser.

Ok, glad to know there's at least a path here. Thanks! :slightly_smiling_face:

Yes! if you get the segmentation done, you can give unity that texture (which can just be a texture full of 0s and 1s). Then combine that with the webcam texture in a shader to remove the background.

Sounds like, according to theo, if you do this part outside of unity then you can just pass in 1 texture (the floating head) to unity.

So you’ve got options! Excited to see this in action, would love if you posted back here if you get something working

Will do! I think for now we are waiting for the LiveKit native Unity SDK &amp; centralized API to be released before we take things further than screen share with LiveKit. But I am *very* excited for the possibilities once the native Unity SDK is released! :slightly_smiling_face:

<@U01KZ61GV5Z> <@U01KZ0SCDKL> have your performance team noticed something similar?

where is the source of the out of order packets? on the publisher side or subscriber?

Hi David, sorry for the late reply. The analysis was based on v1.3.5. I can see some patches are applied afterwards which could enhance the situation. Let me find out if the  benchmarks on v1.4.1 show similar behavior.

Hi David, I can confirm that in v1.4.1, the patches have resolved the issue (as far as I tested). I run the same benchmark with both versions, and 1.3.5 had the issue, while 1.4.1 didn't seem to have drop performance with high CPU usage. Thanks!

Sure, I am working on the performance of LiveKit on AArch64, which you can get a reasonable price-performance running on AWS

feel free to open a discussion in <#C025KM0S1CK|> or open PRs if you have performance-related suggestions :slightly_smiling_face:

logging is controlled by `LiveKit.logLevel` . by default this is set to `OFF`

nope. the client only communicates with the server it connects to

<@U04QZP14ZPB> `LocalTrackPublished` is only fired when the current participant publishes a track (i.e. the local video)

You can loop through all the participants in the room (`Room.Participants`) and then check for their tracks upon joining.. also listen to `TrackPublished` for new tracks being published to the room

Hey <@U055GUEG0HX> — maybe you can try giving this a shot:

<https://docs.livekit.io/oss/deployment/vm/#deploy-to-a-vm|https://docs.livekit.io/oss/deployment/vm/#deploy-to-a-vm>

Note the part on using a cloud-init script 

Hello, thanks. I will read it :slightly_smiling_face:

let me know if anything is unclear — we want to make this easier/better

I have another question. I will write here.

While I was inquiring, I had been wondering whether such technology might be suitable for a project I plan to develop. This project involves videocalls that are made by “only” two people, for a total of 500 people.

Then each of these 500 people will have to make the videocall with certain people in that project. Would such technology turn out to be suitable for such a project?