LiveKit Community

IMG_7362.png

ah ok, this is to really provide sane defaults so that users don't trip themselves up.. it's definitely possible to make it a user-configurable option. can you open an issue for it?

(and if you are up for making a PR for it too, that'd be even better :slightly_smiling_face: )

Perfect I will check it on Monday with my team :grinning: 

hello <@U0402GZ32CC> — there shouldn’t be that long of a delay. Copying <@U01UB0LHTTQ> who may be able to advise.

Hi,
I read <https://github.com/livekit/egress> and got following lines in the configuration:
```local_directory: base path where to store media files before they get uploaded to blob storage. This does not affect the storage path if no upload location is given.```
Does it add delay to the flow from Egress to RTMP stream or it just effects to S3? Can we disable this?

how are you starting the track composite?

Hi, When my backend get the notification from webhook that video is published (event = track_published), I send a track compose egress request to livekit server, then when I get the webhook with event = egress_started, I start to capture video from RTMP client

I’ll take a look to see if the delay is from gstreamer or the rtmp server, but I don’t think it’s on the egress side since it starts to output data pretty quickly

it looks like you are measuring the total latency.. and these webhooks back &amp; forth could definitely add a bit to it. what is your RTMP client? how much latency would that add?

I'm using OpenCV to open rtmp streaming. Because it just reads stream &amp; save to files, I dont think it take much time.
Acctually, i dont need to start capture/get video from the begining of client. Just need to get capture/video at the time that Egress started (when i see the event: egress_stared).

I see, I'm not sure how your server works (is it buffering? etc), or when the OpenCV client starts streaming. I cannot see a scenario where Egress would add more than 500ms of latency to the stream.

Hi,
Is there solution that can capture frame directly from egress streaming video ? Or another solution for my scenario above? Please advise this.

Hi,
One more thing, How can I setting the frame_hiegh, frame_width &amp; FPS of video streaming when I do egress? At the client site, I set it is h90 (160, 90, 15) but it seems that the egress still send with others.

if you are looking for a low-level solution to grab the frames, Go SDK is what you'd use. connect to the room and read the packets out. Egress uses Go SDK internally.

I'm not sure what you mean with the 2nd question.

About 2nd question, as I read <https://github.com/livekit/livekit-recorder>, Livekit-recorder has a parameters as PRESET below:
```{
    "template": {
        "layout": "speaker-dark",
        "room_name": "my-room"
    },
    "filepath": "path/filename.mp4",
    "options": {
        "preset": "HD_60"
    }
}```
So as I understand that, we can set the same when we use egress for extract video with hiegh, width, fps as we want. I just want to reduce the quality of video for better latency at output side.

you would set custom encoding options: <https://github.com/livekit/protocol/blob/main/livekit_egress.proto#L61>

```{
...
  advanced: {
    width: xx,
    height: xx,
    framerate: xx,
    video_bitrate: xx,
    key_frame_interval: xx
  }
}```

Screen Recording 2023-02-06 at 2.12.17 PM.mov

when do you think that it will be available?

Hey, I estimate the arrival of the SDK in 1 to 2 weeks :slightly_smiling_face:

Is it possible to share a preview so we can test it?

Hey <@U022KB72NFJ>! Any News on the Unity SDK? I am looking forward to it.

Hm, hard to say without seeing your application code. My guess is that those errors are printed before you interact with the page, and then you interact without realizing it (maybe triggering a React re-render?), which starts the video.

Are you using just Javascript, or our React SDK, or our Components toolkit?

If you mean mute local media stream, this method is correct, if you want to stop receiving RemoteTrack, you can try `RemoteTrackPublication.unsubscribe()`

I can indeed mute with this.

However when I try to unmute I get an error.

image.png

I think it will throw an exception to call setMicrophoneEnabled(false); multiple times in the loop, can you move it outside the loop?

That loop runs only once. It is gaurenteed to run only once.

Even when I do this, I get the same error.

Have you successfully published localtrack before? If yes, track should not be null.
does client-sdk-flutter/example work as expected?

Before I press the mute button, I can properly talk to others.

Can you try adding `await` before these two calls

Nah nah,

adding async didn't work. However I got a fix.

I said that this won't work without actually checking it.

I previously check similar code, by turning it true then to false with a await.

However in the above code, I forgot to write await. Hence, I hadn't actually checked that.

Turns out this works
```liveKitRoom!.localParticipant?.setMicrophoneEnabled(true);

liveKitRoom!.localParticipant?.setMicrophoneEnabled(player.isMuted);```
But this does not
```await liveKitRoom!.localParticipant?.setMicrophoneEnabled(true);

await liveKitRoom!.localParticipant?.setMicrophoneEnabled(player.isMuted);```


Don't actually know why though.

It should be async as written in the docs.

Hi Saurav, please thread your messages when possible. Thanks! :pray::skin-tone-4:  

So just calling that will do?

Or is RemoteTrackPublication a class, whose object I should find before calling this method.

<@U04JZKKA6TV> What platform are you reproducing the error on? I found a similar problem in flutter windows (already found a fix).

Has anyone here experimented with using gstreamer to read from a file and publish to a LK room via webrtc, with simulcast?

streaming only video over rtmp works fine using:

```livekit-cli start-track-composite-egress --request egress-rtmp.json```
```{
  "room_name": "test",
  "video_track_id": "TR_VCQ6XdT7YCorTb",
  "stream": {
    "urls": [
      "<rtmp://10.19.129.59:1935/live/app>"
    ]
  }
}```
does not work if i add a audio stream to the composite

now i’m trying to send audio over websocket and i get
```2023-02-09T10:56:46.841Z	INFO	egress	service/service_deprecated.go:65	request accepted	{"nodeID": "NE_7A3LAxgg3BVB", "egressID": "EG_dS9rZAwogwGv", "requestID": "RPC_nZoRqeNeG9jr", "senderID": "ND_aoDZr002"}
2023-02-09T10:56:46.858Z	INFO	egress	redis/redis.go:82	connecting to redis	{"nodeID": "NE_7A3LAxgg3BVB", "handlerID": "EGH_EmiDLCX4SPev", "egressID": "EG_dS9rZAwogwGv", "simple": true, "addr": "10.238.29.84:6379"}
2023-02-09T10:56:46.888Z	INFO	logger/logger.go:130	received offer for subscriber	{"nodeID": "NE_7A3LAxgg3BVB", "handlerID": "EGH_EmiDLCX4SPev", "egressID": "EG_dS9rZAwogwGv"}
2023-02-09T10:56:47.914Z	INFO	logger/logger.go:130	ICE connected	{"nodeID": "NE_7A3LAxgg3BVB", "handlerID": "EGH_EmiDLCX4SPev", "egressID": "EG_dS9rZAwogwGv"}
2023-02-09T10:56:48.076Z	INFO	logger/logger.go:130	received offer for subscriber	{"nodeID": "NE_7A3LAxgg3BVB", "handlerID": "EGH_EmiDLCX4SPev", "egressID": "EG_dS9rZAwogwGv"}
2023-02-09T10:56:48.087Z	INFO	logger/logger.go:130	track subscribed	{"nodeID": "NE_7A3LAxgg3BVB", "handlerID": "EGH_EmiDLCX4SPev", "egressID": "EG_dS9rZAwogwGv", "participant": "pebfq_pub_0", "track": {}, "kind": {}}
2023-02-09T10:56:48.131Z	INFO	egress	service/handler_deprecated.go:156	egress updated	{"nodeID": "NE_7A3LAxgg3BVB", "handlerID": "EGH_EmiDLCX4SPev", "egressID": "EG_dS9rZAwogwGv", "egressID": "EG_dS9rZAwogwGv", "request_type": "track", "output_type": "websocket", "status": "EGRESS_ACTIVE"}
2023-02-09T10:56:53.365Z	ERROR	egress	pipeline/pipeline.go:811	pipeline error	{"nodeID": "NE_7A3LAxgg3BVB", "handlerID": "EGH_EmiDLCX4SPev", "egressID": "EG_dS9rZAwogwGv", "element": "GstOpusDec", "message": "Decoding error (-4): corrupted stream", "error": "Could not decode stream."}
<http://github.com/livekit/egress/pkg/pipeline.(*Pipeline).handleError|github.com/livekit/egress/pkg/pipeline.(*Pipeline).handleError>
	/workspace/pkg/pipeline/pipeline.go:811
<http://github.com/livekit/egress/pkg/pipeline.(*Pipeline).messageWatch|github.com/livekit/egress/pkg/pipeline.(*Pipeline).messageWatch>
	/workspace/pkg/pipeline/pipeline.go:300
<http://github.com/tinyzimmer/go-gst/gst.goBusFunc|github.com/tinyzimmer/go-gst/gst.goBusFunc>
	/root/go/pkg/mod/github.com/livekit/go-gst@v0.0.0-20220603230042-cef031256427/gst/cgo_exports.go:128
_cgoexp_752bbda710ba_goBusFunc
	_cgo_gotypes.go:17363
runtime.cgocallbackg1
	/usr/lib/go-1.18/src/runtime/cgocall.go:314
runtime.cgocallbackg
	/usr/lib/go-1.18/src/runtime/cgocall.go:233
runtime.cgocallback
	/usr/lib/go-1.18/src/runtime/asm_amd64.s:971
runtime.systemstack_switch
	/usr/lib/go-1.18/src/runtime/asm_amd64.s:436
runtime.cgocall
	/usr/lib/go-1.18/src/runtime/cgocall.go:167
<http://github.com/tinyzimmer/go-glib/glib._Cfunc_g_main_loop_run|github.com/tinyzimmer/go-glib/glib._Cfunc_g_main_loop_run>
	_cgo_gotypes.go:2822
<http://github.com/tinyzimmer/go-glib/glib.(*MainLoop).Run.func1|github.com/tinyzimmer/go-glib/glib.(*MainLoop).Run.func1>
	/root/go/pkg/mod/github.com/tinyzimmer/go-glib@v0.0.25/glib/gmainloop.go:53
<http://github.com/tinyzimmer/go-glib/glib.(*MainLoop).Run|github.com/tinyzimmer/go-glib/glib.(*MainLoop).Run>
	/root/go/pkg/mod/github.com/tinyzimmer/go-glib@v0.0.25/glib/gmainloop.go:53
<http://github.com/livekit/egress/pkg/pipeline.(*Pipeline).Run|github.com/livekit/egress/pkg/pipeline.(*Pipeline).Run>
	/workspace/pkg/pipeline/pipeline.go:214
<http://github.com/livekit/egress/pkg/service.(*HandlerV0).Run.func2|github.com/livekit/egress/pkg/service.(*HandlerV0).Run.func2>
	/workspace/pkg/service/handler_deprecated.go:84
2023-02-09T10:56:53.367Z	WARN	egress	service/handler_deprecated.go:144	egress failed	{"nodeID": "NE_7A3LAxgg3BVB", "handlerID": "EGH_EmiDLCX4SPev", "egressID": "EG_dS9rZAwogwGv", "egressID": "EG_dS9rZAwogwGv", "request_type": "track", "output_type": "websocket", "error": "Could not decode stream."}```

```livekit-cli start-track-egress --request egress-ws.json```
```{
  "room_name": "test",
  "track_id": "TR_AMJQMwzJtbzdj9",
  "websocket_url": "<ws://10.19.129.59:8766>"
}```

ok the problem is the audio track created by the livekit-cli, it works with an audio track from a “normal user” but not for the ones created by the cli

<@U04NASSD06T> there are over 1700 people in this channel. Please be considerate and thread your messages whenever possible. Posting messages in threads to the top-level channel is unnecessary. :pray::skin-tone-4:  

You’re using the same credentials across the different machines/environments? If not, may I suggest making a test key to try using on the different environments?

Yes, same credentials for this test environment across team members, works for them not for me :S

And are you using Cloud or a self-hosted LiveKit instance?

I just noticed a slightly different behavior, when I run livekit-cli list-rooms (using therefore the default project, I get:

twirp error internal: unexpected HTTP status code 301 "Moved Permanently" received, Location="/twirp/livekit.RoomService/ListRooms"

(for your reference I am on a Macbookpro M2 based, using livekit-cli 1.2.3 installed via brew)

This is super weird and my hunch is that you have a surprising proxy somewhere

Have you tried just curling the server IP?

I have pinged it, traceroute it, I can def. see it

If you curl -i to see the headers, is there anything useful?

even tried two wifi connections and a vpn, all with same results

Screenshot 2023-02-10 at 1.43.51 AM.png

(thx for spending some time on this btw, greatly appreciate it)

those don't look like livekit headers.. `via 1.1 google` seems like a clue

these are livekit headers:
```% curl -i <http://localhost:7880>
HTTP/1.1 200 OK
Vary: Origin
Date: Fri, 10 Feb 2023 01:51:43 GMT
Content-Length: 2
Content-Type: text/plain; charset=utf-8```

I was assuming 1.1 google is because this is on a GCP (google cloud platform) cluster

but I will compare it with my peers as soon as they wake up tomorrow

maybe GCP has a frontend proxy? but let us know!

we do have a proxy indeed, but the proxy is there for me, as it is for other developers on my team

but yes, this is a good lead and I will compare with them asap and get back to this channel, thank you both for the time

Hi, could you help me to find an example how to send samples (from my own producer) to livekit using go-sdk?

To accomplish what you described - display HD locally, transmit lower res. The simplest path would be to use dynacast on the publisher side.

On the publisher side, acquire HD res locally, and publish to the server with both dynacast and simulcast enabled.

On the receiver side, use setVideoDimensions to request the layer that you need.

Dynacast will ensure that the highest resolution layer will be disabled during publishing, since there are no consumers of it.

the other way to accomplish this is to call getUserMedia separately for the local display.

Or with one of the „create local track“ methods (see question 1) right? It’s at the end the same 

How many simulcast layers can I create/customize?

I found the following in the documentation, what is a bit unclear to me:
&gt; custom video simulcast layers for camera tracks, defaults to h180, h360, h540 You can specify up to two custom layers that will be used instead of the LiveKit default layers. Note: the layers need to be ordered from lowest to highest quality
So do the 2 custom layers replace 2 of the default ones, but I have still 3 at the end or do I have at the end only the 2 custom ones instead of having the 3 defaults?

Regarding  the other way, there seems to be maybe an issue:

<https://livekit-users.slack.com/archives/C01KVTJH6BX/p1676232384071879|https://livekit-users.slack.com/archives/C01KVTJH6BX/p1676232384071879>

On the simulcast layers, you can publish 3 total, but one of them has to be the original track. So you would specify two additional layers.

For the alternative.. I didn't realize Chrome didn't allow you to acquire them separately. Maybe this is an OS/browser specific restriction. To be honest I have not tried it myself, just that it seems that it should be possible given what the docs state.

Snipaste_2023-02-14_11-18-36.jpg

Perhaps that error message can be improved, but try a subdomain without a `.`

~The quality score isn’t the same as the layer subscriptions — “quality” is a score from 1-5~

~(It approximates Mean Opinion Score, if you’ve seen that in other media applications)~

<@U04NASSD06T> log line you are referring to is server-side support for Dynacast, where unused video tracks/layers are paused on the publisher side. VideoQuality refers to <https://github.com/livekit/protocol/blob/main/livekit_models.proto#L128>, an integer that maps to a layer of video in simulcast

To understand what the clients are consuming, you'll want to look at a different log line instead

that line will say `updating video layer` <https://github.com/livekit/livekit/blob/master/pkg/rtc/subscribedtrack.go#L202>

Egress will automatically latch onto the livekit-server deployment. So your API requests should be directed towards livekit-server

*sudo docker run --rm --network=host -e EGRESS_CONFIG_FILE=/out/config.yaml -v ~/livekit-egress:/out livekit/egress*

i can run this properly, but which port should i use ??

There's nothing within LiveKit that prevents it.. That seems like a OS specific limitation. Are you on Windows?

Is there any way to get the frame images?

is your app built with our android native SDK?

hey <@U04NQNFQE5S> what are you using to take screenshots?

screenshot package
<https://pub.dev/packages/screenshot>

It may be that the video stream is rendered on flutter from gl txeture, you can try <https://github.com/flutter-webrtc/flutter-webrtc/blob/main/lib/src/native/media_stream_track_impl.dart#L74|https://github.com/flutter-webrtc/flutter-webrtc/blob/main/lib/src/native/media_stream_track_impl.dart#L74> of flutter webrtc

Pretty sure the answer is “yes” but since I’m not too familiar with Unity I defer to <@U022KB72NFJ> (when he’s awake)

Hey AdionN, our future Unity SDK will support spatial audio :slightly_smiling_face:
( using the Unity Audio Sinks )

Hi, everyone! My name is Saverio. I am a delivery engineer for the Italian metaverse from TechStar s.r.l.

I'm new to this community.

Hi <@U04QT0QLPH6> Welcome to the community!!!

Looks like a great project.

Can you please thread your messages so that all messages on a topic stay grouped?

Also, a better channel to post this might be <#C01KVTJH6BX|>

For the question you are asking, I do not know the answer. Hopefully, somebody has more experience with vpx bindings.

Track Egress sounds useful for your case <https://docs.livekit.io/server/egress/track/>

if the goal is to intercept the tracks before sending them to the other participants in the room, egress might not be useful. If that’s the case, you might see better performance encoding to h264 after decoding the vp8 tracks

Hello, thank you for your response.
The use case is to send the frames to a TCP server that performs NN inference on pictures.
It seems to me that Egress is suited for saving the tracks on storage.

Is there a way to sample a vp8 track into full pictures (jpg/bmp/png) at a given frequency? (lower than the video is ok)

Right now decoding from vp8 works but only for a single track. Deconding multiple vp8 streams drop performance drastically.

Even a complete work around is welcome. Thanks :slightly_smiling_face:

I see. vp8 decoding takes a lot of cpu, I think the only way around the performance issues is to use more (if the work is indeed serialized) and/or bigger servers

as far as vpx bindings and serialization go, I’m afraid I don’t have much experience there

Asterisk the VoIP thing? There’s no libraries for interacting with Asterisk, and I don’t know what Asterisk’s interfaces are, but it should be possible to build. What are you trying to do?

Am trying to build a telemedicine App using react-native where all the video/voice calls go through Asterisk VOIP server.

And by "monitor" i mean something like a VU-meter.

Hey Michalis, if I understand correctly, the custom `AudioContext` route should work for remote participants, but you are also looking to include the local participant's audio as well?

We have a function <https://github.com/livekit/client-sdk-js/blob/b6cb8140c0912a4739af5e1c319a06285820c959/src/room/utils.ts#L292|createAudioAnalyzer> in the JS SDK gives you this for a single track. Hopefully that could be a template of how one could combine multiple tracks in the analysis.

It should work with the local participant as well.

Thank you <@U01KZ61GV5Z> :pray: Exactly, the provided audio analyzer looks like a good starting point, combining the tracks and getting an output level for all of them  gets a little more difficult. I'll give it a try later today. Concern I have is, to be able to monitor combined output even after adjusting a participant's volume (and this is what I don't know - for example room admin is able to adjust output level for each participant, essentially controlling their input/mic level).

Hey Oleg, you'd have to specify Egress output dimensions. You could use the portrait preset <https://github.com/livekit/protocol/blob/main/livekit_egress.proto#L41|here>

Hello <@U045UBXUYTW>, if I am understanding correctly, Livekit could accomplish this pretty much out of the box. You would have your audio source (broadcaster) being the publisher (another pc, a cellphone) and have your system speaker connecting and subscribing to that room and output the audio.

Thanks Pat, I guess I’m looking for a way to do something like this with the CLI. But I am assuming I’ll have to use one of the clients. Was hoping for a C++ client but it seems like that is still in the works.

There's always a way through the GO sdk through CLI.

As for the C++ client, yup I know the LiveKit team are working on it but unsure whats the status of it. Maybe <@U01KZ61GV5Z> might know .

Maybe I’ll need to do a deeper dive. If you have any docs that might send me down the right path I would greatly receive them!

Here's the docs section, they've done a wonderful job at it. <https://docs.livekit.io/|https://docs.livekit.io/>

As for the GO SDK, feel free to have a glance here: <https://github.com/livekit/server-sdk-go|https://github.com/livekit/server-sdk-go> examples are also provided :)

<@U045UBXUYTW> the CLI should automatically subscribe to tracks in the room. Though it is not able to playback directly (Go SDK doesn't currently decode or playback the tracks).

What you are looking to do is achievable with our Rust SDK as well. Rust SDK would decode tracks and give you the raw buffers for playback

Alright I’ll take a look at that :). Guess I’ll need to learn a little rust. 

After taking a look at the README for the rust SDK it seems like it does not support publishing yet. So I am guessing I would likely need to continue to use two SDK's for the time being to achieve my goal?

Andres, publishing support for Rust is coming pretty soon. we are actively working on it!

Hello <@U04PR7GN092>, currently the LiveKit Dashboard found in their cloud service, isn't open sourced at the moment. I'm not aware of any other dashboard either which would come with LiveKit.

Hi, please don't access the `AVAudioSession` directly since the SDK will frequently override it.

Please see:
<https://github.com/livekit/client-sdk-swift#audiosession-management>

Use `RTCAudioSession` instead and make sure to lock / unlock :
<https://github.com/livekit/client-sdk-swift/blob/1f5959f787805a4b364f228ccfb413c1c4944748/Sources/LiveKit/Track/AudioManager.swift#L199-L203>

Looks like the aar file for this version is missing <https://jitpack.io/com/github/webrtc-sdk/android/104.5112.05/>

CC: <@U01NZKTNE1H>

<@U031T10JC9W> do you wanna try `104.5112.07`?

Yeah, we have already moved to the 07 and it works correctly but it looks like only 06 and 07 versions are available

I found this issue <https://github.com/jitpack/jitpack.io/issues/5289> and it seems to be still accurate

thanks for pointing this to our attention. this is rather unfortunate :disappointed:

maybe jitpack isn't the right provider to be using for package distribution

ick this the second time, basically subsequent releases. yeah definitely need to migrate over to maven central

Guys please  tell us when this lib will be migrated to another provider. Thank you for your effort.

quick update on this: will be moved over to Maven Central soon. pr at <https://github.com/webrtc-sdk/android/pull/9>

104.5112.07 is uploaded (should sync to maven central within an hour or so) at io.github.webrtc-sdk:android:104.5112.07 (note the change in group id)

Awesome. Will this fix issue for 1.1.6? It still seems to be looking for older version. "Could not find com.github.webrtc-sdk:android:104.5112.05.
Required by:
    project :app &gt; io.livekit:livekit-android:1.1.6"

For older versions, you'll need to exclude the com.github.webrtc-sdk:android dependency and manually depend on the io.github.webrtc-sdk:android dependency. The group ids are different, so we can't get a clean replacement here.

<https://docs.gradle.org/current/userguide/dependency_downgrade_and_exclude.html#sec:excluding-transitive-deps|https://docs.gradle.org/current/userguide/dependency_downgrade_and_exclude.html#sec:excluding-transitive-deps>

Hi <@U01NZKTNE1H> just to follow up on this, with the following gradle setup, I experience serious performance issue similar to that described in <https://github.com/livekit/client-sdk-android/issues/189>

```// WebRTC
implementation 'io.github.webrtc-sdk:android:104.5112.07'

// LiveKit
implementation ("io.livekit:livekit-android:1.1.6") {
    exclude group: 'com.github.webrtc-sdk', module: 'android'
}```
Is the webrtc version compatible with LiveKit 1.1.6? Or do you have any suggestions? Thanks for any help.

hmm, the binaries should be compatible, as there weren’t any changes related to video encoding between .05 and .07

in the mean time, I’ll reupload the older versions to maven central later today

104.5112.05 and 105.5112.03 have been released onto maven central

Fantastic. Thanks <@U01NZKTNE1H> - it's working for me now.

Absolutely loving this sample!

Would this work with Unity + WebGL as well?

Working through these samples today to cook up a demo to unlock the company card.

<https://github.com/livekit/client-sdk-unity-web>
<https://github.com/livekit/client-unity-demo>

Also saw that the Unity SDK might be getting reworked.

Hey Neil!  I goofed and hit send prematurely.  Would you be the right person to answer the Unity questions I added above?

Hrmm I’d have to look more into our unity sdk (cc <@U022KB72NFJ>) about using it with WebAudio nodes.

I’ve done something in the past before the unity sdk existed where I kept my webrtc in a react app and had unity communicating to web myself with player positions and things like that.

If that’s the route you take, this file: <https://github.com/livekit-examples/spatial-audio/blob/main/src/controller/SpatialAudioController.tsx> is the one that will get you spatial audio once you do get unity to communicate to the web with player positions

for unity -&gt; browser communication: <https://docs.unity3d.com/Manual/webgl-interactingwithbrowserscripting.html>

We achieved it with ThreeJS + PositionalAudio :slightly_smiling_face: