This message was deleted.

cc @chilly-animal-65189

@fancy-author-2900 what’s the source of your input video?

@acoustic-engineer-41666 https://github.com/savonet/liquidsoap .. outputs reliably as RTMP or SRT

WHIP is only likely to give you meaningfully better latency than RTMP if you skip transcoding, which may or may not be possible with your application. I can't really comment on A/V sync issues without more details. ingress logs may have something relevant in debug mode. Do you know if livekit behaves worse than other services with your setup?

Thanks, I really think this is a problem with the

mpegts-whip

library so I don't want to take up your time time with it.

Nothing strikes as unusual in the logs. We use the RTP/RTCP timestamp as the source of truth for A/V sync. Also, if there is a large "interleave" delay between the audio and video tracks, sync may be off at the start but should converge over time.

thanks, will keep digging on thi send

Hi here is a summary of findings so far: Hi LiveKit people, here is a quick summary of our findings so far. Our project contains a fixed list of video-audio/audio-only streams and we mux the original audio track with some background audio tracks before pushing it to LiveKit’s RTMP server/WHIP endpoint. While testing, it is noticed that the original audio track may not be in sync with the video track at the beginning if pushed to pass-through WHIP endpoint. The delay for audio is around 1 second. However, this delay becomes negligible in 10 seconds. To reproduce the problem and find out possible causes: 1. Eyevinn’s whip endpoint was first used as a comparison and such problems do not occur. 2. LiveKit’s ingress/egress pipeline was built and different streaming protocols/stream contents were tested. 3. Our playlist contains a list of MKV files (codec: AVC/PCM)) 4. It is found that: a. This problem occurs most often when the pass-through feature in whip-ingress is enabled (1~2 times out of 10). It works well for rtmp-ingress or non-pass-through whip-ingress. b. The problem can be observed from participants (browser) once the content is pushed to the whip endpoint. Refreshing the tab (reconnecting to the room) does not solve the problem. c. The problem can be repeatedly reproduced whenever a stream is restarted (instead of happening just once). 5. A short video clip is included as a record. Here we replaced the original contents with two testing contents (Me counting from 1 to 5). It can be seen from the clip that when first connected, there was a delay in audio (around 1s) and this delay disappeared in a few seconds. It’s worth noting that the first 1~2 second of the video did not show up but the audio “this is a test file, 1, 2 …” was still complete. 6. It is guessed that when pass-through feature is enabled for whip endpoint, some video frames in the beginning are somehow dropped while the audio is preserved. This causes the out-of-sync problem at the start of streaming but becomes less of a problem over time.

Curious, what is the key frame interval in the video above, in the original file?

@quiet-address-2256 you should probably answer the question from @chilly-animal-65189 as I'm not 100% sure

Also, in the way your pipeline is setup, will mpegts-whip always start the whip stream with an IDR frame?

The keyframe interval is 10 seconds and the delay seems to be constant (1 second). I’m not sure if mpegts-whip always starts with an IDR frame. But it is pushing VP8/OPUS streams to the whip endpoint.

@quiet-address-2256 is there anyone you can ask, the other developers on the mpegts-whip project, regarding the IDR frame? I know that liquidsoap (what we are using to generate the stream that then goes to mpegts-whip) allows for the specification of a

g

parameter ( https://stackoverflow.com/questions/66218565/what-does-g-flag-mean-in-ffmpeg ) .. which I think is related, but I don't know if

mpegts-whip

is somehow over-riding this value.

Let me check.

@chilly-animal-65189 Is there by any chance a written specification or standard or example you could point to, that @quiet-address-2256 could compare

mpegts-whip

's output to, which, if he were to try to amend `mpegts-whip`'s output, he might might be most likely to provide the needed stream data such that the video and audio of

mpegts-whip

output stream does not lose synchronization when played through LiveKit with

bypass_encoding: true

?

Same situation happens to me too. It has a constant delay around 1 second.

If good synchronization is never achieved, checking the TS validity may be worth it. In the case of VP8, key frames would not be called "IDR", but the same question holds. Stepping back, a 10s key frame interval will not work well with WHIP without transcoding because the viewer will need for a key frame to arrive to start showing video (no PLI without transcoding). It is possible there is an issue with timestamp handling, particularly when there us a long gap between key frames in livekit ingress. I can't tell for sure until I get the time to try your use case with extra instrumentation. It may be a few days before I can do that

Thanks for that insight! We will keep digging into that.

@chilly-animal-65189 ... This is where we are with this: Achieving a stable ingress into LiveKit is our final barrier to using LiveKit as our streaming service -- the API, rooms, etc., all of that is working and this, the problem of ingress, is all that is left as a barrier. You can see our experiments by looking up ingress id:

IN_2C6Rhf7FfUYv

.. and also we tried a WHIP ingress with

bypass_whip_transcoding: false

, with whip ingress id

IN_KN2rwqSjGFdd

So far we are unable to achieve a stable ingress into LiveKit. RTMP has great audio/video synchronization. Maybe only once or twice have we seen problems with audio/video synchronization with RTMP, out of 50+ attempted streams. The problem with RTMP is random dropouts and with time/pitch shifting. Our service uses background music, so random dropouts or pitch shifting, which might not be a big deal for videoconference, are a real problem. WHIP ingress is much more stable with regard to random dropouts or pitch shifting -- these almost never happen. WHIP also has lower overall latency when compared with RTMP. The problem with WHIP ingress is the audio/video synchronization. We're using

whip-mpegts

to convert a local SRT stream to a WHIP stream. We've tried both setting

bypass_whip_transcoding: true

and

bypass_whip_transcoding: false

, and in both cases there can be audio/video synchronization problems, typically the audio falling behind the video. The issue is very challenging to debug, because sometimes the synchronization is perfect -- I can stop and start my stream ten times, and each time it is perfect. But then I come back an hour later, and do the same thing, and each time I start and stop the stream, the synchronization is way off, and it looks like the person speaking in our video's lips do not match the audio. I'm beginning to think the issue might be on LiveKit's end, or in any case, I'm not sure how to troubleshoot it better on my end. If anyone is interested I'd be happy to share a docker compose file which you could quickly start, put in your own WHIP endpoint url, and you should be able to reproduce the problem (again, it might take a few tries for the problem to show.)