https://livekit.io logo
Join Slack
Powered by
This message was deleted.
# helpdesk
s
This message was deleted.
b
cc @jolly-daybreak-70827
as a background our experience is that the bandwidth / congestion detection isn’t working well in our customers environment and sometimes falls back to a low resolution when we actually should have enough bandwidth available
f
You can disable congestion control using this. Sized based subscription changes (adaptive streaming) will still work and will be the only deciding factor which layer is chosen for forwarding. There is 
TrackStreamStateChanged
here, but that is a binary thing and does not indicate if the layer has been changed due to bandwidth limitation.
👍 1
j
Related to the questions in this thread: what conditions would cause only HD and QHD streams (or layers) to be seen in WebRTC internals and no FHD stream? I used Network Link Conditioner on my mac to limit the bandwidth and I still see the FHD stream in WebRTC internals (with the message that the 
qualtityLimitationReason
is bandwidth). However, in the production environment within Citrix, I never see an entry for the FHD stream.
Also, here is the log from when the layer changed:
Copy code
Aug 28 14:19:11 a303-9011-0707 dwilivekit: 2023-08-28T14:19:11.939+0200#011DEBUG#011livekit#011sfu/streamtrackermanager.go:133#011StreamTrackerManager OnStatusChanged#011{"room": "ad97dc4d-f897-4cf5-b414-78d19f5010cf", "roomID": "RM_dyoqiEFb39Lh", "participant": "CRUIBmJqQFGGuV_-2C0QPg", "pID": "PA_jaRuGeYEeVKS", "remote": false, "trackID": "TR_VCQvrB6ikXDwuR", "relayed": false, "mime": "video/vp8", "layer": 1, "status": "stopped"}
Aug 28 14:19:11 a303-9011-0707 dwilivekit: 2023-08-28T14:19:11.941+0200#011INFO#011livekit#011sfu/streamtrackermanager.go:426#011available layers changed - layer gone#011{"room": "ad97dc4d-f897-4cf5-b414-78d19f5010cf", "roomID": "RM_dyoqiEFb39Lh", "participant": "CRUIBmJqQFGGuV_-2C0QPg", "pID": "PA_jaRuGeYEeVKS", "remote": false, "trackID": "TR_VCQvrB6ikXDwuR", "relayed": false, "mime": "video/vp8", "removed": 1, "availableLayers": [0]}
Aug 28 14:19:21 a303-9011-0707 dwilivekit: 2023-08-28T14:19:21.939+0200#011DEBUG#011livekit#011sfu/streamtrackermanager.go:133#011StreamTrackerManager OnStatusChanged#011{"room": "ad97dc4d-f897-4cf5-b414-78d19f5010cf", "roomID": "RM_dyoqiEFb39Lh", "participant": "CRUIBmJqQFGGuV_-2C0QPg", "pID": "PA_jaRuGeYEeVKS", "remote": false, "trackID": "TR_VCQvrB6ikXDwuR", "relayed": false, "mime": "video/vp8", "layer": 1, "status": "active"}
Aug 28 14:19:21 a303-9011-0707 dwilivekit: 2023-08-28T14:19:21.940+0200#011INFO#011livekit#011sfu/streamtrackermanager.go:393#011available layers changed - layer seen#011{"room": "ad97dc4d-f897-4cf5-b414-78d19f5010cf", "roomID": "RM_dyoqiEFb39Lh", "participant": "CRUIBmJqQFGGuV_-2C0QPg", "pID": "PA_jaRuGeYEeVKS", "remote": false, "trackID": "TR_VCQvrB6ikXDwuR", "relayed": false, "mime": "video/vp8", "added": 1, "availableLayers": [0, 1]}
Aug 28 14:19:36 a303-9011-0707 dwilivekit: 2023-08-28T14:19:36.533+0200#011DEBUG#011livekit#011sfu/streamtrackermanager.go:133#011StreamTrackerManager OnStatusChanged#011{"room": "ad97dc4d-f897-4cf5-b414-78d19f5010cf", "roomID": "RM_dyoqiEFb39Lh", "participant": "NR33yLhLQ-i6_gn-okXgMA", "pID": "PA_fKAfdUyz3iVu", "remote": false, "trackID": "TR_VCRLR5EpcZ5UiK", "relayed": false, "mime": "video/vp8", "layer": 1, "status": "stopped"}
Aug 28 14:19:36 a303-9011-0707 dwilivekit: 2023-08-28T14:19:36.533+0200#011INFO#011livekit#011sfu/streamtrackermanager.go:426#011available layers changed - layer gone#011{"room": "ad97dc4d-f897-4cf5-b414-78d19f5010cf", "roomID": "RM_dyoqiEFb39Lh", "participant": "NR33yLhLQ-i6_gn-okXgMA", "pID": "PA_fKAfdUyz3iVu", "remote": false, "trackID": "TR_VCRLR5EpcZ5UiK", "relayed": false, "mime": "video/vp8", "removed": 1, "availableLayers": [0]}
Aug 28 14:19:36 a303-9011-0707 dwilivekit: 2023-08-28T14:19:36.533+0200#011INFO#011livekit#011sfu/forwarder.go:1209#011stream allocation: optimal#011{"room": "ad97dc4d-f897-4cf5-b414-78d19f5010cf", "roomID": "RM_dyoqiEFb39Lh", "participant": "CRUIBmJqQFGGuV_-2C0QPg", "pID": "PA_jaRuGeYEeVKS", "remote": false, "trackID": "TR_VCRLR5EpcZ5UiK", "relayed": false, "allocation": "VideoAllocation{pause: NONE, def: false, bw: 142381, del: -17086, rates: [[59657 87885 142381 0] [0 0 0 0] [0 0 0 0]], target: VideoLayer{s: 0, t: 3}, req: 0, max: VideoLayer{s: 1, t: 3}, dist: 0}"}
Aug 28 14:19:36 a303-9011-0707 dwilivekit: 2023-08-28T14:19:36.533+0200#011DEBUG#011livekit#011sfu/downtrack.go:459#011sending PLI for layer lock#011{"room": "ad97dc4d-f897-4cf5-b414-78d19f5010cf", "roomID": "RM_dyoqiEFb39Lh", "participant": "CRUIBmJqQFGGuV_-2C0QPg", "pID": "PA_jaRuGeYEeVKS", "remote": false, "trackID": "TR_VCRLR5EpcZ5UiK", "relayed": false, "generation": 11, "layer": 0}
Aug 28 14:19:36 a303-9011-0707 dwilivekit: 2023-08-28T14:19:36.533+0200#011DEBUG#011livekit#011buffer/buffer.go:354#011send pli#011{"room": "ad97dc4d-f897-4cf5-b414-78d19f5010cf", "roomID": "RM_dyoqiEFb39Lh", "participant": "NR33yLhLQ-i6_gn-okXgMA", "pID": "PA_fKAfdUyz3iVu", "remote": false, "trackID": "TR_VCRLR5EpcZ5UiK", "relayed": false, "mime": "video/vp8", "layer": 0, "ssrc": 3753353732, "force": false}
Aug 28 14:19:36 a303-9011-0707 dwilivekit: 2023-08-28T14:19:36.734+0200#011DEBUG#011livekit#011sfu/downtrack.go:459#011sending PLI for layer lock#011{"room": "ad97dc4d-f897-4cf5-b414-78d19f5010cf", "roomID": "RM_dyoqiEFb39Lh", "participant": "CRUIBmJqQFGGuV_-2C0QPg", "pID": "PA_jaRuGeYEeVKS", "remote": false, "trackID": "TR_VCRLR5EpcZ5UiK", "relayed": false, "generation": 11, "layer": 0}
Aug 28 14:19:36 a303-9011-0707 dwilivekit: 2023-08-28T14:19:36.775+0200#011INFO#011livekit#011sfu/forwarder.go:1461#011downgrading layer#011{"room": "ad97dc4d-f897-4cf5-b414-78d19f5010cf", "roomID": "RM_dyoqiEFb39Lh", "participant": "CRUIBmJqQFGGuV_-2C0QPg", "pID": "PA_jaRuGeYEeVKS", "remote": false, "trackID": "TR_VCRLR5EpcZ5UiK", "relayed": false, "current": "VideoLayer{s: 1, t: 3}", "target": "VideoLayer{s: 0, t: 3}", "max": "VideoLayer{s: 1, t: 3}", "layer": 0, "req": 0, "maxPublished": 1, "feed": 3753353732}
Aug 28 14:19:36 a303-9011-0707 dwilivekit: 2023-08-28T14:19:36.775+0200#011DEBUG#011livekit#011sfu/forwarder.go:1370#011switching feed#011{"room": "ad97dc4d-f897-4cf5-b414-78d19f5010cf", "roomID": "RM_dyoqiEFb39Lh", "participant": "CRUIBmJqQFGGuV_-2C0QPg", "pID": "PA_jaRuGeYEeVKS", "remote": false, "trackID": "TR_VCRLR5EpcZ5UiK", "relayed": false, "from": 1219075716, "to": 3753353732}
Aug 28 14:19:36 a303-9011-0707 dwilivekit: 2023-08-28T14:19:36.775+0200#011DEBUG#011livekit#011sfu/downtrack.go:555#011forwarding key frame#011{"room": "ad97dc4d-f897-4cf5-b414-78d19f5010cf", "roomID": "RM_dyoqiEFb39Lh", "participant": "CRUIBmJqQFGGuV_-2C0QPg", "pID": "PA_jaRuGeYEeVKS", "remote": false, "trackID": "TR_VCRLR5EpcZ5UiK", "relayed": false, "layer": 0}
Aug 28 14:19:37 a303-9011-0707 dwilivekit: 2023-08-28T14:19:37.439+0200#011DEBUG#011livekit#011sfu/streamtrackermanager.go:133#011StreamTrackerManager OnStatusChanged#011{"room": "ad97dc4d-f897-4cf5-b414-78d19f5010cf", "roomID": "RM_dyoqiEFb39Lh", "participant": "CRUIBmJqQFGGuV_-2C0QPg", "pID": "PA_jaRuGeYEeVKS", "remote": false, "trackID": "TR_VCQvrB6ikXDwuR", "relayed": false, "mime": "video/vp8", "layer": 1, "status": "stopped"}
Aug 28 14:19:37 a303-9011-0707 dwilivekit: 2023-08-28T14:19:37.439+0200#011INFO#011livekit#011sfu/streamtrackermanager.go:426#011available layers changed - layer gone#011{"room": "ad97dc4d-f897-4cf5-b414-78d19f5010cf", "roomID": "RM_dyoqiEFb39Lh", "participant": "CRUIBmJqQFGGuV_-2C0QPg", "pID": "PA_jaRuGeYEeVKS", "remote": false, "trackID": "TR_VCQvrB6ikXDwuR", "relayed": false, "mime": "video/vp8", "removed": 1, "availableLayers": [0]}
f
As long as the stream is started, I believe it is shown in 
<chrome://webrtc-internas>
. Like you mentioned, whether it is sending packets or not is determined by browser client doing bandwidth estimation and congestion control. Maybe, in your production, the stream is not even started. I do not know exactly when the browser client decides to start a stream. It usually starts up all streams almost immediately at start up as far as I have seen. The logs above is server doing tracking. As browser clients do not have any indication when it will stop/resume a simulcast layer (there are some proposals to indicate that, but not all clients are going to implement it uniformly at least for a while), server determines that looking for stoppages in packet flow.
j
I’m not convinced yet that the reason we’re dropping down a layer is because of congestion, though, which is why @best-parrot-43500 asked the original questions about finding the reason. When I simulate a congested system I can clearly see an indication of the quality drop in the 
qualityLimitationReason
field. However, in our prod system when it drops down a layer, the reason shows 
None
(vs. the word Bandwidth in my simulated case)
f
There might be some confusion here between up stream and down stream. Upstream congestion control is done by browser/client. You can see that in webrtc-internals where there is a quality limitation reason. That also corresponds to the logs you posted, i. e. server is receiving layers and trying to figure out if browser/client has stopped/resumed some layer because of congestion happening/congestion relieving. I thought David asked about down stream congestion. That is the NONE you are referring to I am guessing. SFU uses one peer connection for all downstream tracks. It has to try and fit as many tracks as it can. While doing that, it will switch layers. That is the setting I pointed out to David to disable. Upstream, there is no control for that. Clients/Browsers automatically do it.
👍 1
b
thanks for the input @fancy-wire-61616 yes I was referring to downstream congestion. We have to do some more tests to figure out what exactly is happening
👍🏽 1
Upstream congestion control is done by browser/client.
@fancy-wire-61616 could you help me clarify this. Is the congestion control done by the livekit client js or is it done into the baked in WebRTC stack of the browser?
f
It is baked into browser.
b
okay thanks for the fast answer 👍
6 Views
Open in Slack
PreviousNext
Powered by