https://livekit.io logo
Join Slack
Powered by
This message was deleted.
# helpdesk
s
This message was deleted.
m
cc @eager-raincoat-52616
a
@magnificent-art-43333 Thanks. I quickly checked my MediaSoup Android client and found, that it explicitly selects the OpenH264 software encoder if simulcast is required. On newer systems this works perfectly, because these are strong enough. Unfortunately most of the DJI flight controllers are still Android 7 devices....It is a while ago since I messed with the Android client, but I think to remember, that using H.264 hardware encoders with simulcast was a PITA.
e
it theoretically does support it (as in, it’ll try to do it), but some android devices don’t support simultaneous h264 hardware encoding
a
Good news. Is there any OOB quick-test app for Android, which could be used, compiled and run against my own server for check?
e
you could try one of the sample apps on the repo, just need to change the video publish codec to h264
a
e
yeah, that’d work
a
I unfortunately appear gradle build issues:
Copy code
/home/ubuntu/AndroidStudioProjects/client-sdk-android/sample-app/src/main/AndroidManifest.xml: Error: When targeting Android 13 or higher, posting a permission requires holding the POST_NOTIFICATIONS permission (usage from leakcanary.NotificationEventListener) [NotificationPermission]
  
   Explanation for issues of type "NotificationPermission":
   When targeting Android 13 and higher, posting permissions requires holding
   the runtime permission android.permission.POST_NOTIFICATIONS.
  
  
 The full lint text report is located at:
  /home/ubuntu/AndroidStudioProjects/client-sdk-android/sample-app/build/intermediates/lint_intermediate_text_report/debug/lint-results-debug.txt
Maybe the more speaking error is in the log: The project is using an incompatible version (AGP 8.0.2) of the Android Gradle plugin. Latest supported version is AGP 7.4.2
Looks like a downgrade of the buildtools to 7.4.2 did help
OK, sample-basic is running, but I suppose it is not the right sample. It perfectly joined the room and subscribed to the video from the PI, but I need to publish 🙂 EDIT: O sh... I see it is publishing in parallel. But practically no framerate and poor quality. Trying to figure out, how to do simulcast and H.264 now
No, sorry for bother, but this is not going to work for me. OK, I did run the app first on my OneNote, Android 13. This device is one of those, which does not support H.264 hardware encoding w/o any help of a patched libwebrtc. But if you run the "right" libwebrtc, it does 🙂...woodooo The result was: VP8, but not more than 20 fps at 960x540. So far away from what I'm used to see.... When changing the preference to H.264 (hoping I did it right):
Copy code
room.videoTrackPublishDefaults = room.videoTrackPublishDefaults.copy(videoCodec = "h264")
it doesn't even offer any video in the sent OFFER. Then I changed to my old Nexus 6P (Android 8.1), form which I know for sure that it supports H.264 hardware encoding out of the box, even with the oldest M8x libwebrtc. Unfortunately the same result: VP8 yes, H.264 no. I just hope to have made a mistake in selecting the desired coded, otherwise this doesn't work for me.
Correction: The Nexus sends a valid offer, containing H.264 only. And simulcast. But there seems to be no output, since the remote end does only receiver an audio track, no video.
Should stop for today, I'm seeing too many issues right now: The response from the LK server looks ugly... Several media sections are doubled, for instance 127 and 96. Not sure if this counts, but the Android client desparately tries to activate VP8, whereas the H.264 codecs are clearly placed at front... Pausing for now. m=video 9 UDP/TLS/RTP/SAVPF 127 127 96 96 39 98 96 a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f a=rtcp-fb:127 goog-remb a=rtcp-fb:127 transport-cc a=rtcp-fb:127 ccm fir a=rtcp-fb:127 nack a=rtcp-fb:127 nack pli a=rtcp-fb:127 goog-remb a=rtcp-fb:127 transport-cc a=rtcp-fb:127 ccm fir a=rtcp-fb:127 nack a=rtcp-fb:127 nack pli a=rtpmap:127 H264/90000 a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f a=rtcp-fb:127 goog-remb a=rtcp-fb:127 transport-cc a=rtcp-fb:127 ccm fir a=rtcp-fb:127 nack a=rtcp-fb:127 nack pli a=rtcp-fb:127 goog-remb a=rtcp-fb:127 transport-cc a=rtcp-fb:127 ccm fir a=rtcp-fb:127 nack a=rtcp-fb:127 nack pli a=rtpmap:96 VP8/90000 a=rtcp-fb:96 goog-remb a=rtcp-fb:96 transport-cc a=rtcp-fb:96 ccm fir a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli a=rtcp-fb:96 goog-remb a=rtcp-fb:96 transport-cc a=rtcp-fb:96 ccm fir a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli a=rtcp-fb:96 goog-remb a=rtcp-fb:96 transport-cc a=rtcp-fb:96 ccm fir a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli a=rtpmap:96 VP8/90000 a=rtcp-fb:96 goog-remb a=rtcp-fb:96 transport-cc a=rtcp-fb:96 ccm fir a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli a=rtcp-fb:96 goog-remb a=rtcp-fb:96 transport-cc a=rtcp-fb:96 ccm fir a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli a=rtcp-fb:96 goog-remb a=rtcp-fb:96 transport-cc a=rtcp-fb:96 ccm fir a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli a=rtpmap:39 AV1/90000 a=rtcp-fb:39 goog-remb a=rtcp-fb:39 transport-cc a=rtcp-fb:39 ccm fir a=rtcp-fb:39 nack a=rtcp-fb:39 nack pli a=rtpmap:98 VP9/90000 a=fmtp:98 profile-id=0 a=rtcp-fb:98 goog-remb a=rtcp-fb:98 transport-cc a=rtcp-fb:98 ccm fir a=rtcp-fb:98 nack a=rtcp-fb:98 nack pli a=rtpmap:96 VP8/90000 a=rtcp-fb:96 goog-remb a=rtcp-fb:96 transport-cc a=rtcp-fb:96 ccm fir a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli a=rtcp-fb:96 goog-remb a=rtcp-fb:96 transport-cc a=rtcp-fb:96 ccm fir a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli a=rtcp-fb:96 goog-remb a=rtcp-fb:96 transport-cc a=rtcp-fb:96 ccm fir a=rtcp-fb:96 nack a=rtcp-fb:96 nack pli
d
@adventurous-twilight-28575 can you attach the full answer from LK server that you've captured? It shouldn't be duplicating media sections. It seems suspicious you are getting limited FPS with VP8, that doesn't make a ton of sense to me. There should be no issues publishing with VP8 simulcast even on a moderately dated device. Re: H.264, as DL mentioned, we do not allow it to be used when simulcast is enabled due to unpredictable hardware encoder support. If this was allowed, how would you determine which device could truly support it?
a
@dry-elephant-14928 Yes, will catch the full exchange. Regarding H.264: I suppose simulcast was allowed, I recall, you are allowing it implicitly and I didn't disable it. I will also check the available frame rate and resolution with VP8. It was a first shot on a long day, maybe I wasn't concentrated enough anymore.
OK, first attempt this morning: H.264, no simulcast.
Copy code
room.videoTrackPublishDefaults = room.videoTrackPublishDefaults.copy(videoCodec = "h264", simulcast = false)


// Connect to server.
room.connect(
    url,
    token,
    options = ConnectOptions(autoSubscribe = false, audio = false)
)
I thought with "audio=false" I would also get rid of any audio parameters in offer and answer, but that was not the case. Audio was offered. The result was a video, which visibly ramped up in resolution. But it was no simulcast. I think, this is the normal behaviour of libwebrtc codecs, at least for H.264 observed many times. It finally landed at 544x960 @ 25fps (portrait). That doesn't speak for a hardware codec (or I missed some way to enable 720p @ 30fps), even though the codec signature in the Android traces looks like hardware codec for me: 
I/OplusFeedbackInfo: 0xb40000729bfdf200 c2.mtk.avc.encoder 544x960 inputFps=25 outputFps=25 renderFps=0, discardFps=25
Then I checked the OFFER/ANSWER game between Android and server. This time I couldn't find that much duplicates in the server's ANSWER, but there was at least one (127). I have removed all 
candidates
except my local network for privacy. There was also some more back and forth with offers and answers, I hope I was checking the right ones, video related: OFFER from Android
Copy code
v=0
o=- 3856291409814067501 3 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=extmap-allow-mixed
a=msid-semantic:WMS PA_JVc9WdMfdXE7
m=audio 58931 UDP/TLS/RTP/SAVPF 111 63 9 102 0 8 13 110 126
c=IN IP4 192.168.188.33
a=rtcp:9 IN IP4 0.0.0.0
a=candidate:3010281474 1 udp 2122194687 192.168.188.33 58931 typ host generation 0 network-id 3 network-cost 10
a=ice-ufrag:BEhm
a=ice-pwd:NCTvcGDqJ2smVL/+/eiwJSjs
a=ice-options:trickle renomination
a=fingerprint:sha-256 B0:7B:29:83:09:45:EC:F0:7A:4E:FB:2E:4C:E4:09:F4:A6:B7:76:50:4F:4D:02:07:A5:26:C4:45:4E:1B:A5:53
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:2 <http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time>
a=extmap:3 <http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01>
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendonly
a=msid:PA_JVc9WdMfdXE7 087dfd65-45cb-49fc-87de-a047356171ad
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:63 red/48000/2
a=fmtp:63 111/111
a=rtpmap:9 G722/8000
a=rtpmap:102 ILBC/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:13 CN/8000
a=rtpmap:110 telephone-event/48000
a=rtpmap:126 telephone-event/8000
a=ssrc:1484863682 cname:bQGemKAtMYo57Gx9
a=ssrc:1484863682 msid:PA_JVc9WdMfdXE7 087dfd65-45cb-49fc-87de-a047356171ad
m=application 48223 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 192.168.188.33
a=candidate:3010281474 1 udp 2122194687 192.168.188.33 48223 typ host generation 0 network-id 3 network-cost 10
a=ice-ufrag:BEhm
a=ice-pwd:NCTvcGDqJ2smVL/+/eiwJSjs
a=ice-options:trickle renomination
a=fingerprint:sha-256 B0:7B:29:83:09:45:EC:F0:7A:4E:FB:2E:4C:E4:09:F4:A6:B7:76:50:4F:4D:02:07:A5:26:C4:45:4E:1B:A5:53
a=setup:actpass
a=mid:1
a=sctp-port:5000
a=max-message-size:262144
m=video 9 UDP/TLS/RTP/SAVPF 127 103 127 103 104 105 96 97 39 40 98 99 104 105 106 107 108
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:BEhm
a=ice-pwd:NCTvcGDqJ2smVL/+/eiwJSjs
a=ice-options:trickle renomination
a=fingerprint:sha-256 B0:7B:29:83:09:45:EC:F0:7A:4E:FB:2E:4C:E4:09:F4:A6:B7:76:50:4F:4D:02:07:A5:26:C4:45:4E:1B:A5:53
a=setup:actpass
a=mid:2
a=extmap:14 urn:ietf:params:rtp-hdrext:toffset
a=extmap:2 <http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time>
a=extmap:13 urn:3gpp:video-orientation
a=extmap:3 <http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01>
a=extmap:5 <http://www.webrtc.org/experiments/rtp-hdrext/playout-delay>
a=extmap:6 <http://www.webrtc.org/experiments/rtp-hdrext/video-content-type>
a=extmap:7 <http://www.webrtc.org/experiments/rtp-hdrext/video-timing>
a=extmap:8 <http://www.webrtc.org/experiments/rtp-hdrext/color-space>
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=extmap:12 <https://aomediacodec.github.io/av1-rtp-spec/#dependency-descriptor-rtp-header-extension>
a=sendonly
a=msid:PA_JVc9WdMfdXE7 614c6f17-1359-4ae7-9713-c558fdc3e25c
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:127 H264/90000
a=rtcp-fb:127 goog-remb
a=rtcp-fb:127 transport-cc
a=rtcp-fb:127 ccm fir
a=rtcp-fb:127 nack
a=rtcp-fb:127 nack pli
a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtpmap:103 rtx/90000
a=fmtp:103 apt=127
a=rtpmap:127 H264/90000
a=rtcp-fb:127 goog-remb
a=rtcp-fb:127 transport-cc
a=rtcp-fb:127 ccm fir
a=rtcp-fb:127 nack
a=rtcp-fb:127 nack pli
a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtpmap:103 rtx/90000
a=fmtp:103 apt=127
a=rtpmap:104 H265/90000
a=rtcp-fb:104 goog-remb
a=rtcp-fb:104 transport-cc
a=rtcp-fb:104 ccm fir
a=rtcp-fb:104 nack
a=rtcp-fb:104 nack pli
a=rtpmap:105 rtx/90000
a=fmtp:105 apt=104
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:39 AV1/90000
a=rtcp-fb:39 goog-remb
a=rtcp-fb:39 transport-cc
a=rtcp-fb:39 ccm fir
a=rtcp-fb:39 nack
a=rtcp-fb:39 nack pli
a=rtpmap:40 rtx/90000
a=fmtp:40 apt=39
a=rtpmap:98 VP9/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 profile-id=0
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:104 H265/90000
a=rtcp-fb:104 goog-remb
a=rtcp-fb:104 transport-cc
a=rtcp-fb:104 ccm fir
a=rtcp-fb:104 nack
a=rtcp-fb:104 nack pli
a=rtpmap:105 rtx/90000
a=fmtp:105 apt=104
a=rtpmap:106 red/90000
a=rtpmap:107 rtx/90000
a=fmtp:107 apt=106
a=rtpmap:108 ulpfec/90000
a=ssrc-group:FID 1263303468 4291335920
a=ssrc:1263303468 cname:bQGemKAtMYo57Gx9
a=ssrc:1263303468 msid:PA_JVc9WdMfdXE7 614c6f17-1359-4ae7-9713-c558fdc3e25c
a=ssrc:4291335920 cname:bQGemKAtMYo57Gx9
a=ssrc:4291335920 msid:PA_JVc9WdMfdXE7 614c6f17-1359-4ae7-9713-c558fdc3e25c
ANSWER from server:
Copy code
v=0
o=- 7738754379143647972 1701157906 IN IP4 0.0.0.0
s=-
t=0 0
a=fingerprint:sha-256 CF:C8:C3:78:25:97:ED:C7:F3:E7:31:B1:C2:69:41:27:D9:20:13:66:73:E0:20:97:03:D1:87:05:55:AC:20:75
a=extmap-allow-mixed
a=group:BUNDLE 0 1 2
m=audio 9 UDP/TLS/RTP/SAVPF 63 111
c=IN IP4 0.0.0.0
a=setup:active
a=mid:0
a=ice-ufrag:chaUbTCluAOsCwSJ
a=ice-pwd:rnhSZQkCKpGsRXqGuLwVsnQIolzmIySX
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:63 red/48000/2
a=fmtp:63 111/111
a=rtpmap:111 opus/48000/2
a=fmtp:111 minptime=10;useinbandfec=1;usedtx=1
a=rtcp-fb:111 transport-cc 
a=rtcp-fb:111 nack 
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=recvonly
a=candidate:2994961013 1 udp 2130706431 192.168.188.88 7882 typ host
a=end-of-candidates
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=setup:active
a=mid:1
a=sendrecv
a=sctp-port:5000
a=ice-ufrag:chaUbTCluAOsCwSJ
a=ice-pwd:rnhSZQkCKpGsRXqGuLwVsnQIolzmIySX
m=video 9 UDP/TLS/RTP/SAVPF 127 127 96 39 98
c=IN IP4 0.0.0.0
a=setup:active
a=mid:2
a=ice-ufrag:chaUbTCluAOsCwSJ
a=ice-pwd:rnhSZQkCKpGsRXqGuLwVsnQIolzmIySX
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:127 H264/90000
a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtcp-fb:127 goog-remb 
a=rtcp-fb:127 transport-cc 
a=rtcp-fb:127 ccm fir
a=rtcp-fb:127 nack 
a=rtcp-fb:127 nack pli
a=rtcp-fb:127 goog-remb 
a=rtcp-fb:127 transport-cc 
a=rtcp-fb:127 ccm fir
a=rtcp-fb:127 nack 
a=rtcp-fb:127 nack pli
a=rtpmap:127 H264/90000
a=fmtp:127 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtcp-fb:127 goog-remb 
a=rtcp-fb:127 transport-cc 
a=rtcp-fb:127 ccm fir
a=rtcp-fb:127 nack 
a=rtcp-fb:127 nack pli
a=rtcp-fb:127 goog-remb 
a=rtcp-fb:127 transport-cc 
a=rtcp-fb:127 ccm fir
a=rtcp-fb:127 nack 
a=rtcp-fb:127 nack pli
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 goog-remb 
a=rtcp-fb:96 transport-cc 
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack 
a=rtcp-fb:96 nack pli
a=rtpmap:39 AV1/90000
a=rtcp-fb:39 goog-remb 
a=rtcp-fb:39 transport-cc 
a=rtcp-fb:39 ccm fir
a=rtcp-fb:39 nack 
a=rtcp-fb:39 nack pli
a=rtpmap:98 VP9/90000
a=fmtp:98 profile-id=0
a=rtcp-fb:98 goog-remb 
a=rtcp-fb:98 transport-cc 
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack 
a=rtcp-fb:98 nack pli
a=extmap:3 <http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01>
a=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=recvonly
Going to check VP8 simulcast now for frame rate and resolution
The VP8 quality is more or less not acceptable client-side this morning 🙂 It starts visibly at some higher resolution, but falls down soon to 181 x 320 (!) at 15 fps. I'm running on an OnePlus Android 13 or so, the device is not that bad, even though I don't use it. Application was run in debugger. No duplicates in server answer
Will be leaving Android for the rest of the week but would be available for follow ups and tests of all kind. I'd like to make that run more satisfyingly. We can even use another exchange channel, if you like. DM me, in case.
I need to revert my statement to VP8. I was having Network Link Conditioner active, w/o noticing it. So at least it proves, that simulcast is working for VP8, sorry for that. 544x960 @ 25 fps. All fine, except the lower frame rate. Running in portrait.
> Re: H.264, as DL mentioned, we do not allow it to be used when simulcast is enabled due to unpredictable hardware encoder support. If this was allowed, how would you determine which device could truly support it? Forgot to answer the question: Since I know my devices from other (and many other) WebRTC projects
Would it be possible to resume here? My particular questions: 1. The repetition of media sections, is that sane and can be ignored? 2. The low framerate: I can assure you that my OnePlus is capable of doing 30 fps @ 720 p in VP8 and H.264 and that it is also capable of doing simulcast H264 with 3 streams at the same frame rate, using libwebrtc M105 compiled with OpenH264 support. 3. I agree, getting proper simulcast support out of libwebrtc with H.264 hardware codecs, but I really would like to give that another try, now with your software. Would there be a way to unleash your code and let it try to go with H.264 hardware codecs for additional tests?
Anybody to help?
d
Hey Neil, how are you capturing and publishing the track? there shouldn't be a problem doing 30fps.
e
I think you might be using the default capture params, which is set at 960x540. you can change the capture params by passing in your own 
videoTrackCaptureDefaults
when creating the room like so:
Copy code
val room = LiveKit.create(
        appContext = application,
        options = RoomOptions(
            videoTrackCaptureDefaults = LocalVideoTrackOptions(
                captureParams = VideoPreset169.H720.capture 
                // or pass in your own values
                // captureParams = VideoCaptureParameter(1280, 720, 30)
            )
        ),
    )
a
For both VP8 and H.264 it is not reaching 720p I have used the 
roomOptions
element in the 
room.connect
function. Hoping it is correct. PS: But I know libwebrtc sometimes has its own opinion about encoding resolutions. Let me cross check this with my MS Android client.
e
try setting it in the 
LiveKit.create
function, I think that 
roomOptions
parameter was exposed on accident and won’t correctly reflect the capture params set there
a
Here two screenshots showing the results of my MediaSoup client in simulcast mode working against my MediaSoup server instance at AWS for H.264 and VP8. I admit the input is a test video made of 30 images taken from GStreamer videotestsrc and played back as I420 input at 30 fps to the encoder(s). There is not much complexity in the image, but I payed attention to have a fair comparison with your app by letting the cam stare at a white wall, so I guess it is similar. I'm using a M105 home made libwebrtc with some patches to reach out for more hardware support for H.264. In any case for the known reasons I prefer H.264 software encoders for simulcast too. > try setting it in the 
LiveKit.create
function, I think that 
roomOptions
parameter was exposed on accident and won’t correctly reflect the capture params set there OK, will try that later and report
Second screenshot
> try setting it in the 
LiveKit.create
function, I think that 
roomOptions
parameter was exposed on accident and won’t correctly reflect the capture params set there
Yes, this is working finally. 720x1280 (portrait) 30 fps for both. H.264 frame rate decreases dramatically to 20 fps if there is some complexity in the image (not so VP8), but I see that coming from the encoder already. Good so far. Now simulcast for both 🙂 I know it's been working for VP8. For H.264 the hardware codec seems to be used (in contrary to what has been said earlier), but the results are bull...t.
Copy code
I/OplusFeedbackInfo: 0xb4000076159e2c00 c2.mtk.avc.encoder 90x160 inputFps=1 outputFps=1 renderFps=0, discardFps=1
I/OplusFeedbackInfo: 0xb4000076381a3e00 c2.mtk.avc.encoder 720x1280 inputFps=1 outputFps=1 renderFps=0, discardFps=1
I/OplusFeedbackInfo: 0xb4000076d3da3e00 c2.mtk.avc.encoder 180x320 inputFps=1 outputFps=1 renderFps=0, discardFps=1
Remote displays a still image of 90x160 🙂 Any chance for providing an updated 
libwebrtc
by myself?
e
we use a forked version of libwebrtc here that isn’t fully api compatible with the base libwebrtc. if you patch the forked version with your own changes, you can exclude our libwebrtc dependency and substitute with your own
a
OK, thanks. Is the build process the same as documented by Google? The build process which ends up like so?
Copy code
./tools_webrtc/android/build_aar.py --extra-gn-args 'is_debug=false is_component_build=false is_clang=true rtc_include_tests=false rtc_use_h264=true rtc_enable_protobuf=false use_rtti=true use_custom_libcxx=true ffmpeg_branding="Chrome"' --build-dir ./out/release-build/
I was trying my Nexus 6P, form which I definitely know, that it supports HW H.264 non-simulcast encoding with 720p@30fps. I'm not getting more than 13 fps on the far end. (I'm sorry, all my messages leave too early with ENTER, I hate that somehow...) Here is what Android says when it tries to select the HW codec:
Copy code
W/VideoCapabilities: Unrecognized profile 2130706433 for video/avc
W/OMXUtils: do not know color format 0x7fa30c04 = 2141391876
W/OMXUtils: do not know color format 0x7f000789 = 2130708361
I/ACodec: setupAVCEncoderParameters with [profile: Baseline] [level: Level32]
I/ACodec: [OMX.qcom.video.encoder.avc] cannot encode color aspects. Ignoring.
I/ACodec: [OMX.qcom.video.encoder.avc] cannot encode HDR static metadata. Ignoring.
I/ACodec: setupVideoEncoder succeeded
13 fps is most likely not hardware
e
the build scripts we use are here: vpython ./tools_webrtc/android/build_aar.py --build-dir webrtc_android --output ./webrtc_android/libwebrtc.aar --arch armeabi-v7a arm64-v8a x86_64 x86 --extra-gn-args ‘is_java_debug=false rtc_include_tests=false rtc_use_h264=false is_component_build=false use_rtti=true rtc_build_examples=false treat_warnings_as_errors=false’
a
Sorry, I have updated my last post inline, since I always forget to NOT hit ENTER...
Try rtc_use_h264 = true for enabling OpenH264. BTW: I have a private pragmatic gist documenting exactly the required changes for libwebrtc files M114.5, and most likely also for upcoming versions. I could share that with you, in case you would consider to enable at least OpenH264 support.
e
IIRC we can't distribute openh264 due to licensing restrictions, which is why it's disabled
a
It’s an OS project with Simplified BSD license is there a major problem?
d
Neil, this is related to patents licensing instead of software. not a lawyer etc, but distributing binary encoders/decoders to end user typically involves dealing with that stuff, something we'd like to stay away from
a
Sure. If this is the case I would also drop that. Need to investigate on this.
Seems, you are right. I will have to revert those OpenH264 switches from my builds. Anyway, I still don't understand, why my Nexus 6P behaves that bad in this scenario (H.264 hardware, no simulcast). Would you see any other way to bring simulcast support for H.264 to Android?
I did some cross reading here https://github.com/cisco/openh264/releases Looks like Cisco (?) is still providing pre-compiled libopenh264 binaries, which cover the MPEG-LA fees. Not sure it this would be a way out of the dilemma. Maybe like so https://webrtcbydralex.com/index.php/2019/07/21/free-h264-in-lib-webrtc-for-free/ even though I have no idea, what they did there :)
d
we haven't dived too deep there, really just following WebRTC's lead here. IIRC, to get coverage you'd need to download those binaries directly from Cisco onto the end user's machine. Currently we are not investing any time into H.264 on Android, so this isn't something we've looked into deeply. unfortunately this is the state of the world for H.264, it's plagued by hardware encoder limitations and licensing challenges.
a
I see. And understand your position completely. Thanks for elaborating
Fun fact: Even my weakest Android device (which supports H.264 hardware encoding OOB) is capable to run 3 H.264 encodings in parallel at decent load (about 60 percent on a 4 cores, 15% on each). So the limitation is not the system, but libwebrtc, seemingly. This result encourages me to adopt my Python solution (using pre-encoded streams and provide that to a LiveKit module linked as DLL).
Copy code
I/OplusFeedbackInfo: 0xb40000766a9fca00 c2.mtk.avc.encoder 320x180 inputFps=30 outputFps=30 renderFps=0, discardFps=30
I/OplusFeedbackInfo: 0xb40000766a9fc800 c2.mtk.avc.encoder 640x360 inputFps=30 outputFps=31 renderFps=0, discardFps=30
I/OplusFeedbackInfo: 0xb4000076d4e6a200 c2.mtk.avc.encoder 1280x720 inputFps=29 outputFps=30 renderFps=0, discardFps=29
Finally, my Python approach (hardware encoding via GStreamer and Unix Pipes into a GOLANG module utilizing LK SDK) works also for Android. For the first time ever I'm having H.264 simulcast running from an Android device using a GOLANG module (utilizing GOMOBILE and LiveKit-SDK), which uses the inbuilt H.264 hardware encoding, but now no longer via libwebrtc but via MediaCodec API and UnixSockets. Prototype runs nicely.
🙌 1
2 Views
Open in Slack
PreviousNext
Powered by