A description of the higher level problem I'm trying to solve, so maybe you can help me to find a better alternative: • I want to do further processing (ML inference) with the video frames I receive from the media streams. A good way to do that is to have the video frames decoded by the GPU (nvdec or equivalent), so they are already in GPU memory, ready to be used for inference. Otherwise the decoded frames are copied to RAM, and then need to be copied again to the GPU.

And I take it your video source is livekit webrtc?

yep

I’ve been working the other direction with an external video source, but I think the

VideoTrack

can be read as an

I420

frame, but I haven’t looked to see if you could hook up a callback to receive each frame.

the problem is that the VideoFrame seems to be in RAM

We have plan to support NvCodec in the future. (Directly as a webrtc decoder). con.unity.webrtc is already implementing it here. But they are still copying the VideoFrame into the RAM. Maybe we can still create a custom VideoBuffer type with kNative

Yeah. I’ve been writing new

VideoFrames

from Gstreamer. I think you’ll get a

PeerConnection

from a

RemoteParticipant

and be able to use an

OnTrack

callback to send it to your process

thank you! yeah... for now I think the easiest would be for me to get the encoded frames and decode myself

This discussion is super interesting. So, @boundless-energy-78552 if I understand right, using the the

kNative

buffer, we could create a set of gstreamer plugins for

livekit-webtc

sources & sinks?

if

kNative

is a gpu surface, then gstreamer has support for it as something like

'video/x-raw(memory:CUDAMemory),format=I420'