why is my screen all green?

Thanks for reaching out, Aaron. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

I'm running it from an html page with frames being served from ffmpeg. The direct frames are correct but in livekit they are green

Thanks — that context helps. You're seeing green only in LiveKit when frames are served from FFmpeg. I can help troubleshoot. A few targeted questions to pinpoint the cause: • Which LiveKit SDK and version are you using in the browser (e.g., JS/TS web SDK, version)? • How are you injecting frames into LiveKit? (e.g., using the WebRTC insertable streams, a MediaStream/Canvas, the

createLocalVideoTrack

from frames, or server-side ingestion?) • What pixel format are your FFmpeg frames in when sent to the browser (e.g., YUV420p, NV12, RGB24)? If you’re piping raw frames, what pixel format did you specify to FFmpeg? • Are you converting or drawing frames into a Canvas before sending to LiveKit, or sending raw byte frames directly? • Do you see any warnings or errors in the browser console or LiveKit server logs when the stream starts? • Can you confirm the browser and OS where this happens and whether other browsers show the same issue? Quick likely cause: a pixel-format / YUV↔️RGB mismatch (green tint/solid green usually means chroma channels are misinterpreted). If you prefer, share the FFmpeg command you use to produce the frames and the code snippet that creates/sends the track — I’ll point out exactly where to convert formats or adjust rendering.

The green flickers, - Device/OS/Browser: Pop!_OS (Linux), NVIDIA RTX 3080. Browser: Chrome (desktop). - Where it happens: Only in LiveKit video track playback. The same frames saved to PNG look correct. - SDKs: - LiveKit JS (browser): livekit-client v2.16.1 (UMD build loaded from node_modules/livekit-client/dist/livekit-client.umd.js). - LiveKit Python SDK: FFI server v0.12.42 (log: livekit_ffi::server v0.12.42). - Ingestion method: Server‑side Python LiveKit SDK (not FFmpeg). - Using rtc.VideoSource(width, height) - LocalVideoTrack.create_video_track(...) - Per-frame rtc.VideoFrame(...) with raw bytes, then VideoSource.capture_frame(...). - Frame format & conversion attempts: - Frames are numpy BGR uint8 arrays, size (720, 720, 3); saved PNG looks normal. - Tried publishing as: RGB24, I420, NV12, RGBA, BGRA. - All formats still show green in LiveKit. - Raw buffer sizes match expected bytes (e.g., I420 = widthheight1.5). - UI: simple HTML page using LiveKit JS to join the room, auto‑subscribe to video/audio. - No JS errors besides audio autoplay (user‑gesture requirement). - Room shows remote track and “AvatarBot” is connected. - Repro: I can share the exact minimal Python code for VideoSource + VideoFrame publish if needed.

Thanks — this is very helpful detail. You're seeing green flicker only in LiveKit playback when publishing server-side numpy BGR frames via `rtc.VideoSource`/`VideoFrame`. That points to a pixel-format / channel-order mismatch. Here's the focused checklist and minimal fixes to try. 1) Most likely fix — convert BGR → RGB and publish as RGB24 • Your numpy frames are BGR (OpenCV default). Convert to RGB before creating the VideoFrame; the SDK will not swap channels for you. Example:

import cv2
rgb = cv2.cvtColor(bgr_frame, cv2.COLOR_BGR2RGB)
buf = rgb.tobytes()
video_frame = rtc.VideoFrame(
    type=rtc.VideoFrameType.RGB24,
    width=rgb.shape[1],
    height=rgb.shape[0],
    data=buf,
)
video_source.capture_frame(video_frame)

2) If you want YUV (I420/NV12) • You must convert properly to planar YUV and pack planes in the expected order/size. Passing RGB/BGR bytes as I420/NV12 will corrupt chroma and produce green artifacts. • Example conversion (I420) using OpenCV:

yuv = cv2.cvtColor(bgr_frame, cv2.COLOR_BGR2YUV_I420)
# yuv is planar: Y then U then V — pack as bytes and use VideoFrameType.I420 with correct sizes

• Confirm byte lengths: RGB24 = W_H_3; I420 = W_H_1.5; NV12 = W_H_1.5 with appropriate plane order. 3) Quick checks to validate • Save and inspect the exact bytes you send for a single frame (compare channel values at a pixel). • Publish a simple known-pattern frame (solid red/green/blue) and verify what the browser shows — this quickly shows channel swaps. • Inspect

<chrome://webrtc-internals>

to confirm codec/frame info. If converting BGR→RGB and publishing

RGB24

doesn't fix it, please paste a minimal reproducible snippet (server code that builds the VideoFrame and the exact

VideoFrameType

used). I’ll review the bytes/packing and point out the exact mismatch. Sources: client-sdk-unity-web | livekit/livekit_composite | DeepWiki | livekit/livekit_composite | DeepWiki

Repro details + what we already tried - OS/Device/Browser: Pop!_OS (Linux), RTX 3080, Chrome desktop - LiveKit SDKs: - Python SDK (server‑side publisher): livekit.rtc v1.0.23 (FFI server v0.12.42) - Browser JS SDK: livekit-client v2.16.1 (UMD build) - Ingestion method: server‑side publish via Python SDK: - rtc.VideoSource(width, height) - LocalVideoTrack.create_video_track(...) - per frame: rtc.VideoFrame(...) + VideoSource.capture_frame(...) - Frame source: numpy BGR uint8 arrays (H, W, 3) from OpenCV. - Frames are correct when saved to PNG. - Test pattern (color bars) also shows green when published to LiveKit. - Formats tested (all green): - RGB24 (after cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)) - I420 (cv2.cvtColor(bgr, cv2.COLOR_BGR2YUV_I420)) - NV12 (cv2.cvtColor(bgr, cv2.COLOR_BGR2YUV_NV12)) - RGBA / BGRA - Byte lengths checked and matched expected size (RGB24 = W×H×3, I420/NV12 = W×H×1.5) - Minimal publish snippet (RGB24 path): source = rtc.VideoSource(width, height) track = rtc.LocalVideoTrack.create_video_track("avatar-video", source) await room.local_participant.publish_track(track) rgb = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB) frame = rtc.VideoFrame( width=rgb.shape[1], height=rgb.shape[0], type=rtc.VideoBufferType.Value("RGB24"), data=rgb.tobytes(), ) source.capture_frame(frame) - Result: browser shows solid green / green flicker only for LiveKit playback; direct PNG saved from same frames is correct. - No browser JS errors besides audio autoplay warnings. If helpful I can share a minimal standalone script that publishes a generated test pattern frame (still green). ——— If you want, I can also attach a minimal script and ask them if there’s a known issue with the Python SDK’s VideoSource on Linux or if they want a different buffer type / stride handling. Separately on our side: next thing I’ll try is forcing codec to VP8/H264 (in case the encoder path is the culprit).

Thanks — great repro detail. This looks like either a stride/alignment mismatch or a buffer-layout issue between the Python FFI and the encoder. You’ve already tried correct pixel formats and conversions, so focus on these two targeted checks (quick, ordered steps): 1) Ensure row stride alignment (most likely cause) • Encoders/WebRTC often expect rows padded to 16-byte (or similar) boundaries. If each row is packed as width*3 without padding, the encoder can read extra bytes and produce green/cyan artifacts. • Fix: create a padded buffer so each row length is aligned (e.g., 16). Example:

def aligned_stride(width, bpp=3, align=16):
    return (width * bpp + align - 1) & ~(align - 1)

stride = aligned_stride(width, 3, 16)
padded = np.zeros((height, stride // 3, 3), dtype=np.uint8)  # or shape (height, stride, 3) if using bytes
padded[:, :width, :] = rgb  # rgb already converted BGR->RGB
data = padded.tobytes()
# use VideoFrame with data

• If the Python API exposes a `stride`/`row_stride` field, set it to the aligned value. If it does NOT, ensure the buffer you pass already has the aligned row layout. 2) Verify plane packing for YUV formats (if using I420/NV12) • For I420, pack as Y plane (W*H) then U (W/2 * H/2) then V (W/2 * H/2). For NV12, pack Y then interleaved UV. • Confirm total byte length equals expected (I420 = W_H_1.5). • Use OpenCV conversion helpers (e.g., cv2.COLOR_BGR2YUV_I420) but then verify the exact plane sizes/order you pass to VideoFrame. 3) Quick sanity tests to isolate: • Publish a single solid-color test frame (pure red, green, blue) after conversion/stride changes. If colors map correctly, it's fixed; if red/blue swap or green persists, stride/packing still wrong. • Force encoder to VP8 or H264 when publishing to rule out codec-specific decode issues. 4) Debug dump • Save the exact bytes you pass (e.g., frame.bin). Play with ffplay to confirm the bytes render as expected: ffplay -f rawvideo -pix_fmt rgb24 -s WxH frame.bin If you want, share a minimal standalone script (the small snippet that builds the VideoFrame and calls capture_frame). With that I can point out the exact byte/stride packing issue or confirm if a Python SDK stride field is needed. Sources: Rust SDKs | livekit/livekit_composite | DeepWiki | Yeah, but can it run DOOM?

We already convert BGR→RGB and publish RGB24, and also tried I420/NV12 (OpenCV conversion). We even bypassed MuseTalk and published a known test pattern (color bars). All formats still render green in LiveKit. Details: - Python SDK: livekit.rtc 1.0.23 (FFI server 0.12.42) on Linux - Browser JS SDK: livekit-client 2.16.1 - Publish method: VideoSource + VideoFrame + capture_frame - Frame bytes match expected sizes (RGB24 = W×H×3; I420 = W×H×1.5) - VideoFrame in this SDK doesn’t expose custom stride; it always uses width * bytesPerPixel. Do you know if there’s a known issue with server‑side VideoSource on Linux, or a requirement to force a specific codec/encoder? We can also share a minimal test‑pattern publisher if helpful.

Thanks — I hear you: you’ve tried conversions, known test patterns, and sizes match, and the Python VideoFrame API doesn't expose stride. Given that, here’s the concise guidance and next steps. Short answer • There is no documented, widely‑known LiveKit bug specifically for Linux server-side VideoSource that forces green output. The most likely causes remaining are encoder/FFI expectations about memory alignment or an encoder/code path on Linux that misinterprets the buffer (so forcing codec or providing a repro are the quickest next steps). What to try next (ordered, quick) 1. Force the encoder to VP8 or H264 when publishing (rules out decoder quirks). ◦ Publish with explicit VideoEncoding/codec option (VP8 or H264) and test. 1. Create a minimal repro script and share it (most helpful). ◦ A 30–50 line script that creates a color‑bar RGB24 buffer, constructs VideoFrame, calls capture_frame, and publishes is ideal. I can inspect padding/bytes and confirm API usage. 1. If you can’t share the script, try this local debug step: ◦ Save the exact bytes you pass to VideoFrame to disk (frame.bin). ◦ Play it locally with ffplay to confirm the bytes render correctly: ffplay -f rawvideo -pix_fmt rgb24 -s WIDTHxHEIGHT frame.bin • If ffplay shows correct colors, the problem is in the ingestion/encoding path. 1. Try a different ingestion path on Linux as a comparison: ◦ Publish the same frames through a browser-based track (canvas→MediaStream) or use a GStreamer→LiveKit pipeline; if those render correctly, the issue is isolated to the Python FFI/encoder path. 1. As a last targeted check: alter image width to a multiple of 16 (or try several widths) and republish — even though SDK doesn’t expose stride, some encoders require aligned row sizes. What I can do for you • If you share the minimal test publisher script (or paste it here), I’ll inspect the exact VideoFrame creation and bytes and point to the precise mismatch or confirm it’s an encoder-side issue. • If you prefer, tell me which codec you force-tested (VP8 or H264) and the result — I’ll recommend the next low-level check. Which would you like to do next: share the minimal repro script, or confirm whether forcing VP8/H264 changed the behavior?

Subject: Server‑side VideoSource publishes green frames (Python SDK, Linux) thanks for the guidance. We already tried the recommended pixel‑format conversions and a hardcoded test pattern, but everything still renders green in LiveKit. Environment - OS: Pop!_OS (Linux) - GPU: RTX 3080 - Browser: Chrome desktop - LiveKit JS SDK: livekit-client 2.16.1 (UMD) - LiveKit Python SDK: livekit.rtc 1.0.23 (FFI server v0.12.42) Ingestion method Server‑side publish using Python SDK: - VideoSource(width, height) - LocalVideoTrack.create_video_track(...) - VideoFrame(...) + VideoSource.capture_frame(...) Important: Frames are not FFmpeg pipe — they’re numpy arrays. Saving the same frames to PNG looks correct. Formats tested (all green) - RGB24 (after cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)) - I420 / NV12 (cv2.cvtColor(bgr, cv2.COLOR_BGR2YUV_I420/NV12)) - RGBA / BGRA - We also tried VideoFrame.convert(I420) in SDK - Test pattern (solid RGB bars) still shows green Minimal repro (RGB24) import cv2 import numpy as np from livekit import rtc source = rtc.VideoSource(width, height) track = rtc.LocalVideoTrack.create_video_track("avatar-video", source) await room.local_participant.publish_track(track) # Test pattern frame = np.zeros((height, width, 3), dtype=np.uint8) frame[:, :width//3] = (0, 0, 255) # red frame[:, width//3:2*width//3] = (0, 255, 0) # green frame[:, 2*width//3:] = (255, 0, 0) # blue rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) video_frame = rtc.VideoFrame( width=rgb.shape[1], height=rgb.shape[0], type=rtc.VideoBufferType.Value("RGB24"), data=rgb.tobytes(), ) source.capture_frame(video_frame) Checks - Buffer sizes match expected (RGB24 = W×H×3, I420/NV12 = W×H×1.5) - PNG dump looks correct, but LiveKit playback is green Question Is there a known issue with Python VideoSource on Linux, or an encoder/stride requirement not exposed via the Python SDK? The VideoFrame API in this version doesn’t allow setting per‑row stride. Happy to provide a minimal script or logs if you want. ——— If you want, I can also attach the I420 variant snippet.

Thanks — I appreciate the clear repro. You’ve done the right conversions and tests; given that, the most likely root cause is a stride/pitch or memory-layout mismatch in the Python FFI path (the Python VideoFrame API doesn’t let you set per‑row stride, and the underlying encoder expects aligned rows). That mismatch causes the encoder to misread chroma/luma and produces solid/green artifacts. Next steps I recommend (ordered, quick): 1. Try widths that are already stride‑friendly (multiples of 16/64). If a stride‑aligned width renders correctly, that confirms stride is the issue. 2. Force a specific codec on the client (VP8 or H264) and test: ◦ If forcing VP8/H264 fixes it, the default encoder path was the problem. ◦ If it doesn’t, stride/packing remains the likely cause. 1. Quick local sanity check: save the exact bytes you pass to VideoFrame (frame.bin) and play with ffplay: ffplay -f rawvideo -pix_fmt rgb24 -s WIDTHxHEIGHT frame.bin • If ffplay shows correct colors, the problem is in ingestion/encoding, not the source bytes. 1. Alternative ingestion for a short-term workaround: ◦ Use an FFmpeg pipe or GStreamer → LiveKit pipeline on Linux to publish the same frames; if that renders correctly, it confirms the Python FFI/stride path is the blocker. 1. Share a minimal reproducible script (30–50 lines) that publishes the test pattern. If you paste it here I will review the exact VideoFrame creation and bytes and call out the precise packing/stride mismatch or confirm it’s an encoder/FFI limitation. Background note (why this is happening): The Python SDK currently does not expose a

stride

field for VideoFrame, while libwebrtc expects per‑plane stride/pitch and sometimes aligned rows. That mismatch is a known limitation pattern (not a public / documented bug), which is why padding/stride or using an alternate ingestion path is the practical path forward. Which do you want to try next — force VP8/H264 and report results, or paste your minimal repro script so I can inspect the exact bytes? Sources: Rust SDKs | livekit/livekit_composite | DeepWiki