Does the screen share work for both Realtime agents & TTS/STT Pipelines agents?

Thanks for reaching out, Thor. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

React frontend, using livekit cloud hosting with a python built voice agent. TTS/STT pipeline

Thanks — I’ll help you enable screen sharing for a React frontend (LiveKit Cloud) with a Python voice agent using the TTS/STT pipeline. Quick confirmation: screen sharing is supported for TTS/STT pipeline agents and the agent will receive the published screen track as the video input (LiveKit uses the most recently published video track). To enable it in your React app and ensure the Python voice agent receives the screen stream, follow these steps: 1. Publish a screen track from your React client • Use the LiveKit React SDK to capture and publish a screen track. Example (browser):

import { connect, createLocalVideoTrack } from "livekit-client";

// get room (after connecting with token)
const room = await connect(url, token);

// capture screen
const screenStream = await (navigator.mediaDevices as any).getDisplayMedia({ video: true, audio: true });
// create track(s) from MediaStream
const [screenTrack] = await createLocalVideoTrack(screenStream.getVideoTracks()[0]);
// publish
await room.localParticipant.publishTrack(screenTrack);

• Ensure you publish audio if you want system/app audio captured (getDisplayMedia({ audio: true })) and your browser supports it. 1. Ensure only the intended video track is published (LiveKit uses the most recent video track) • If you have an active camera track and then publish a screen track, the most recently published video track becomes the agent’s input. Unpublish or stop the camera track if you want screen-only input:

const cameraTrack = await createLocalVideoTrack();
await room.localParticipant.unpublishTrack(cameraTrack);
cameraTrack.stop();

1. Confirm agent-side (Python voice agent / TTS-STT pipeline) receives video • The TTS/STT pipeline agent subscribes to VideoTrack instances published in the room. No extra config is typically required in the agent beyond subscribing to the participant’s video track(s). • Verify the agent’s subscription logic is listening for incoming video tracks and processing the currently active video track. 1. Troubleshooting checks • Confirm the React client is connected to the same LiveKit Cloud room and the token grants publish/subscribe permissions. • Verify browser permissions for screen sharing and (if needed) audio. • Make sure the agent’s code subscribes to remote video tracks and that it processes the latest published video track. If your agent expects a specific participant identity or metadata, confirm those match. • If your app dynamically switches between camera and screen, ensure your logic unpublishes/stops unused tracks so the agent receives the expected stream. If you’d like, I can: • Provide a complete React component example for screen sharing and track switching. • Review the Python agent subscription snippet you’re using to confirm it picks up screen tracks. Which do you want next? , with interactive elements

Provide the python snippet for setting up the agent with the TTS/STT pipeline, specifically tailored for screen sharing

Below is a concise Python example showing how a TTS/STT pipeline voice agent can connect to a LiveKit room, subscribe to remote video tracks (including screen share), and forward frames to your pipeline. It assumes you’re using LiveKit’s Python server SDK or a WebSocket-based agent that receives LiveKit participant/track events. Adjust imports and pipeline hooks to match your code.

# Example: Python TTS/STT pipeline agent subscribing to screen-share video
# - Connects to LiveKit room (use appropriate auth/token)
# - Subscribes to remote video tracks
# - Extracts frames and sends them to a placeholder `process_frame()` handler
# Replace SDK/client code with your LiveKit client implementation.

import asyncio
from livekit import Room, RoomEvent  # adjust import to your LiveKit Python client
# If you don't have an official client, use an existing websocket client library
# and handle LiveKit signalling/events similarly.

async def process_frame(frame):
    """
    Placeholder: send frame to your TTS/STT pipeline or vision processor.
    `frame` could be raw bytes, a numpy array, or an encoded image depending on your capture method.
    """
    # e.g., convert to numpy and run vision model, or pass into STT/TTS flow
    pass

async def handle_track_published(participant, track):
    # Called when a remote participant publishes a track
    if track.kind == "video":
        print(f"Video track published by {participant.identity}: {track.sid} ({track.source})")
        # Subscribe to the track (some clients auto-subscribe)
        await track.subscribe()
        # Start consuming frames
        asyncio.create_task(consume_video_track(track))

async def consume_video_track(track):
    """
    Consume frames from a subscribed video track. Implementation depends on SDK:
    - Some SDKs provide `track.on_frame(callback)` or an async iterator.
    - Others provide raw RTP or encoded frames; decode accordingly.
    """
    try:
        async for frame in track.frames():  # pseudocode: iterate incoming frames
            # frame may include metadata like `source` indicating "screen" vs "camera"
            # If you need to only process screen sharing, check frame.source or track.source
            if getattr(track, "source", None) == "screen" or getattr(frame, "source", None) == "screen":
                await process_frame(frame)
            else:
                # Optionally process camera frames or ignore
                await process_frame(frame)
    except Exception as e:
        print("Video consumption ended:", e)

async def run_agent(livekit_url, token, room_name):
    room = await Room.connect(livekit_url, token, room_name=room_name)

    # Listen for existing participants and tracks
    for p in room.participants:
        for t in p.tracks:
            if t.kind == "video":
                asyncio.create_task(consume_video_track(t))

    # Listen for future track publishes
    async for event in room.events():  # pseudocode event loop
        if isinstance(event, RoomEvent.TrackPublished):
            await handle_track_published(event.participant, event.track)
        elif isinstance(event, RoomEvent.Disconnected):
            break

if __name__ == "__main__":
    livekit_url = "wss://your.livekit.cloud"
    token = "<YOUR_JOIN_TOKEN>"
    room_name = "my-room"
    asyncio.run(run_agent(livekit_url, token, room_name))

Notes and tips: • LiveKit exposes track

source

metadata (e.g., "camera" vs "screen")—use it to detect screen shares. • Ensure your agent’s token grants subscribe permissions for the room. • If your SDK provides encoded frames (VP8/VP9/H264), decode them to images before sending to vision/STT logic. • If multiple video tracks exist, LiveKit uses the most recently published video track as the primary input; track

published_at

or

source

can help select which to process. • If you want a concrete integration with your specific TTS/STT code, share the pipeline entrypoint (how it expects frames) and I’ll adapt the snippet., with interactive elements

Great. Thanks for sending the python code. Please send the react code that matches, for my frontend

Below is a concise React example that matches the Python agent snippet. It connects to LiveKit Cloud, publishes a screen-share track (with optional audio), ensures the screen track becomes the active video track (unpublishes camera if desired), and includes simple UI controls. Save as a React component (TypeScript example). Adjust imports, token retrieval, and styling to match your app.

import React, { useState, useRef } from "react";
import {
  connect,
  Room,
  LocalVideoTrack,
  createLocalVideoTrack,
} from "livekit-client";

type Props = {
  livekitUrl: string; // e.g. "wss://ws.livekit.cloud"
  getToken: () => Promise<string>; // function that returns a join token for the room
  roomName: string;
};

export default function ScreenShareAgent({ livekitUrl, getToken, roomName }: Props) {
  const [room, setRoom] = useState<Room | null>(null);
  const [isSharing, setIsSharing] = useState(false);
  const localCameraTrack = useRef<LocalVideoTrack | null>(null);
  const localScreenTrack = useRef<LocalVideoTrack | null>(null);
  const localVideoEl = useRef<HTMLVideoElement | null>(null);

  async function join() {
    if (room) return;
    const token = await getToken();
    const r = await connect(livekitUrl, token, { roomName });
    setRoom(r);

    // Optionally create & publish camera track on join
    try {
      const camTrack = await createLocalVideoTrack();
      localCameraTrack.current = camTrack;
      await r.localParticipant.publishTrack(camTrack);
      // attach to local preview
      camTrack.attach(localVideoEl.current!);
    } catch (e) {
      console.warn("Camera unavailable or blocked:", e);
    }
  }

  async function startScreenShare() {
    if (!room) await join();
    if (!room) return;

    try {
      // request display media (video + optional audio)
      // audio capture from system/apps is browser-dependent
      const stream = await (navigator.mediaDevices as any).getDisplayMedia({
        video: true,
        audio: true,
      });

      // create LocalVideoTrack from the first video track of the MediaStream
      const videoTrack = await createLocalVideoTrack(stream.getVideoTracks()[0]);
      localScreenTrack.current = videoTrack;

      // Publish screen track
      await room!.localParticipant.publishTrack(videoTrack);

      // Ensure screen becomes the active video input: unpublish camera if needed
      if (localCameraTrack.current) {
        try {
          await room!.localParticipant.unpublishTrack(localCameraTrack.current);
          localCameraTrack.current.stop();
          localCameraTrack.current = null;
        } catch (_) {}
      }

      // Attach screen track for local preview
      videoTrack.attach(localVideoEl.current!);
      setIsSharing(true);

      // Listen for when user stops the screen share via browser UI
      const track = stream.getVideoTracks()[0];
      track.onended = () => stopScreenShare();
    } catch (err) {
      console.error("Screen share failed:", err);
    }
  }

  async function stopScreenShare() {
    if (!room) return;
    const track = localScreenTrack.current;
    if (!track) return;

    try {
      await room.localParticipant.unpublishTrack(track);
    } catch (_) {}
    try {
      track.stop();
    } catch (_) {}
    localScreenTrack.current = null;
    setIsSharing(false);

    // Optionally re-enable camera
    try {
      const camTrack = await createLocalVideoTrack();
      localCameraTrack.current = camTrack;
      await room.localParticipant.publishTrack(camTrack);
      camTrack.attach(localVideoEl.current!);
    } catch (_) {}
  }

  async function leave() {
    if (!room) return;
    try {
      await room.disconnect();
    } finally {
      setRoom(null);
      setIsSharing(false);
      localCameraTrack.current = null;
      localScreenTrack.current = null;
    }
  }

  return (
    <div>
      <video ref={localVideoEl} autoPlay playsInline muted style={{ width: "640px", height: "360px", background: "#000" }} />
      <div style={{ marginTop: 8 }}>
        {!room ? (
          <button onClick={join}>Join Room</button>
        ) : !isSharing ? (
          <button onClick={startScreenShare}>Start Screen Share</button>
        ) : (
          <button onClick={stopScreenShare}>Stop Screen Share</button>
        )}
        {room && <button onClick={leave} style={{ marginLeft: 8 }}>Leave</button>}
      </div>
    </div>
  );
}

Notes and tips: • Use

getDisplayMedia({ audio: true })

only if you need system/app audio; browser support varies. • The agent (Python TTS/STT pipeline) will receive the published screen VideoTrack. LiveKit sets the most recently published video track as primary; unpublishing the camera ensures the screen track is used. • Ensure the join token returned by

getToken()

has publish/subscribe permissions for the room and that the Python agent is connected and subscribing to video tracks., with interactive elements