I would like to allow livekit get like a realtime information on the position of certain characters in my screen and then communicate with the frontend to show a dot there. Is it possible to do this while still having the AI in context?

Thanks for reaching out, Brian. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

So the idea is the AI will scan the screen and know where the character is and send the update to the frontend on where it should place the markers this is why I think we need multi modal

Great — that clarifies the flow. Yes: you can have a multimodal AI scan the screen (video or images), extract character positions, and send those positions to the frontend over LiveKit data messages so the UI draws markers — while keeping the AI context active. Below is a concise plan plus a few targeted questions so I can give concrete next steps for your setup. How this can work end-to-end • Capture frames or a video stream that the AI can analyze. This can be: ◦ A screen-share/video track published into the LiveKit room, or ◦ Periodic screenshots sent to a backend AI service for analysis. • AI (multimodal) analyzes frames, detects character positions (x,y + id + optional timestamp/velocity). • AI/backend publishes position updates into the LiveKit room as data messages (or data tracks): ◦ Include an id for each character, timestamp, and optionally confidence/velocity. ◦ Use RELIABLE for important updates, or UNRELIABLE for high-rate updates where occasional loss is okay. • Frontend listens for data messages and renders dots/markers at the supplied coordinates, using interpolation/smoothing if needed. • Maintain AI conversation/context in session state: ◦ Keep persistent facts in session.userdata and full history in session.history. ◦ Include recent position samples in the agent’s chat context when the AI needs to reason about spatial relationships. Why multimodal is useful here • Visual input (frames/screen) lets the AI detect and track characters. • Text/voice context helps the AI interpret what to track or how to label markers (e.g., “track my ally”). • Combining both lets the AI update positions while still carrying conversational state and intent. Practical considerations and recommendations • Where to run the AI: ◦ Backend service: more control, easier to keep context and publish messages. ◦ In-room agent: possible if you run the agent as a participant that can both see the track and publish data. • Update frequency: ◦ Moderate (5–15 Hz) is often sufficient and reduces bandwidth. For very smooth motion, 30 Hz or client-side interpolation can be used. • Packet kind: ◦ Use UNRELIABLE for very high-rate, non-critical updates; RELIABLE for critical or less frequent updates. • Coordinate system: ◦ Agree on coordinate space (canvas pixels vs normalized 0–1) and include a viewport id/size with messages so frontend maps coordinates correctly. • Timestamps & smoothing: ◦ Include timestamps; do client-side interpolation/extrapolation to smooth jitter. • Bandwidth/load: ◦ If analyzing full-frame frequently, consider sending downscaled frames or only regions of interest to the AI to reduce load. Clarifying questions so I can give exact code snippets and architecture • How is the screen being provided to the AI now? (screen-share track in LiveKit, periodic screenshots, or something else) • Is the AI running in your backend or as an in-room agent/participant? • Which frontend tech are you using (React + canvas, WebGL, plain DOM, game engine)? • Expected update rate (low: <10 Hz, medium: 10–30 Hz, high: >30 Hz)? • Do you want markers to be globally authoritative from the AI, or should the frontend perform local prediction/authoritative reconciliation? If you tell me those details I’ll provide: • A concrete publish/subscribe example for your environment. • Recommendations on message format (JSON schema) including timestamps and coordinate normalization. • Guidance on RELIABLE vs UNRELIABLE and client smoothing techniques.

1. Screenshare track since sending periodic screenshots can be hard on low end mobiles I can be wrong though need help to decide 2. AI is running on the backend, its an in room participant 3. Am using nextjs its not a canvas its presentation style like Google meet 4. Expected update rate is low since it happens on slide changes 5. Do I want markers to be globally authoritative I'm not sure but I believe so, not sure I understand the terms

Thanks — this is a clear setup. I’ll address each of your points, clarify the “authoritative” concept, and give specific recommendations for your Next.js presentation UI with a backend AI participant analyzing a screen-share track. Short answers up front • Screen-share track is the sensible default for low-end mobiles (less upload overhead than repeatedly uploading screenshots). • Running the AI on the backend as an in-room participant is a good approach: it can subscribe to the screen-share track and publish position updates as data messages. • Low update rate (on slide changes) makes RELIABLE delivery reasonable and simplifies smoothing. • “Globally authoritative” means the AI’s published positions are treated as the single source of truth for marker placement; the frontend simply renders what the AI says (no local prediction or user-side overrides unless reconciled). Recommendations and concrete flow 1) Capture and analyze the screen-share • Have the presenter publish a screen-share track into the LiveKit room. • The backend AI participant subscribes to that track (it receives frames) and runs multimodal detection to locate characters/objects on slide changes. 2) Data message schema (keep it simple and explicit) • Use normalized coordinates (0..1) so different viewport sizes map cleanly. • Include id, normalized x/y, timestamp, optional confidence, and viewport size if you prefer pixel coordinates. • Example JSON: { "type":"position", "id":"char-1", "x":0.42, "y":0.73, "ts":1670000000000, "confidence":0.95 } 3) Publishing updates from the AI participant • Because your updates are low-frequency (on slide changes), publish as RELIABLE data packets so messages aren’t lost. • Publish only when positions change (or when the slide changes) to reduce noise. 4) Frontend handling (Next.js presentation-style) • The UI subscribes to LiveKit data messages and maps normalized coordinates to the presentation DOM or overlay layer (e.g., absolutely-positioned divs over the slide). • Use the viewport size or scale transform to convert normalized coords to pixel positions. • Render markers directly from the AI data if you want the AI to be authoritative. 5) Authoritative vs local prediction — explained • Authoritative: AI updates are the source of truth. Frontend renders exactly what the AI publishes. This avoids conflicts and keeps everyone synchronized, especially useful when multiple viewers must see the same markers. • Local prediction: The frontend may locally estimate positions (or allow the presenter to adjust) and later reconcile with AI. Useful if latency or intermittent disconnects would otherwise create poor UX. • For your low-frequency slide-change case, authoritative AI updates are simpler and preferred. 6) Practical tips and pitfalls • Coordinate space: normalized coordinates (0..1) are best for presentation UIs; include a viewport id/slide id in messages if the same session can show different pages/layouts. • Timestamps: include them so the frontend can ignore stale messages when slides change rapidly. • Versioning: include a slide or frame id to ensure markers correspond to the correct slide. • Security: ensure only authorized participants (your AI backend) can publish position messages or include an id/role in payloads to validate source. • Rate control: publish only on meaningful changes (slide transition or when detection differs) to reduce processing and network load. 7) Example publish / receive pattern (conceptual) • AI backend (as in-room participant) publishes: room.localParticipant.publishData( new TextEncoder().encode(JSON.stringify(payload)), DataPacket_Kind.RELIABLE ); • Next.js frontend: room.on('dataReceived', (payload) => { const d = JSON.parse(payload.utf8Value); if (d.type === 'position' && d.slideId === currentSlideId) { const left = d.x * slideWidth; const top = d.y * slideHeight; // position an absolutely-positioned marker element } }); Follow-up questions so I can provide exact code • Do you want normalized coordinates (0..1) or pixel coordinates for the markers? I recommend normalized. • Do slides have different aspect ratios or dynamic resizing (responsive)? If so, I’ll show how to convert coords robustly. • Would you like a small example component for Next.js that overlays markers on presentation slides (HTML/CSS example)? If you want, I can produce: • A ready-to-use JSON schema and TypeScript types for the messages. • A small Next.js overlay example (no canvas) that maps normalized coords to DOM markers and handles slide changes and stale messages.