After enabling “video” in python agent, configured for OpenAI Realtime API and gpt-realtime model (which is now supported in livekit-agents@1.2.9 — vision is working well, but appears to cause progressively greater latency (i.e. time from end-of-prompt to start-of-response) as the conversation continues. I’ve not tested extensively to prove with data that this is the dynamic, but it felt like that on a couple extended smoke tests of the agent… 1. Is it expected for video input to cause quick degradation of latency performance by the agent, as still images quickly stack up (at a rate of 1/sec when speaking and 0.33/sec when not speaking) in the chat transcript? 2. If so, are there built-in levers offered by LiveKit, which help to manage this — without entirely losing the benefit of the images in the chat transcript, so coherence across the conversation can still remain strong when discussing something previously shown in an image from N minutes earlier in the conversation? I think the ideal for me would be if it were possible to enable “compressing” of older images into a text description, so that key facets of the context that the image offered can be preserved as durable without needing to burn as many input tokens as the full image on every new request to the API. But I’m not sure if this or a similarly “compression-style” solution (i.e. not just dropping images entirely) is offered natively by LiveKIt, or if my only mitigation option would be to roll my own solution for managing older images in the transcript. Please help me understand whatever options LiveKit offers in this problem space, where to look for documentation, etc.

Thanks for reaching out, Dan. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

1. I’m using OpenAI’s Realtime API with gpt-realtime model; I understand this API accepts only still images, but when “video” is enabled with LiveKit and the OpenAI plugin, the LiveKit abstraction accepts video streaming from device to cloud agent and then automatically handles the sampling and encoding of frames and attaching them to API requests appropriately (which I think is subtly different from Gemini Live API, which actually accepts the full video feed all the way to their endpoint and handles any frame sampling/encoding internally). 2. My main focus is reduing LLM processing latency (was as high as 15 seconds on my last smoke test, with a very simple prompt like “I’m about to time how long it takes you to respond, so I’m about to start my stopwatch and you should reply ASAP, right NOW.” 3. I’d prefer automated solutions that only require enabling with a flag or simple tuning of parameters — as I’m not super technical and get nervous customizing the complexities of multimodal agent chat. If my only option is to wire up my own workflow for “compressing” images to text descriptions after some time/count/turns threshold, I’ll consider tackling that later. But if there was a simple way to preserve the meaning in the image with some kind of “lossy compression” (especially preserving any written words that were shown in a prior image), that would be hugely helpful.

Thank you so much for sharing the details about your setup and goals. I really appreciate you taking the time to clarify everything! Based on what you’ve shared, here are a few important points and some friendly suggestions to help you along: • The OpenAI Realtime API with the gpt-realtime model, when used with LiveKit, doesn’t natively support video input. Instead, LiveKit manages video streaming by sampling and encoding frames into still images, which are then included in the API requests. • To help reduce latency in LLM processing, LiveKit provides options like frame sampling (for example, capturing 1 frame per second while speaking, and 1 frame every 3 seconds when not) and resizing/encoding frames to keep data size manageable. These features are thoughtfully designed to limit how much visual data is sent to the model, which can really help speed things up. • At the moment, there isn’t a built-in way to “compress” older images into text descriptions. If you’re interested in that, it would require some custom logic on your side—like periodically swapping out older images in the chat transcript with summarized text descriptions. Since you’re leaning towards automated solutions, I’d suggest experimenting with adjusting the frame sampling and resizing settings to keep the data sent to the model as light as possible. And if you want to explore custom workflows for turning images into text summaries, that’s definitely doable, but it would be outside what LiveKit offers natively. Please feel free to reach out if you’d like any help or guidance with configuring these parameters—I’m here to support you every step of the way! Sources: plugins/openai/src/realtime/realtime_model.ts | OpenAI Realtime API integration guide | LiveKit Docs | agent/main.py | OpenAI and LiveKit partner to turn Advanced Voice into an API | livekit-plugins/livekit-plugins-google/livekit/plugins/google/beta/realtime...