Is it possible to add images to gemini realtime ?

Thanks for reaching out, Aditya. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

Like HOW ?

You're asking how to add images to Gemini Realtime — here’s a concise, copy‑pasteable example and the steps to enable live video/frame input. 1) Install the Google plugin (if using the LiveKit agents package):

uv add "livekit-agents[google]~=1.3"

2) Minimal Python example — enable video input and (optionally) customize sampling:

from livekit.agents import AgentSession, VideoSampler
from livekit.plugins import google
import livekit.protocol.room_io as room_io
import os
import asyncio

os.environ["GOOGLE_API_KEY"] = "YOUR_GOOGLE_API_KEY"

async def run(session_ctx):
    llm = google.realtime.RealtimeModel(
        model="gemini-2.5-flash-native-audio-preview-09-2025",
        voice="Puck",
        temperature=0.8,
    )

    # Optional: customize frames per second
    video_sampler = VideoSampler(
        active_speaker_sps=1.0,    # frames/sec while speaking
        inactive_sps=0.333         # frames/sec while silent (1 every 3s)
    )

    session = AgentSession(
        llm=llm,
        video_sampler=video_sampler,  # optional
    )

    await session.start(
        room=session_ctx.room,
        room_options=room_io.RoomOptions(
            video_input=True,   # ← enables sending frames (images) to Gemini Live
        ),
    )

# session_ctx is your existing context that includes a connected LiveKit room
# run inside your async context that has `session_ctx`

What happens after enabling `video_input=True`: • LiveKit samples frames from the user's camera or screen share and sends them to Gemini Live. • Default sampling: ~1 frame/sec while speaking, ~1 frame every 3s when silent. • Frames are resized to fit within 1024×1024 and JPEG-encoded automatically. • Only the most recently published video track is used. 3) Customization notes: • Change sample rates with

VideoSampler

(adjust

active_speaker_sps

/

inactive_sps

). • LiveKit handles encoding and resizing; you do not need to manually send images or URLs. • Ensure your chosen Gemini Live model supports multimodal realtime (use a Gemini Live-capable model). Would you like a JavaScript/WebRTC example for a browser client or help wiring this into your existing session code? Sources: Vision | LiveKit docs | Gemini Live API plugin | LiveKit docs

Is it possible to intercept the transcription in gemini realtime ?

Acknowledging your question about intercepting transcriptions in Gemini Realtime — yes, you can. You can capture real-time transcriptions by listening to the AgentSession

user_input_transcribed

event (it provides interim and final transcripts). Example (Python):

from datetime import datetime

@session.on("user_input_transcribed")
def on_transcript(transcript):
    if transcript.is_final:
        ts = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
        print(f"[{ts}] FINAL: {transcript.transcript}")
    else:
        print(f"INTERIM: {transcript.transcript}")

Other options: • Use a

transcription_node

in your agent pipeline to access transcription text and timing for further processing or TTS alignment. • Log, store, or run custom logic on transcripts as they arrive (interim or final). Would you like a full example showing saving transcripts to a DB, or how to attach a transcription_node in your agent pipeline? Sources: How to capture real-time agent transcriptions during conversations | Text and transcriptions | LiveKit docs | Pipeline nodes and hooks | LiveKit docs

How can i make the gemini realtime to see what's going on in a screenshare ?

You're asking how to let Gemini Realtime "see" a screen share — yes, you can. LiveKit will send the published screen-share video track as frames to Gemini Live when video input is enabled. Here’s how to do it and key details. 1) Enable video input for the AgentSession • Set

video_input=True

(or

video_input

in RoomOptions) when starting the session so LiveKit samples frames from the active video track (camera or screen share). Python example:

from livekit.agents import AgentSession
from livekit.plugins import google
import livekit.protocol.room_io as room_io

session = AgentSession(
    llm=google.realtime.RealtimeModel(model="gemini-..."),
)

await session.start(
    room=ctx.room,
    room_options=room_io.RoomOptions(
        video_input=True,
    ),
)

2) Publish the screen-share track in the LiveKit room • From the client (browser or native), publish a screen-capture track like any other video track (e.g., getDisplayMedia in the browser). • LiveKit will consider the most recently published video track (camera or screen) as the source to sample frames from. Browser example (high level): • Use

navigator.mediaDevices.getDisplayMedia()

to obtain the screen track. • Publish it to the LiveKit room so the agent session can access it. 3) Sampling behavior & encoding (built-in) • Default sampling: ~1 frame/sec while speaking, ~1 frame every 3s when silent. • Frames are resized to fit within 1024×1024 and JPEG-encoded automatically. • If the user publishes multiple video tracks, the most recent published video track is used. 4) Customize sampling rates (optional) • Pass a

VideoSampler

to

AgentSession

to change sampling rates:

from livekit.agents import VideoSampler

video_sampler = VideoSampler(
    active_speaker_sps=2.0,   # frames/sec while speaking
    inactive_sps=0.5          # frames/sec while silent
)

session = AgentSession(
    llm=google.realtime.RealtimeModel(...),
    video_sampler=video_sampler,
)

5) Notes and best practices • Ensure your chosen Gemini Live model supports visual input. • Screen-share frames are sent directly to Gemini Live (you don’t send external image URLs). • If you need higher temporal fidelity (video understanding), raise sampling rates — but be mindful of bandwidth and model processing constraints. • If you want the model to act on visual changes immediately, consider manual turn control or event-driven logic, since video input is passive for turn detection. Would you like a concrete browser-to-server code snippet showing how to publish a screen track and the corresponding AgentSession startup flow?

How would it decide which track to screen share track to subscribe ?

You're asking how Gemini Realtime (via LiveKit) chooses which video track to use when multiple are published — it uses the single most recently published video track. Key details and guidance: • Selection rule: LiveKit sends frames from the most recently published video track (camera or screenshare) when

video_input=True

. • Behavior: It does not subscribe to multiple video tracks simultaneously — only the latest published track is sampled. • Sampling/encoding: Default ~1 FPS while speaking, ~1 every 3s when silent; frames are resized to fit 1024×1024 and JPEG-encoded. • To force a specific source (e.g., always use screenshare): publish the screenshare track after other video tracks or republish it when switching to make it the most recent. • To change sampling rates, provide a custom

VideoSampler

to your

AgentSession

. Would you like a code snippet showing how to republish a screenshare track so it becomes the active input? Sources: Vision | LiveKit docs

In realtime models, there are not going to be any tts node data hook right ?

You're asking whether realtime models have a

tts_node

data hook — short answer: no. Details and what to use instead: • Realtime models do not expose a

tts_node

because TTS is handled internally by the realtime model (speech is generated natively, not via a separate TTS pipeline). • Use

realtime_audio_output_node

to intercept or modify audio output from the realtime model. This node receives the audio frames the model produces and lets you process/transform them before they are sent to the user. Example (Python sketch):

async def realtime_audio_output_node(self, audio: AsyncIterable[rtc.AudioFrame], model_settings: ModelSettings) -> AsyncIterable[rtc.AudioFrame]:
    async for frame in Agent.default.realtime_audio_output_node(self, audio, model_settings):
        # modify frame (volume, effects, logging, etc.)
        yield frame

Would you like a concrete example that applies a gain change or records the model's audio output to storage? Sources: Pipeline nodes and hooks | LiveKit docs | Realtime model plugins | LiveKit Docs