does livekit node sdk support realtime video apis like gemini live?

Thanks for reaching out, Tim. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

I'm using inference, can you give a code sample on how that plugin would be set up?

Thanks — I’m replying about your request for a code sample showing how to set up the Gemini Live API plugin when using LiveKit with Node.js. One important point up front: Gemini Live (the realtime model with native audio/video) is not available through LiveKit Inference. You must use the Google plugin (livekit-agents-plugin-google) and provide direct Google API credentials. Below is a concise example and notes to get you started. 1) Install the plugin

pnpm add "@livekit/agents-plugin-google@1.x"

2) Minimal Node.js example using the Gemini Live plugin with an AgentSession

// server.ts (Node.js / TypeScript)
import * as google from "@livekit/agents-plugin-google";
import { voice } from "@livekit/agents";

// Example agent implementation (replace with your agent logic)
class MyAgent extends voice.Agent {
  async onMessage(msg: voice.AgentMessage) {
    // handle incoming messages/events from the realtime model
  }
}

async function startAgentSession(room, ctx) {
  const session = new voice.AgentSession({
    llm: new google.realtime.RealtimeModel({
      // Gemini Live realtime model id (example preview id)
      model: "gemini-2.5-flash-native-audio-preview-12-2025",
      // Name of the voice to synthesize from the model
      voice: "Puck",
      // Optional tuning
      temperature: 0.7,
      instructions: "You are a helpful assistant.",
      // Google API key (required for plugin-based realtime usage)
      api_key: process.env.GOOGLE_API_KEY!,
    }),
  });

  // Start the session and enable video input so the model receives live frames
  await session.start(room, new MyAgent(), {
    room_options: {
      // allow the session to receive video frames from participants
      video_input: true,
      // you can also configure audio input options here if needed
    },
  });

  return session;
}

3) Key notes and requirements • Gemini Live functionality (realtime audio + video) is not accessible through LiveKit Inference. Use the plugin and direct Google API credentials (

GOOGLE_API_KEY

). • Set

room_options.video_input: true

when starting the session to enable live video frames from camera or screen share. • The plugin does low-latency bidirectional streaming (audio/video frames in, audio + events out). Make sure your client bandwidth and codec settings match your needs. • Replace the preview model id above with the exact model id you have access to; some model names or preview IDs may change over time. • You can customize TTS voice, temperature, and other realtime options in the

RealtimeModel

constructor. If you’d like, I can: • Provide a small client-side example showing how to publish camera/screen to the LiveKit room so the agent receives frames. • Show how to wire up event handlers to capture the model’s responses (audio chunks, transcripts, or custom events). • Help confirm which Gemini realtime model ID you should use based on your access. Which follow-up would be most useful for you? Sources: Google Gemini LLM | LiveKit Documentation | Gemini Live API plugin | LiveKit Documentation

I'm seeing this: Property 'realtime' does not exist on type 'typeof import("/Users/tim.kao/monterey-ai-monorepo/node_modules/@livekit/agents-plugin-google/dist/index")'.

Thanks — that error means TypeScript can’t find the

realtime

export on the Google plugin you imported. I’ll walk through the most likely causes and quick fixes, and then ask for a couple details if it still fails. What I’m addressing: the TypeScript error Property 'realtime' does not exist on type 'typeof import(...)' when importing @livekit/agents-plugin-google. Possible causes and fixes 1) Wrong or incompatible package version • Ensure you have the plugin v1.x installed:

pnpm add "@livekit/agents-plugin-google@1.x"

• Run

pnpm list @livekit/agents-plugin-google

to confirm the installed version. 2) TypeScript/IDE caching • Restart the TypeScript server in your editor (VS Code: “TypeScript: Restart TS Server”). • Remove node_modules and reinstall:

rm -rf node_modules pnpm install

. 3) Module entrypoint / build artifacts missing • Check that

node_modules/@livekit/agents-plugin-google/dist/index.js

contains an export for

realtime

, for example:

export * as realtime from './realtime';

• Confirm

node_modules/@livekit/agents-plugin-google/dist/realtime/index.js

exists. 4) Temporary workaround: import the realtime type directly • If the namespaced export is not visible to TypeScript for some reason, import the RealtimeModel directly from the built path:

import { RealtimeModel } from "@livekit/agents-plugin-google/dist/realtime";

  const session = new voice.AgentSession({
    llm: new RealtimeModel({ /* ... */ }),
  });

• This is a workaround—not preferred long-term—but it helps unblock you while we diagnose exports/types. 5) Type definitions mismatch • If the package's

dist/index.d.ts

doesn’t declare

realtime

, TypeScript will complain even if the JS export exists. Reinstalling the correct package version usually fixes this. If these steps don’t resolve it, please share the following so I can diagnose further: • The output of

pnpm list @livekit/agents-plugin-google

• Your import lines and the small snippet where you construct the AgentSession • TypeScript version (from package.json or

tsc -v

) • Node.js version (if different from typical 18+) I can then suggest the exact change (or an upstream issue workaround) and provide a corrected minimal example. Which details can you share? Sources: Gemini Live API plugin | LiveKit Documentation

i have 1.0.41 installed

Thanks — that helps. Since you have @livekit/agents-plugin-google@1.0.41 installed, that version should provide the

realtime

export. Let’s go through a short checklist and a couple of concrete fixes to get past the TypeScript error. What I’m addressing: you saw Property 'realtime' does not exist on type 'typeof import(...)' even though you have version 1.0.41. Quick checklist & fixes 1) Confirm your import matches the documented pattern • Use the namespace import:

import * as google from "@livekit/agents-plugin-google";
import * as voice from "@livekit/agents";

• Then:

const session = new voice.AgentSession({
  llm: new google.realtime.RealtimeModel({ /* ... */ }),
});

2) Restart TypeScript / reinstall node_modules • Restart the TS server in your editor (VS Code: “TypeScript: Restart TS Server”). • Remove and reinstall modules to refresh types:

rm -rf node_modules pnpm install

3) Verify the package actually contains the realtime export • Inspect these files: ◦ node_modules/@livekit/agents-plugin-google/dist/index.js ◦ node_modules/@livekit/agents-plugin-google/dist/index.d.ts • index.js should export the realtime namespace (something like

export * as realtime from './realtime'

). • index.d.ts should declare

export * as realtime from './realtime'

or equivalent types. 4) Workaround: import the RealtimeModel directly • If TypeScript still can’t see the namespace, you can import the implementation directly from the built path as a temporary workaround:

import { RealtimeModel } from "@livekit/agents-plugin-google/dist/realtime";
import * as voice from "@livekit/agents";

const session = new voice.AgentSession({
  llm: new RealtimeModel({ /* ... */ }),
});

• This bypasses the namespace typing issue while we find the root cause. 5) Check TypeScript config & version • TypeScript can sometimes fail to resolve certain export styles depending on: ◦

tsconfig.json

module settings (

moduleResolution

,

esModuleInterop

,

module

,

target

) ◦ TypeScript version • If you can, tell me: ◦ your

tsconfig.json

compilerOptions (module, moduleResolution, esModuleInterop) ◦

tsc -v

(TypeScript version) ◦ Node.js version If the above steps don’t fix it, please share these minimal items and I’ll diagnose further: • The exact import lines and the AgentSession setup snippet (2–6 lines) •

pnpm list @livekit/agents-plugin-google

output •

tsconfig.json

compilerOptions • TypeScript version (

tsc -v

) I can then propose the exact fix (tsconfig tweak, recommended import, or a patch/workaround). Which of those details can you share?

got it to work. Now how do I feed it the video stream?

Great — glad you got the plugin working. Below I’ll show exactly how to feed a live video stream (camera or screen share) into an AgentSession so the Gemini Live realtime model receives frames. What I’m addressing: enabling video input on the server AgentSession and publishing a video track from the client so the agent can consume it. Server (Node.js) — enable video input when starting the session 1. Ensure you enable video input in the RoomOptions when calling

session.start

. The agent will then receive frames from the room’s published video track(s).

import * as google from "@livekit/agents-plugin-google";
import * as voice from "@livekit/agents";
import { room_io } from "@livekit/agents"; // if your environment exposes room_io

const session = new voice.AgentSession({
  llm: new google.realtime.RealtimeModel({
    model: "gemini-2.5-flash-native-audio-preview-12-2025",
    voice: "Puck",
    instructions: "You are a helpful assistant with vision.",
    api_key: process.env.GOOGLE_API_KEY!,
  }),
});

// Start session and enable video input
await session.start({
  room: ctx.room, // your JobContext room or LiveKit room object
  roomOptions: new room_io.RoomOptions({
    videoInput: true, // IMPORTANT — enables live video frames for the model
  }),
});

Notes: • The agent automatically receives frames from the most recently published video track (camera or screen). • Frame sampling is handled for you: typically the model samples frames at a low rate (e.g., while the model is speaking vs idle) — you don’t need to stream every frame manually. • Use the realtime plugin (not LiveKit Inference) for Gemini Live realtime audio/video. Client (Web) — publish camera or screen-share to the LiveKit room Your client must publish a video track to the LiveKit room. Example using the web SDK:

// Example with livekit-client (web)
import { connect, createLocalVideoTrack } from "livekit-client";

const room = await connect(url, token);

// Publish camera
const videoTrack = await createLocalVideoTrack({ 
  // deviceId, width, height, fps are optional
});
await room.localParticipant.publishTrack(videoTrack);

// Or publish screen share
const screenTrack = await createLocalVideoTrack({
  // Many apps use getDisplayMedia wrapper to capture screen
  // Or use a method that converts MediaStream to track for LiveKit
});
await room.localParticipant.publishTrack(screenTrack);

If you use LiveKit UI components or @livekit/components-react, simply enabling camera/screen-share on join will publish the track: •

localParticipant.setCameraEnabled(true)

or equivalent. Behavior & tips • The agent consumes whichever video track is published last (camera vs screen). If you switch to screen share, the agent will begin receiving those frames. • Bandwidth and fps: you don’t need extremely high framerates — the model expects sampled frames. But keep client encoding settings reasonable for network conditions. • If you need custom sampling or want to send higher frequency frames, mention that and I can show how to adjust publishing or server-side helpers. Want a full end-to-end example? I can provide: • A minimal server file showing the full AgentSession lifecycle and event handlers to capture model responses (audio, transcripts). • A small client example (React or vanilla) that joins the room, publishes camera/screen, and demonstrates switching tracks. Which would you like next — server full example, client example, or both? Sources: Vision | LiveKit Documentation