I am planning to build a voice agent that will currently use OpenAI Realtime Voice (this may change depending on feasibility). The system will function as a voice chatbot with the following capabilities: Features: The agent should behave like a slide presenter, guiding the conversation in a structured way similar to presenting slides during a demo. It should use a RAG-based knowledge base to retrieve relevant information when needed. If the user interrupts or asks questions, the agent should respond appropriately using the knowledge base. When relevant, the agent should be able to pull up supporting materials such as images or PDFs and walk the user through them step-by-step, similar to a guided demo. The interaction style should resemble a pre-discovery sales call, where the agent introduces information, explains key points, and answers questions naturally. 1. After answering questions or explaining supporting materials, the agent should continue the structured flow of the presentation. The goal is to create a conversational voice agent that can present information, answer questions, and guide users through visual materials during the conversation. Any example project available.

Thanks for reaching out, Samuel. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

I use livekit cloud. Python SDK for the backend and JSSDK for the frontend. Web (react).

Great — thanks for the details. Since you’re using LiveKit Cloud with a Python backend and a React (web) frontend, here’s a concise, practical scaffold and recommended next steps you can use to build the voice slide-presenter agent with OpenAI Realtime Voice, RAG, interruption handling, and synced slide/PDF/image presentation. What I’ll cover: • High-level architecture and components • Recommended example projects to fork • Concrete implementation sketch (Python backend + React frontend) • Key code snippets and flow for: Realtime voice, RAG lookup, interruption handling, and sending slides to the frontend • Next steps I can provide if you want detailed code High-level architecture • LiveKit Cloud room hosts the session and audio/video/data tracks. • Backend (Python) runs the agent logic using LiveKit Agents + the OpenAI Realtime model for voice I/O and turn detection. • RAG: perform retrieval in the backend (e.g., LlamaIndex) on user turns and inject retrieved content into the assistant’s reply generation. • Frontend (React + LiveKit JS SDK) connects to the room, plays/records audio, and listens on a data track (or room metadata) to receive commands to show images/PDFs and page changes. • Presentation flow state (current slide, step) is maintained in the backend agent; after handling interruptions the agent resumes to the next slide or step. Example projects to use as building blocks • examples/voice_agents/llamaindex-rag — RAG integration with a voice agent (adapt retrieval logic). Fork this for the RAG pattern. • examples/survey and examples/bank-ivr — structured, stateful conversation flow examples to model slide sequencing and resumption logic. • Frontend React examples using @livekit/components-react — use these for connecting, publishing/subscribing, and receiving data messages. Concrete implementation sketch 1) Agent setup (Python backend) • Use the RealtimeModel configured for audio and text modalities and semantic VAD/turn detection so interruptions are handled cleanly. • Use an on_user_turn_completed hook to run RAG retrieval and to decide whether to: ◦ answer immediately (inject retrieved content into the reply), or ◦ show a slide / PDF (send data message to frontend), then narrate and walk through it. Minimal conceptual flow: 1. User speaks -> audio arrives in LiveKit -> RealtimeModel converts to partial text and triggers turn detection. 2. on_user_turn_completed receives final user text: ◦ Run RAG lookup with the user text + current slide context. ◦ Determine whether to show a slide or answer verbally. ◦ If showing visual material, send a data message: { type: "show_slide", url: "...", page: n } ◦ Generate assistant reply including retrieved facts and instructions to the frontend (e.g., highlight area, step index). 1. After finishing the answer, advance presentation state and continue with the next slide unless the user interrupts. 2) RAG integration • Place RAG lookup in on_user_turn_completed so retrieval happens after user finishes speaking but before generating the assistant reply. • Example: call your LlamaIndex retriever, get top-k contexts, and append them to the prompt for the RealtimeModel reply. • Save short-term context in the session (current slide, last retrieved docs) for continuity. 3) Interruption handling & resuming flow • Use semantic VAD/turn detection (configured on the RealtimeModel) so user interruptions are detected fast and reliably. • When an interruption occurs: ◦ Immediately capture the partial user message or new turn. ◦ If the agent was mid-speech, cancel or pause TTS generation (Realtime API supports low-latency turn switching). ◦ Answer the interruption using RAG if needed. ◦ After answering, use your stored presentation state to call a resume function that continues from the last slide/step. 4) Slide/PDF/image sync to React frontend • Use LiveKit DataChannel (sendData) or a dedicated data track to send JSON commands from the backend agent to all connected clients. • Example command payloads: ◦ { "type": "show_slide", "url": "", "slide": 1 } ◦ { "type": "show_pdf", "url": "https://cdn.example.com/doc.pdf", "page": 3 } ◦ { "type": "highlight", "coords": {x,y,w,h}, "slide": 2 } ◦ { "type": "resume_presentation", "next_slide": 4 } • Frontend listens for data messages and updates the viewer and UI accordingly (render PDF via pdf.js or <embed>, show images with an overlay for highlights). Also reflect the agent’s speech state (speaking/listening). Key code snippets (conceptual) Python: RealtimeModel config (conceptual)

session = AgentSession(
    llm=openai.realtime.RealtimeModel(
        voice="alloy",
        modalities=["text","audio"],
        turn_detection=TurnDetection(type="semantic_vad", eagerness="medium"),
    ),
    vad=some_vad_if_needed,
)

Python: RAG lookup in on_user_turn_completed (conceptual)

async def on_user_turn_completed(self, turn_ctx, new_message):
    user_text = new_message.text_content()
    retrieved = await my_llamaindex_retriever.search(user_text, top_k=3)
    turn_ctx.add_message(role="system", content=f"RAG_CONTEXT:{retrieved}")
    # optionally instruct client to show slide
    if should_show_slide(user_text):
        await turn_ctx.session.send_data({"type":"show_slide", "url": slide_url, "slide": slide_idx})
    # generate reply (agent will synthesize audio via RealtimeModel)
    await turn_ctx.generate_reply(instructions="Answer and then continue presentation")

React frontend: listen for data messages (conceptual)

room.on('dataReceived', (payload, participant) => {
  const msg = JSON.parse(new TextDecoder().decode(payload));
  switch(msg.type) {
    case 'show_slide':
      setSlideUrl(msg.url);
      setSlideIndex(msg.slide);
      break;
    case 'show_pdf':
      setPdfUrl(msg.url);
      setPdfPage(msg.page);
      break;
    case 'highlight':
      highlightOnCanvas(msg.coords);
      break;
  }
});

Practical tips and considerations • Keep the slide assets accessible via HTTPS (CDN) and small enough for quick loading. Preload next slide in the frontend when possible. • Use shorter retrieved contexts for RAG to keep latency low; fetch top-3 documents and summarize before injecting into prompts. • Test interruption flows thoroughly: mid-speech interruptions, quick follow-ups, and long questions during a slide walk-through. • Maintain a simple presentation state machine in the backend (current_slide, paused_for_question, last_user_turn_id) to make resumption deterministic. Next actions I can provide for you (pick one or more) • A starter Python project scaffold (LiveKit agent + OpenAI Realtime config + LlamaIndex RAG wiring) tailored to LiveKit Cloud. • A React example that subscribes to data messages and renders images/PDFs with page sync and highlight support. • A combined minimal demo repo that ties backend + frontend together (code snippets for auth tokens, send_data usage, and a basic state machine). • Detailed code for interruption handling and resume logic showing Realtime turn switching and session state usage. Which of these would you like next? I can begin by generating the Python scaffold (agent + RAG hook) plus the React data listener and slide viewer if you want a full starter demo.

Any example project available for this use case?

Thanks — great question. You’re looking for example projects you can use as direct references for a LiveKit Cloud + Python backend + React frontend voice agent that uses OpenAI Realtime, RAG, interruption handling, and synchronized visual materials. While there’s no single example that implements every piece end-to-end, there are several concrete LiveKit examples and guides that map directly to each capability you need. You can combine these building blocks into the slide-presenter agent. Below are the most relevant example projects and what each one shows that you can reuse. 1) Anam avatar (most directly relevant for syncing visuals) • What it shows: A voice AI assistant with a talking avatar, plus LiveKit RPC calls that update frontend state in real time as the agent speaks. This demonstrates how to trigger UI updates (forms, visuals) from agent logic — the same pattern you can use to show slides, PDFs, or images and step through them. • Use for: Real-time RPC/data messages to sync visuals with voice and agent-driven UI changes. • Link: Build a healthcare intake assistant with an Anam avatar (LiveKit blog) https://livekit.com/blog/build-healthcare-intake-assistant-anam-avatar 2) OpenAI Realtime plugin guide • What it shows: How to configure and use OpenAI Realtime within LiveKit Agents for low-latency audio/text modalities and turn detection (semantic VAD). This is the core of the low-latency voice I/O and interruption handling you need. • Use for: RealtimeModel setup, TTS/voice selection, and semantic turn detection configuration. • Link: OpenAI Realtime API plugin guide https://docs.livekit.io/agents/models/realtime/plugins/openai 3) RAG / external data pattern • What it shows: The lifecycle hook

on_user_turn_completed

intended for retrieval-augmented generation. Use this to run your LlamaIndex (or other retriever) lookups after each user turn and inject retrieved context into the assistant response. • Use for: Placing RAG lookups at the right point in the conversation and injecting retrieved text into replies. • Link: External data and RAG (Agents logic) https://docs.livekit.io/agents/logic/external-data/ 4) Examples for structured, stateful flows (presentation sequencing) • What they show:

examples/survey

and

examples/bank-ivr

illustrate stateful, step-by-step conversational flows and how to progress through steps/tasks deterministically. • Use for: Modeling slide sequencing, pause/resume logic, and deterministic state machines for your presentation flow. • Link: Examples overview (deepwiki) https://deepwiki.com/livekit/livekit_composite/1.5-examples 5) React starter + voice UI examples • What they show: Starter React agent frontends (e.g.,

agent-starter-react

, agent widgets) illustrating how to connect to LiveKit rooms, publish audio, receive audio, show transcriptions, and listen for data messages. • Use for: Implementing the slide/PDF viewer that listens to data messages and updates view/page/highlights. • Link: Examples and starter templates (examples list) https://deepwiki.com/livekit/livekit_composite/1.5-examples 6) Kitt / LemonSlice (additional patterns) • What they show: Voice-driven meeting assistants and avatar-driven agents that sync visuals/avatars with agent states. Useful patterns for state-to-visual mapping. • Use for: Inspiration for styling, state mapping, and frontend asset switching. • Links: ◦ Meet Kitt (LiveKit blog) — voice agents in meetings https://livekit.com/blog/meet-kitt • LemonSlice avatar example — avatar + visuals tied to state https://livekit.com/blog/build-salary-negotiation-coach-lemonslice-avatar How to combine these into your slide-presenter agent (high level) • Backend (Python): ◦ Use the RealtimeModel for audio+text and set turn_detection to

semantic_vad

for interruption handling. ◦ Implement

on_user_turn_completed

to run RAG (LlamaIndex or your retriever), decide whether to show a slide, and inject retrieved context into the assistant reply. ◦ Use session.send_data or RPC to push JSON commands (show_slide, show_pdf, highlight, resume_presentation) to the frontend. ◦ Maintain a simple presentation state machine: current_slide, paused_for_question, last_user_turn_id. • Frontend (React): ◦ Connect via LiveKit JS SDK and listen for data messages (

room.on('dataReceived', ...)

) to receive show/resume/highlight commands. ◦ Render images or PDF pages (pdf.js or <embed>), preload the next slide, and support highlight overlays. ◦ Update UI state (current slide, agent speaking/paused) based on messages from the backend. • Test interruptions: ◦ Verify that mid-speech interruptions are detected, the backend switches turns, performs RAG, and either answers then resumes or directly resumes presentation flow. If you’d like, I can do one of these next: • Provide a focused starter repo scaffold (Python agent + RAG hook + minimal React viewer) that ties the Anam-style RPC pattern to slide commands. • Produce a short walkthrough showing the exact backend send_data payloads and the React handler code to render images/PDFs and resume the flow. • Share a compact checklist and minimal code snippets to wire RealtimeModel + on_user_turn_completed + LlamaIndex retrieval. Which of those would be most helpful to you now? Sources: Build a healthcare intake assistant with an Anam avatar | LiveKit | OpenAI Realtime API plugin guide | LiveKit Documentation | Pipeline nodes and hooks | LiveKit Documentation | Examples | livekit/livekit_composite | DeepWiki | Live conversations with AI using ChatGPT and WebRTC | LiveKit