Hello

Thanks for reaching out, use. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

How to Send Attachments to Backend for Gemini Analysis via RPC? i have a react app (used the starter react) Attachment Storage:

// Storing files in this array
const [attachments, setAttachments] = useState<File[]>([]);

RPC Method to Send Attachments:

async function sendAttachmentsWithMessage() {
  // Convert files to Base64 for transmission
  const attachmentData = await Promise.all(
    attachments.map(async (file) => {
      return new Promise((resolve) => {
        const reader = new FileReader();
        reader.onload = (e) => {
          resolve({
            name: file.name,
            type: file.type,
            size: file.size,
            data: e.target?.result, // Base64 encoded file data
            lastModified: file.lastModified
          });
        };
        reader.readAsDataURL(file);
      });
    })
  );

  const payload = {
    message: message,
    attachments: attachmentData,
    timestamp: new Date().toISOString(),
    requestId: `req_${Date.now()}`
  };

  // Send via RPC to backend
  const response = await room.localParticipant.performRpc({
    destinationIdentity: '',
    method: 'analyzeAttachments',
    payload: JSON.stringify(payload),
    responseTimeout: 10000
  });
}

Backend Questions: 1. RPC Handler Setup: How should I structure the RPC handler on the backend to receive this

analyzeAttachments

call? 2. File Processing: What's the best way to Pass these files to Gemini's vision/document analysis capabilities? backend tools.py

@function_tool()
async def analyze_attachments(context: RunContext, ...):
    # How do I structure this to:
    # 1. Receive RPC call with attachments
    # 2. Process Base64 data back to files  
    # 3. Send files to Gemini for analysis
    # 4. Return structured response
    pass

Let's walk through setting up an RPC handler together so you can receive attachments and process them for Gemini analysis. Here’s a step-by-step guide to help you along the way: 1. Receive Attachments Using Byte Streams:

room.registerByteStreamHandler('files', async (reader, participant) => {
  const info = <http://reader.info|reader.info>;
  console.log(`Started receiving file "${info.name}" from ${participant.identity}`);

  // Optional: track progress
  reader.onProgress = (progress) => {
    console.log(`Progress: ${(progress * 100).toFixed(0)}%`);
  };

  // Read all chunks
  const chunks = [];
  for await (const chunk of reader) {
    chunks.push(chunk);
  }

  const fileBlob = new Blob(chunks, { type: info.mimeType });
  const fileName = info.name;

  // Process the file (e.g., send to Gemini)
  await processFileForGemini(fileBlob, fileName);
});

2. Process File for Gemini Analysis: Next, you’ll want to convert the file to Base64 and use LiveKit Agents to integrate with Gemini. Here’s how you can do that:

import { ImageContent } from 'livekit-agents';

const base64Data = await blobToBase64(fileBlob);
const imageContent = new ImageContent({
  mimeType: fileBlob.type,
  base64Data,
  description: "User-uploaded file",
});

chatContext.addMessage({ role: 'user', content: [imageContent] });

3. Full Integration with LiveKit Agents: To complete the process, use RPC to signal when a file arrives and trigger the analysis like this:

const response = await room.localParticipant.performRpc({
  destinationIdentity: 'agent-identity',
  method: 'analyze-file',
  payload: JSON.stringify({
    fileName: info.name,
    mimeType: info.mimeType,
    size: info.size,
  }),
});

On the backend side, you can process the file and integrate with Gemini as follows:

@agent.on("rpc")
async def on_rpc(ctx: JobContext, data: RpcInvocation):
    if data.method == "analyze-file":
        payload = json.loads(data.payload)
        image_content = llm.ImageContent.create(
            url=payload["file_url"],  # or base64_data=payload["base64"]
            mime_type=payload["mimeType"]
        )
        ctx.chat_ctx.add_message(
            llm.ChatMessage(role="user", content=[image_content])
        )
        return "Processing started"

If you'd like, I can also provide more details on how to upload files to cloud storage before analysis. Just let me know—I'm here to help! Sources: Google Gemini integration guide | LiveKit Docs | Sending files & bytes | LiveKit Docs | README.md | agents/src/llm/provider_format/google.ts | livekit-plugins/livekit-plugins-google/livekit/plugins/google/beta/realtime...