https://livekit.io logo
Join Slack
Powered by
can i give my agent files as pdf images and stuff?
# ask-ai
b
can i give my agent files as pdf images and stuff?
t
Thanks for reaching out, Ibrahem. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.
Yes, LiveKit agents can handle files such as PDFs, images, and other types. This is done using byte streams, which allow for the transfer of files and binary data between participants in real-time. For PDFs, agents can process them by converting them to text or images, depending on the configuration. Images can also be handled directly, and certain formats like SVG can be converted to PNG for compatibility. Keep in mind that there are file size limits (e.g., files larger than 500MB may be rejected) and specific MIME types are handled differently. If you have a particular use case or file type in mind, let me know, and I can provide more details. Sources: Realtime data | LiveKit Docs | Web and mobile frontends | LiveKit Docs | tools/server/webui/src/components/useChatExtraContext.tsx | Web and mobile frontends | LiveKit Docs
b
can u give me an example
t
Here’s a friendly example of how you can work with a PDF file using an agent in LiveKit: 1. Uploading a PDF: Someone in the session uploads a PDF file to the agent using byte streams. 2. Processing the PDF: The agent can pull out the text from the PDF or turn it into images to use in different ways. For example, if the PDF has a form, the agent can take a closer look at the text fields. 3. Sharing the Processed Data: After that, the agent can share the extracted text or images with other participants in the session or even help out with tasks like filling forms automatically or creating summaries. If you have a particular situation in mind—like sharing images or working with other types of files—just let me know! I’d be happy to give you a customized example.
b
no code example of how to implmenet that
t
Here’s a handy code example to help you handle PDF or image files with LiveKit agents using byte streams. Let’s walk through it together: Sending Files (Images or PDFs)
Copy code
const file = document.getElementById('file')?.files?.[0];
if (file) {
    const info = await room.localParticipant.sendFile(
        file,
        {
            mimeType: file.type,
            topic: 'files',
            onProgress: (progress) => console.log(`Sending file, progress: ${Math.ceil(progress * 100)}%`),
        }
    );
    console.log(`Sent file with stream ID: ${info.id}`);
}
Receiving Files
Copy code
room.registerByteStreamHandler('files', (reader, participant) => {
    const info = <http://reader.info|reader.info>;

    reader.onProgress = (progress) => {
        console.log(`Receiving file "${info.name}" from ${participant.identity}, progress: ${Math.round(progress * 100)}%`);
    };

    const chunks = [];
    for await (const chunk of reader) {
        chunks.push(chunk);
    }

    const fileBlob = new Blob(chunks, { type: info.mimeType });

    if (info.mimeType.startsWith('image/')) {
        const imageUrl = URL.createObjectURL(fileBlob);
        console.log('Received image:', imageUrl);
    } else if (info.mimeType === 'application/pdf') {
        const pdfUrl = URL.createObjectURL(fileBlob);
        console.log('Received PDF:', pdfUrl);
    }
});
This example shows you how to send and receive files smoothly, keeping track of progress and handling different file types with ease. If anything’s unclear or you want me to walk you through more examples, just let me know—I’m here to help!
b
is that all what i need to change in my agent py file? i am using flutter frontend and python agent and fastapi server backend
t
Hey there! To help your Python agent with a FastAPI backend handle file uploads and byte streams smoothly, here’s what you’ll want to do: 1. Register a Byte Stream Handler in Your Agent:
Copy code
class VisionAssistant(Agent):
    async def on_enter(self):
        def _image_received_handler(reader, participant_identity):
            task = asyncio.create_task(
                self._image_received(reader, participant_identity)
            )
            self._tasks.append(task)
            task.add_done_callback(lambda t: self._tasks.remove(t))
            
        get_job_context().room.register_byte_stream_handler("test", _image_received_handler)
2. Process Received Files/Byte Streams:
Copy code
async def _image_received(self, reader, participant_identity):
        <http://logger.info|logger.info>("Received image from %s: '%s'", participant_identity, reader.info.name)
        try:
            image_bytes = bytes()
            async for chunk in reader:
                image_bytes += chunk

            chat_ctx = self.chat_ctx.copy()
            chat_ctx.add_message(
                role="user",
                content=[
                    ImageContent(
                        image=f"data:image/png;base64,{base64.b64encode(image_bytes).decode('utf-8')}"
                    )
                ],
            )
            await self.update_chat_ctx(chat_ctx)
            print("Image received", self.chat_ctx.copy().to_dict(exclude_image=False))
3. Sending Files from the Frontend:
Copy code
# Use the LiveKit SDK's send_file method to send files
await room.localParticipant.send_file(
    file_path="path/to/image.jpg",
    topic="images",
    destination_identities=["agent"],  # Optional, if you want to send to specific participants
)
These tweaks will get your agent ready to handle file uploads and byte streams like a pro. If anything’s unclear or you want me to walk you through more examples, just give me a shout!
b
tell me in which file exactly i need to do changes
t
Handling file uploads and byte streams in your Python agent with FastAPI is totally doable! Here’s a friendly walkthrough on what you need to tweak: 1. Agent File (like 
agent.py
) This is where your agent class and its handlers live. You’ll want to: • Set up a byte stream handler to catch uploaded files or byte streams. • Process the data right inside that handler. Here’s a quick example:
Copy code
class VisionAssistant(Agent):
    async def on_enter(self):
        def _image_received_handler(reader, participant_identity):
            task = asyncio.create_task(
                self._image_received(reader, participant_identity)
            )
            self._tasks.append(task)
            task.add_done_callback(lambda t: self._tasks.remove(t))
            
        get_job_context().room.register_byte_stream_handler("test", _image_received_handler)

    async def _image_received(self, reader, participant_identity):
        image_bytes = bytes()
        async for chunk in reader:
            image_bytes += chunk

        chat_ctx = self.chat_ctx.copy()
        chat_ctx.add_message(
            role="user",
            content=[
                ImageContent(
                    image=f"data:image/png;base64,{base64.b64encode(image_bytes).decode('utf-8')}"
                )
            ],
        )
        await self.update_chat_ctx(chat_ctx)
2. FastAPI Route Handler File (like 
main.py
) Next up, create or update a FastAPI route that handles file uploads and sends them over to your LiveKit agent using byte streams. Something like this:
Copy code
from fastapi import FastAPI, File, UploadFile
from livekit import rtc

app = FastAPI()

@app.post("/upload-file")
async def upload_file(file: UploadFile = File(...)):
    # Connect to LiveKit room and get the local participant
    await room.local_participant.send_file(
        file.file,
        topic="test",
    )
    return {"message": "File uploaded successfully"}
If anything’s unclear or you’d like more examples, just give me a shout—I’m here to help! Sources: Vision | LiveKit Docs
b
what else would be needed
t
Along with updating the agent and FastAPI files, I wanted to share some extra tips and things to keep in mind when handling file uploads and byte streams in your Python agent with FastAPI: 1. File Uploads from Frontend: Make sure your Flutter frontend uses the LiveKit SDK’s 
sendFile
method to send files like images or PDFs to the agent. It’s a smooth way to transfer files without any hassle. 2. Streaming Bytes: For sending binary data bit by bit, the 
streamBytes
method is your friend. This works great for real-time stuff like audio or video frames. 3. Handling Incoming Byte Streams: Don’t forget to register a byte stream handler in your agent with 
register_byte_stream_handler
. This helps your agent process incoming streams either as they come in or once the whole stream is received. 4. Limitations and Considerations: • Chunk Sizes: The system smartly optimizes chunk sizes for reading and writing, so make sure your agent can handle data in chunks. • Concurrency: It’s important to manage multiple streams properly, especially if you have several participants involved. 5. Integration with Vision: If you’re working with vision tasks, double-check that the images you receive play nicely with your vision model. Process the image bytes accordingly to get the best results. 6. Additional Resources: You might find the Byte Streams Documentation and the Vision Guide super helpful for more details and tips. Feel free to reach out if you want me to walk you through anything or share some examples! Sources: Sending files &amp; bytes | LiveKit Docs | Vision | LiveKit Docs | Bringing AI avatars to voice agents
b
how to change chunk size
t
Hey there! Just a heads-up—the chunk size for byte streams in LiveKit isn’t something you can tweak through the API or SDK settings. The system takes care of it behind the scenes to keep things running smoothly. If you have particular needs around chunk size, you might want to handle some of the data processing on your side to work well with LiveKit’s default setup. Feel free to reach out if you’d like some tips on managing data streams or dealing with large files within these limits—I’m here to help!
b
i am using flutter for frontend
t
Here’s a friendly guide to help you handle file uploads and byte streams in LiveKit with a Flutter frontend: 1. Sending Files:
Copy code
var fileToSend = File('path/to/file.jpg');
var info = await room.localParticipant?.sendFile(
  fileToSend,
  options: SendFileOptions(
    topic: 'file-topic',
    onProgress: (progress) {
      // Keep track of progress here
      print('Progress: ${progress * 100}%');
    },
  ));
2. Streaming Bytes:
Copy code
var writer = await room.localParticipant?.streamBytes(
  StreamBytesOptions(topic: 'byte-stream-topic'));
await writer?.write(utf8.encode('Hello, world!'));
await writer?.close();
3. Receiving Files and Bytes:
Copy code
room.registerByteStreamHandler(
  'file-topic', 
  (ByteStreamReader reader, String participantIdentity) async {
    // Here’s where you process the incoming file chunks
    var file = await reader.readAll();
    // Then handle the file data (like saving it)
  });
Important Things to Keep in Mind: • You can use 
readAll()
to grab the whole content at once, or listen for chunks as they come in. • Both 
sendFile
and 
streamBytes
let you track progress with callbacks. • Remember to close streams when you’re done to keep everything running smoothly. If anything’s unclear or you want me to walk you through more examples, just let me know—I’m here to help!
8 Views
Open in Slack
PreviousNext
Powered by