Is it possible to extend Livekit’s message data models to support other message formats besides text and images? I would like to allow users to upload/attach PDFs and pass those through to the underlying agent model directly.

Thanks for reaching out, Cecchi. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

To clarify I am not talking about the user <> agent communication, I am talking about passing PDFs to the LLM as an

input_file

message. https://developers.openai.com/api/docs/guides/file-inputs/

Thanks — that clarification helps. You’re asking whether LiveKit’s message models can be extended so a PDF can be passed directly to an LLM as an OpenAI-style

input_file

. Short answer: LiveKit does not natively produce OpenAI

input_file

objects, but you can absolutely build a flow that delivers PDFs to the LLM in the same way OpenAI expects. Below is a clear, practical approach you can implement. High-level constraint • LiveKit can transport PDF bytes between participants (via data channels / byte streams), but it does not directly call or create an OpenAI

input_file

for you. You need to make the PDF accessible to the OpenAI API (either by uploading the file to OpenAI’s file endpoint or by providing a storage URL OpenAI can access) and then reference that file in your LLM request. Recommended patterns (two common options) • Option A — Upload to OpenAI file endpoint (preferred if you want OpenAI to manage the file): a. Receive the PDF in your client or agent via LiveKit byte streams (or have the client upload directly). b. Forward the PDF bytes to your backend. c. On your backend, call OpenAI’s file upload endpoint to create an

input_file

resource (the OpenAI API accepts file uploads and returns an ID). d. Use the returned file ID in subsequent model calls per OpenAI’s

input_file

usage. ◦ Pros: OpenAI stores the file and you get an

id

that the model API accepts directly. ◦ Cons: Involves sending file bytes to OpenAI and managing that lifecycle (deletes/retention). • Option B — Host the PDF in your storage and pass a reference (when supported): a. Receive the PDF via LiveKit and store it in your object storage (S3, GCS, etc.) with proper access controls. b. If the LLM accepts URLs or your own ingestion pipeline, either: • Provide the secure URL to your ingestion pipeline that downloads and passes it to the LLM, or • Have your backend download and upload to OpenAI’s file endpoint (same as Option A) and then pass the file id. ◦ Pros: You keep control of storage, privacy, and auditing. ◦ Cons: Extra step to convert storage reference into an

input_file

if the LLM requires that form. Concrete flow using LiveKit + OpenAI file upload 1. User selects PDF in the client UI. 2. Client sends PDF bytes via LiveKit byte streams to your backend or to an agent participant. Alternatively, upload directly to your backend or S3 from the client. 3. Backend receives bytes, validates size/MIME, and then: ◦ Uploads to OpenAI’s file endpoint to get a file id, or ◦ Stores the file and (optionally) downloads it to forward to OpenAI. 1. Make your LLM call referencing the OpenAI

input_file

(file id) according to OpenAI’s API. 2. Handle LLM response and surface results back to the room/agent as needed. Practical considerations and tips • Chunking & reliability: For large PDFs, use byte-stream chunking on the LiveKit side and reassemble reliably on the backend before upload. • Metadata: Attach MIME type, filename, and a stable ID when sending via LiveKit so your backend can set correct metadata for the OpenAI upload. • Security & privacy: Sanitize and enforce access controls. If you pass files to OpenAI, be mindful of data residency and retention. • Latency: Uploading the file to OpenAI adds latency—consider whether you need streaming extracts (client-side parsing) vs. full-file upload. • Size limits: Check OpenAI file-size limits and your own transport limits; implement fallbacks (e.g., split documents, extract pages). If you want, I can: • Provide a concise JS example showing how to send a PDF via LiveKit byte streams and post it to your backend. • Provide a backend snippet (Node/Python) that uploads the received PDF to OpenAI’s file endpoint and returns the file id for model calls. Which example would you like first — client-side send + backend receive, or a backend-only example that takes an uploaded file and creates the OpenAI

input_file

?

We already have the file storage piece, so we have a URL for the documents. How do we pass those as a message to the LLM as part of the

ChatContext

?

Thanks — that’s a helpful clarification. I’ll address exactly how to include your stored PDF URLs in the ChatContext so the LLM can use them, and note the important limitation you must handle. Short answer • Add the document URL as plain text inside a ChatMessage in the ChatContext. The ChatContext is just a sequence of messages, so include the URL (plus an instruction) in the message content. • Important: most LLM providers do not automatically fetch or open arbitrary URLs. If the model cannot fetch the PDF, you must retrieve the PDF yourself (server-side), extract or summarize the text, and include that text in the ChatContext or upload the file to the provider as an input_file first. How to do it (example pattern) 1. Create a ChatContext and add a ChatMessage whose content includes the document URL and a clear instruction:

from livekit.agents.llm import ChatContext, ChatMessage

pdf_url = "<https://your-storage.example.com/docs/doc123.pdf>"

chat_ctx = ChatContext()
chat_ctx.add_message(
    role="user",
    content=f"Please review this document and summarize the key points: {pdf_url}"
)

1. Use that ChatContext when calling your agent/session so the LLM receives the message with the URL. When this works vs when you must fetch the file yourself • Works as-is only if the LLM/provider (or a tool/plugin you’ve enabled) can fetch and read URLs directly. • If the provider cannot fetch the URL, do one of these: ◦ Server-side fetch + include extracted text: Download the PDF on your backend, extract text (or a concise summary), then include that extracted text in the ChatContext instead of — or in addition to — the URL. ◦ Upload to the provider as an input_file: If your provider supports file uploads (e.g., OpenAI file endpoint), upload the PDF and reference the returned file id in the model call per the provider’s API. Practical recommendations • Always include an instruction describing how the LLM should use the URL (e.g., “summarize”, “extract the FAQ section”, “answer questions using the document”). • If the PDF is large, prefer server-side extraction or upload to the provider and supply a condensed excerpt to keep prompt size reasonable. • Provide metadata along with the URL in the message (filename, MIME type, page range to read) so the downstream processor knows what to fetch/process. • For sensitive documents, ensure secure access (signed short-lived URLs) and follow your data-handling policies before sending to an external LLM provider. If you’d like, I can: • Show a short backend example (Node or Python) that fetches a PDF from your storage, extracts text, and constructs the ChatContext with the extracted text. • Or, show how to upload the PDF to OpenAI’s file endpoint and then call the model referencing that file id. Which example would help you move forward? Sources: External data and RAG | LiveKit Documentation | Google Gemini LLM | LiveKit Documentation | OpenAI LLM models | LiveKit Documentation | Pipeline nodes and hooks | LiveKit Documentation

To be clear, we want to have the underlying call that Livekit makes to OpenAI use a structure like:

{
  "model": "gpt-5",
  "input": [
    {
      "role": "user",
      "content": [
        {
          "type": "input_text",
          "text": "Analyze the letter and provide a summary of the key points."
        },
        {
          "type": "input_file",
          "file_url": "<https://www.berkshirehathaway.com/letters/2024ltr.pdf>"
        }
      ]
    }
  ]
}

I do not believe simply including the URL in the text message is reliable. We have looked into managing our own extraction pipeline, but it adds significant latency vs passing the file as an

input_file

to OpenAI

Thanks — that clarification is very helpful. I’ll respond to the exact ask: you want LiveKit’s underlying call to OpenAI to include an

input_file

structured object (not just a URL in text), so OpenAI can ingest the PDF natively and avoid your own extraction pipeline. Short answer • LiveKit does not automatically construct OpenAI

input_file

objects for you inside the Chat payload. You can, however, implement an adapter in your backend that uploads the PDF to OpenAI’s file endpoint (or makes it otherwise available to OpenAI), receives the file id, and then issues the OpenAI model call using the

input_file

style structure you showed. That is the recommended approach to avoid doing full extraction yourself while keeping latency lower than client-side extraction. Why and how (concise) • OpenAI expects file inputs to be provided via its file upload endpoint (or an equivalent provider-specific mechanism). Simply embedding a URL in the message body is unreliable because most models do not fetch arbitrary URLs automatically. • The practical flow is: a. Have your backend fetch the PDF from your storage URL (or accept it directly from the client). b. Upload the PDF bytes to OpenAI’s file endpoint; OpenAI returns a file id. c. Construct the model call using that file id (or the provider’s required

input_file

structure) in the request payload. d. Call the model; handle the response and relay results back into the LiveKit session. Minimal example flows you can implement • Option A — Upload-to-OpenAI then call model (preferred) a. Backend downloads PDF from your storage URL. b. Backend POSTs file bytes to OpenAI file upload endpoint; receive

file_id

. c. Backend calls the model with the

input_file

in the

input

array (or provider-specific placement) referencing

file_id

. d. Return results to LiveKit/agent session. • Option B — If your LLM provider supports remote-file references inside the request body (rare) ◦ Ensure the provider can fetch the URL directly (public or signed URL) and then include

file_url

in the content array as in your example. Confirm provider support before relying on this. Practical notes and recommendations • Latency: uploading a file to OpenAI does add overhead, but you avoid heavier extraction pipelines on your side. You can mitigate latency by: ◦ Uploading files ahead of time (when documents are added to storage) and reuse the returned file id for future model calls. ◦ Uploading asynchronously and notifying the requester when analysis is ready. • Access: If you use signed URLs, ensure your backend can retrieve the file for upload (signed URL lifetime should allow backend fetch). • Size and limits: Verify file size limits and content policies with the provider; for very large PDFs consider splitting or preprocessing. • Security: Be mindful of sending sensitive files to third-party providers. Use encryption, minimal retention, and follow your compliance requirements. If you’d like, I can provide one of the following next steps: • A short backend code example (Node or Python) that downloads a PDF from your storage URL, uploads it to OpenAI’s file endpoint, and then issues a model call referencing that file. • Or, if you want to keep uploads out of the request path, a pattern for pre-ingesting files (upload at document ingestion time and store the returned file id) so runtime calls are fast. Which example would help you implement this fastest — Node or Python?