500 Internal Server Error uploading session report to /observability/recordings/v0. When agent observability/recording is enabled, the SDK fails to upload the session report at the end of a call with a 500 Internal Server Error from the LiveKit Cloud endpoint. Why is this happening?

Thanks for reaching out, Matin. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

Error

aiohttp.client_exceptions.ClientResponseError: 500, message='Internal Server Error',
  url='https://<project>.livekit.cloud/observability/recordings/v0'

Full traceback:

File ".../livekit/agents/job.py", line 275, in _on_session_end
    await _upload_session_report(
File ".../livekit/agents/telemetry/traces.py", line 528, in _upload_session_report
    resp.raise_for_status()
File ".../aiohttp/client_reqrep.py", line 636, in raise_for_status
    raise ClientResponseError(

Environment • livekit-agents: 1.5.1 • livekit-protocol: 1.1.3 • livekit: 1.1.3 • Python: 3.11 • Deployment: Self-hosted agent connecting to LiveKit Cloud • Agent type: Voice agent (STT + LLM + TTS pipeline) Steps to reproduce 1. Enable agent recording (

enable_recording: true

in job dispatch, or pass

record=True

to

session.start()

) 2. Start a voice agent session 3. End the session (participant disconnects) 4. Observe the error in agent logs during session cleanup What we've tried Upgraded all LiveKit packages from 1.3.x to 1.5.1 — same error • Confirmed the JWT is generated with

ObservabilityGrants(write=True)

• Confirmed the request is well-formed (protobuf header + chat history JSON + audio) • The agent session itself works correctly — the error only occurs during the post-session observability upload Notes The 500 is a server-side error from the

/observability/recordings/v0

endpoint — the client successfully constructs and sends the request • Agent observability is enabled in the LiveKit Cloud project settings (

enable_recording: true

appears in the job dispatch) • The error is non-fatal but produces noisy logs on every session • Is there any additional project-level configuration required to enable the observability backend?

Thanks — I’m responding to the 500 error you see when the agent uploads the session report to /observability/recordings/v0 at session end. Summary of what’s happening • The error is a 500 Internal Server Error returned by LiveKit Cloud’s observability backend when the agent posts the session report. • Your environment and steps show the agent constructs and sends the request successfully (error raised on response), ObservabilityGrants(write=True) is present, and this happens only during the post-session upload. Checks to run next (quick list) • Confirm project-level Agent Observability is enabled in the LiveKit Cloud project settings (you noted this is enabled; double-check the same project ID the agent connects to). • Verify the JWT used for the upload definitely includes

ObservabilityGrants(write=True)

at the time of upload (token not expired). • Capture the full HTTP response body returned with the 500 (sometimes it contains an error id or message). • Re-run with agent logs at DEBUG (

--log-level=DEBUG

) and paste the upload-start and upload-finished lines plus any surrounding tracebacks. • Check for intermediate proxies/gateways/load‑balancers that might alter/limit large multipart uploads (request body size or timeouts). • Confirm the uploaded payload size and whether large sessions (long recordings) correlate with the failure. What is likely • A 500 indicates an internal failure on the LiveKit Cloud observability backend rather than a malformed client request. You’ve already verified request construction and grants, so this points to a server-side problem or an edge case (payload size, unexpected content, or a backend service error). If you can share (omit secrets) • The full HTTP response body for the failed request. • The DEBUG agent log around the upload (upload start, request size, response body). • Whether failures are consistent or only for large/long sessions. What I can do next • If you share the response body and DEBUG logs I’ll look for clues and suggest a workaround (e.g., chunking, shorter recordings, retry timing). • If the logs show no actionable client-side cause, I’ll confirm that this is a server-side error and recommend capturing the response details above so we can investigate further. Which of the items above can you provide now? Sources: Voice Agent Architecture: STT, LLM, and TTS Pipelines Explained | LiveKit | Why is my Agent insights data missing? | LiveKit | Why is my Agent insights data missing? | LiveKit | Python Agents: Handling exceptions in your entrypoint | LiveKit | Streamline troubleshooting with Agent Observability | LiveKit, with interactive elements