I am using Egress to record and send the the recording on s3. I receive all egress webhooks for example participant_left and track_unpublished except egress_updated and egress_ended. On the livekit cloud --> Egress tab, I can see instance for subscriber uploaded mp4 to s3.

Thanks for reaching out, Palwisha. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

here is my implementation: import json from fastapi import APIRouter, Depends, HTTPException, Request from livekit.api.access_token import TokenVerifier from livekit.api.webhook import WebhookReceiver from sqlalchemy.ext.asyncio import AsyncSession from app.core.config import settings from app.core.database import get_db from app.core.logging_config import get_logger from app.schemas.interview_session import EgressWebhookResponse from app.services import interview_session as session_service logger = get_logger(name) webhook_router = APIRouter(tags=["Webhooks"]) _webhook_receiver = WebhookReceiver( TokenVerifier( api_key=settings.LIVEKIT_API_KEY, api_secret=settings.LIVEKIT_API_SECRET, ) ) @webhook_router.post("/webhook", response_model=EgressWebhookResponse) async def livekit_webhook( request: Request, db: AsyncSession = Depends(get_db), ): """Handle LiveKit egress webhook events (recording completion).""" body = await request.body() auth_header = request.headers.get("Authorization", "") logger.info( f"Egress webhook received: " f"content_type={request.headers.get('content-type')}, " f"body_len={len(body)}" ) try: event = _webhook_receiver.receive(body.decode("utf-8"), auth_header) webhook_data = { "event": event.event, "egressInfo": { "egressId": (event.egress_info.egress_id if event.egress_info else ""), "roomName": (event.egress_info.room_name if event.egress_info else ""), "status": (event.egress_info.status if event.egress_info else ""), "fileResults": [ { "filename": fr.filename, "location": fr.location, } for fr in (event.egress_info.file_results or []) ] if event.egress_info else [], }, } except (ValueError, KeyError) as e: logger.warning(f"Webhook verification failed, falling back to raw JSON: {e}") try: webhook_data = json.loads(body) except json.JSONDecodeError: raise HTTPException(status_code=400, detail="Invalid webhook payload") except Exception as e: logger.error(f"Unexpected error processing webhook: {e}") raise HTTPException( status_code=500, detail="Internal error processing webhook", ) return await session_service.handle_egress_webhook(db, webhook_data)

Thanks — this is very helpful. I reviewed your FastAPI handler and it looks mostly correct, but a few things could cause LiveKit’s

egress_updated

/

egress_ended

events to be dropped or not reach your processing pipeline. I’ll call out the likely problems, quick fixes to try, and some targeted diagnostics so we can pinpoint the issue. What I’m responding to • Your webhook handler using LiveKit’s

WebhookReceiver

to verify signature, then building

webhook_data

and returning

session_service.handle_egress_webhook(...)

. Possible issues and fixes to try now 1. ACK-fast, process-async ◦ Problem: Your endpoint awaits and returns the result of

session_service.handle_egress_webhook(...)

. If that function does heavy I/O or long work, LiveKit may see higher latency or non-2xx responses and its queue can fall behind (events may be dropped when queue depth/age limits are hit). ◦ Fix: Immediately return 2xx after verifying and enqueuing the payload, then process the payload asynchronously (e.g., push to a task queue like Celery/RQ/Kafka or spawn a background task). ◦ Example pattern: i. Verify signature + minimal validation ii. Put raw payload (or parsed data) into a durable queue iii. Return 200 immediately iv. Worker processes queued payloads and calls

session_service.handle_egress_webhook

1. Response status codes and error handling ◦ Problem: If

handle_egress_webhook

raises an exception, your endpoint returns non-2xx. Repeated non-2xx responses will trigger retries and may lead to delivery problems. ◦ Fix: Ensure the HTTP response to LiveKit is always 2xx after you verify the webhook and enqueue the payload. Log exceptions separately and do not let them affect the webhook HTTP response. 1. Signature verification fallback path ◦ Observation: You fall back to raw JSON if

receive()

raises ValueError/KeyError, which is reasonable. But if signature verification is failing often, LiveKit may still be retrying or the event may be treated as invalid in your app logic. ◦ Diagnostic: Log the specific exception and include the event type (if available) and a truncated webhook body when signature verification fails. Confirm

settings.LIVEKIT_API_SECRET

is correct and matches the LiveKit Cloud webhook secret. 1. Logging to confirm receipt ◦ Action: Add a short log immediately after

body = await request.body()

and again right after

event = _webhook_receiver.receive(...)

that includes: ▪︎ timestamp ▪︎ event type (if extracted) ▪︎ egressId/roomName when present ▪︎ HTTP response you will return (200 vs other) ◦ This confirms whether `egress_updated`/`egress_ended` ever reach your endpoint. 1. Ensure content-type / decoding is handled robustly ◦ Observation: You decode

body.decode("utf-8")

and pass to

receive

. That’s fine for JSON, but ensure there aren’t content-type edge cases (rare). ◦ Action: Log

content-type

header (you already log it) and any parse errors. Targeted diagnostics I need from you 1. Do your server logs show any incoming webhook requests around the times when the Egress tab shows S3 upload complete? Specifically, look for any entries that include request timestamp, status code, and latency for webhook POSTs. 2. When

egress_updated

/

egress_ended

are missing: do you see any retries from LiveKit in your logs (repeated POSTs for the same egressId)? Or no incoming requests at all? 3. How long does

session_service.handle_egress_webhook(...)

typically take? Does it perform DB writes, S3 calls, or other external I/O? 4. Do you see any exceptions from

handle_egress_webhook

in your logs (traceback + timestamp)? 5. Can you temporarily change the handler to this minimal flow and test once, to see whether the missing webhooks arrive? ◦ Immediately verify signature and parse event. ◦ Log the event type + egressId. ◦ Return

{"ok": true}

(HTTP 200) without calling

handle_egress_webhook

. ◦ If `egress_updated`/`egress_ended` appear in logs after this change, the issue is processing time or errors in

handle_egress_webhook

. Small code adjustments I recommend • Return 200 before calling

handle_egress_webhook

(or push to background task). In FastAPI you can use BackgroundTasks or a queue:

from fastapi import BackgroundTasks

@webhook_router.post("/webhook")
async def livekit_webhook(request: Request, background_tasks: BackgroundTasks, db: AsyncSession = Depends(get_db)):
    body = await request.body()
    auth_header = request.headers.get("Authorization", "")
    try:
        event = _webhook_receiver.receive(body.decode("utf-8"), auth_header)
        webhook_data = {...}  # same as you have
    except Exception:
        webhook_data = json.loads(body)  # keep your fallback handling
    # enqueue the real work
    background_tasks.add_task(session_service.handle_egress_webhook, db, webhook_data)
    return {"ok": True}

• If you can’t use

BackgroundTasks

, push the raw body into a durable queue (recommended for reliability). What I can do next • Help you convert

handle_egress_webhook

to an async-safe background job pattern or build a suggested reliable queue flow. • Review relevant snippets from your logs (timestamps + status codes + latencies) to spot missed deliveries. Paste non-sensitive lines and I’ll interpret them. • Review a small excerpt of

handle_egress_webhook

if you want me to spot long-running or failing operations. If you can run the quick minimal-handler test above and paste a short sample of the webhook-access logs (timestamp, path, status, latency) for the suspect time window, I’ll help you determine whether the problem is delivery (no requests reaching you) or processing (requests arrive but are dropped/errored).

The issue is server never recieves the webhook, on livekit cloud can i see somewhere the webhook events send by livekit?

I don't have the answer you're looking for. You could also try asking your question: • in one of the other Slack channels or • to https://deepwiki.com/livekit/livekit_composite which is trained on all LiveKit source code If you find the answer, please post it here to help others!

On livekit cloud can i see somewhere the webhook events sent by livekit?