I came across some scenarios where after egress completed, we are pushing the completion event but due to some reason it is not uploaded and after sometimes we can see it has uploaded and especially for larger audio files

Thanks for reaching out, Yethu. I'm an AI assistant and I'm here to help. Give me a moment while I research this topic.

No I am asking, what is meant by egress complete? is it means recording completed or audio upload completed

Thanks — good question. “Egress complete” means the recording session has finished, the media file has been finalized, and LiveKit has completed (or attempted) the upload to the configured destination. If the primary upload failed and a backup was used, the completion event will indicate that (for example,

backupStorageUsed: true

). Sources: Backup Egress Recordings, with interactive elements

how can I know the file upload has been completed then?

Good question — here’s how to confirm the file upload is finished. • Check the

egress_ended

(or

EGRESS_COMPLETE

) webhook payload: it indicates the recording was finalized and upload attempts completed. The payload includes upload status fields and will contain

backupStorageUsed: true

if primary upload failed and a backup was used. • Inspect the egress job status via your egress API or admin logs: the job moves to a completed/ended state when writing and upload attempts are finished. • Verify the file directly in the destination storage (e.g., S3, GCS, or your configured output): confirm the file exists, size matches expectation, and any expected metadata or checksum is present. • If you rely on webhooks, ensure your receiver logs show a prompt

egress_ended

delivery with a 2xx response. Retries or non-2xx responses indicate the event wasn’t acknowledged and may mean upload/finalization wasn’t confirmed to you yet. If you want, share an example

egress_ended

payload or the storage provider you use and I’ll point to the exact fields and checks to confirm upload success. , with interactive elements

Here's the relevant block — resource_cleanup_handler.py235 296 # get_egress_info stores status as str(enum) — always a string audio_blob_url = None if egress_info and isinstance(egress_info, dict): egress_status_str = str(egress_info.get('status', '')) is_complete = ( 'EGRESS_COMPLETE' in egress_status_str # e.g. "EgressStatus.EGRESS_COMPLETE" or egress_status_str == '3' # raw integer fallback ) # Extract blob URL and size from file_results (authoritative upload proof) file_results = egress_info.get('file_results', []) if file_results: first_file = file_results[0] audio_blob_url = first_file.get('location') # ← the blob URL blob_size_bytes = first_file.get('size', 0) blob_duration_seconds = round(first_file.get('duration_seconds', 0), 2) # Gate: need BOTH COMPLETE status AND a URL to fully confirm if is_complete and audio_blob_url: audio_upload_confirmed = True # → Scenario A: push event with blob_url elif is_complete: audio_upload_confirmed = True # → Scenario B: complete but no URL else: ... # → Scenario C: not complete file_results[0]['location'] comes from get_egress_info in livekit_service.py:363 which maps file_info.location → that's the LiveKit FileInfo.location field containing the Azure blob URL. this is the way e are doing currently and file upload is not gigvn gnay link also

Thanks — I’m responding to your code and the missing

location

in

file_results

. Short answer: To reliably know the file upload finished you need BOTH a completed egress status (EGRESS_COMPLETE) and an authoritative upload location (the `file_results[0].location`/blob URL). If

location

is missing, the upload did not succeed even if the status string reports complete. What to check next (quick checklist): • Inspect the full

egress_ended

payload/webhook for error fields and for

file_results

entries — there may be an error reason or

backupStorageUsed

flag. • Confirm your egress destination config (Azure/S3/GCS): correct container/bucket, credentials, and write permissions for the LiveKit egress service. • Check egress service logs for that job: look for upload errors, timeouts, or fallback-to-backup messages around the same timestamp. • Verify your

get_egress_info

mapping returns the raw

FileInfo

fields (size, duration, location) without modification; log the raw

egress_info

when

location

is missing to capture full context. • Ensure you’re treating

EGRESS_COMPLETE

as meaning “recording finalized and upload attempts finished” — but upload can still fail; presence of

location

is the authoritative proof of a successful upload. Recommended code change (minimal): • Keep your gate that requires BOTH

is_complete

AND

audio_blob_url

. But change the

elif is_complete:

branch to mark upload as FAILED/UNKNOWN (not confirmed) and record the egress job ID and any error details for retry or manual handling. Example flow to implement: 1. If

is_complete

and

audio_blob_url

→ treat as uploaded (normal success). 2. If

is_complete

and NO

audio_blob_url

→ mark as upload failed/needs investigation; surface egress job id and

egress_info

into a retry/alert queue. 3. If not complete → wait/retry checking

get_egress_info

or rely on webhook retries. If you share one raw

egress_ended

payload (with sensitive tokens redacted) where

location

is missing, I can point to the exact fields to inspect and suggest targeted fixes. , with interactive elements

see audio is uploaded it is just that ew are not geting proper status

Thanks — I hear that the audio file itself is uploaded but your service isn't receiving the proper status/location. That confirms the problem is with status propagation, not the upload binary. What to verify next (quick actionable checklist): • Log the raw

egress_info

(full payload) when

location

is missing — capture the exact

status

, any

error

fields,

backupStorageUsed

, and

file_results

contents. • Compare timestamps: check when the file appears in storage vs when

get_egress_info

(or webhook) returned the status. This shows whether status lags finalization. • Confirm you are reading the authoritative field:

file_results[0].location

is the definitive upload URL. If that exists in the raw payload, treat upload as complete. • If the file is present in storage but

location

is missing: ◦ Inspect egress service logs for upload completion vs. upload-to-metadata-write timing; metadata (location) write may be delayed. ◦ Check for transient upload errors or fallback behavior (

backupStorageUsed: true

) in the payload. • Ensure your webhook polling/handler uses the raw response and not a cached/older

egress_info

. If you poll

get_egress_info

, retry with backoff for a short window after egress reports complete. • Update your code gate: leave the gate requiring BOTH

EGRESS_COMPLETE

and

location

, but treat the case "complete but no location" as needs-investigation (enqueue for re-check and alert) rather than confirmed success. If you want, paste one raw

egress_info

where storage contains the file but

file_results[0].location

is empty (redact keys). I’ll point to the exact fields and a minimal retry/alert logic you can add. , with interactive elements

Read(C:\Users\yethu.krishnan\OneDrive - Provana India Pvt. Ltd\Documents\Provana_Projects\ContactCenterProject\cc-livekitagent\ai-agent\services\livekit_service.py · lines 243-307) ⎿ Read 65 lines ● Current state — no tenacity, no real retry logic: while elapsed_time < 600: try: list_egress() ← single attempt, no retry check status → return or continue sleep(5) except ANY error: log warning sleep(5) ← same 5s interval, no backoff continue loop ← just burns more of the 600s budget Problems: 1. No tenacity — list_egress() has no @retry decorator. Network blip → exception → treated same as "still processing" → wastes 5s and loops again 2. No exponential backoff — transient API errors always retry at flat 5s 3. Empty response not handled — if egress_response.items is empty (0 items), loop silently sleeps and retries — no log, no counter, could loop forever on a stale egress_id 4. elapsed_time counting — on success path (line 295), sleep happens AFTER status check, so a 600s call actually checks every 5s correctly — but the timing is slightly off (sleeps even after the last failed poll before timeout) 5. No consecutive-error limit — 100 consecutive API failures just eat the full 10 min budget before timing out What tenacity would give: add @retry(stop=stop_after_attempt(3), wait=wait_exponential(...), retry=retry_if_exception_type(Exception)) around just the list_egress() call — separate transient API errors from "still processing" cleanly. Want me to add tenacity to the poll loop?

Thanks — I see the polling loop and the issues you listed. Yes, adding tenacity-style retries around the single

list_egress()

call is the right move. It separates transient network/API errors from “still processing” state and preserves your 10‑minute budget for actual status polling. Recommended changes (concise): • Wrap only the

list_egress()

call with a retry policy (e.g., 3 attempts, exponential backoff) so transient failures are retried quickly without using the main loop retries. • Keep the outer polling loop for status checks (5s interval), but: ◦ Add logging when

egress_response.items

is empty and increment a small counter to bail early if it remains empty for N consecutive polls. ◦ Use exponential backoff on consecutive API failures and a separate consecutive-error limit to abort early (avoid burning the full 600s). ◦ Treat "is_complete but no location" as FAILED/UNKNOWN: enqueue for re-check and alert rather than marking success. • Add short re-check window after

EGRESS_COMPLETE

(e.g., poll file_results for another 30–60s with backoff) to catch delayed metadata writes. Minimal tenacity example (conceptual): 1. Add retry decorator around `list_egress()`: ◦ stop after 3 attempts ◦ wait exponential (initial=0.5s, max=4s) ◦ retry on network/HTTP exceptions 1. In loop: ◦ call retried

list_egress()

◦ if empty items → log + increment empty counter, if counter > 3 → warn and break ◦ if API exception after retries → increment error counter, if error counter > 5 → abort/publish alert ◦ if

is_complete

and no

location

→ enqueue re-check job with the egress_id + timestamp If you want, I can: • Provide the exact patch/diff to insert tenacity and the empty/error counters into your shown function, or • Produce the concrete Python snippet to paste into

livekit_service.py

. Which would you prefer? , with interactive elements