We seem to also be facing downtime, twilio is not receiving any response from livekit cloud on a certain number of calls, just timing out

Do you know which region the failed calls are originating from? Also can you share a PCAP please? Thank you!

here is a PCAP from one of the failures

yes, we are looking into this sorry

@chilly-lunch-55221 this is helpful.. we are seeing that so far all of the reports are pointing to our chicago region.. and somehow it's not getting delivered. but our request volume and success rate is looking normal in that region

we changed it to TCP

Did changing to TCP make any difference?

we are watching

We don't know yet, just changed it

not all calls were failing

Micah, We are investigating. We are trying to identify if our system is having an issue. There is a cloudflare issue and trying to identify if that maybe related

what % of calls were failing for you? the pcaps was showing that the other side didn't get a response.. and our logs don't show any record of that call. so something has clearly impacted deliverability

FYI jake and i work for the same company since switching over to tcp we have not had an error come through in twilio logs yet, we were getting several a minute before

Hi, we will update our findings in this status page reports

@chilly-lunch-55221 @millions-yacht-25550 just following up on this... I know you did report this also through the dashboard, but one of the primary motivations behind the new "report service outage" workflow, described at https://livekit-users.slack.com/archives/C01KZ61HZRR/p1771597745237399, was to make this process less ad-hoc and easier to triage, identify, and prioritize issues on our side. Even tagging folks from the LK side, it is very easy for us to miss these urgent messages in Slack, and especially since we plan to sunset this channel at the end of the week, I just wanted to make it clear that we highly value these community outage reports.

slack is much more convenient for us but i get it.

Happy to continue reporting stuff there too. The muscle memory around perceived response times from a form fill vs a slack message are hard to overcome. It just "feels" like slack is going produce a quicker response. Maybe some messaging in the form (sorry if i missed it) around response times and how the process works might help?

Understood, and in this case it worked really well (I see Hamdan responded in 2 minutes!), but a big part of that was we were already seeing reports from others, so were actively monitoring this forum. Typically we are not continuously monitoring this Slack, so if a report is initiated here, it is easy to miss - the new process is more push, which sends a strong signal that we need to look into something right away and so will not miss the report. For ongoing communication during any incident, I get your point, and that was at least part of the motivation behind us not replacing this Slack community entirely - it is still the best way for real-time comms if needed. Our goal during an incident would be to provide information on our status page, rather than on a 1:1 basis, but when we are gathering information (asking for logs, additional info) we will always be very responsive regardless of how we are gathering that info. cc @refined-appointment-81829 who created the form for the feedback about workflow and feedback.

@millions-yacht-25550 Thank you and that is very understandable feedback. I guess there are a few dimensions to this too. Slack generally is a "fast feedback" sort of channel but it also counts on Darryn and I getting lucky and seeing the message quickly. The report outages gets the issue in a spot where it is easier to see by a broader group of folks especially those in a position to resolve the issue. Many customers are not in the community slack so it is a better signal across the entire user base and lets us see correlation quickly too (if folks use it). But also what I found helpful last week was having the "real-time" ability to ask for additional or specific information instead of email backup. We will continue to refine this process.

This feels like a great hybrid, form + slack. I think a lot of folks keep slack notifications ever-present but not email, so maintaining slack as a channel makes me "feel" like yall are more receptive and responsive.