https://livekit.io logo
Join Slack
Powered by
<@U08CKJH7C12> <@U09G4AF3NFR> - There is an issue ...
# telephony
p
@refined-appointment-81829 @lively-winter-88668 - There is an issue for certain calls from one our dialers where LiveKit is NOT sending back a response to the dialer. Is it possible to check please? Attaching a SIP invite in the thread. This is a 911 and 100% of our traffic from TCN is down.
Copy code
Made a test call, checked our SIP logs. We are not receiving a response to our invite

02/20/2026 14:28:26.385 -05:00: 10.128.0.4:5060 -> 161.115.181.247:5060
INVITE sip:+XXX@3tqm7ro6kbs.sip.livekit.cloud:5060 SIP/2.0
Record-Route:  <sip:XXX;lr;ftag=as6ba53c1c;vsf=AAAAAB0BDwQMAwwHCAJxA3AfAAABHwMfAxcDOTI-;vst=AAAAABtUXFoFUgxSBFZWUSYHTUhVDkMLDl5bERwXXEceCgtDUytYRABSXldbVC40;nat=yes>
Via:  SIP/2.0/UDP 162.222.181.125:5060;branch=z9hG4bK455a.b3435f12308dcdefff900a3e3e326423.0
Via:  SIP/2.0/UDP 162.222.181.125:5060;branch=z9hG4bKsr-j4IPOlV7MGQKatycM.y7MGZfM.V7M.qLzBjAWByXNEKBgRI6gRPQM.y7MGZfM.V7M.qLzuaVHRaYpB1JNEt736cQkBIvalaJmlPfz.FLgRPLzweAowe4H.jAWBy*
Max-Forwards:  69
From:  "+XXX" <sip:+XXXXX@10.0.112.192>;tag=as6ba53c1c
To:  "+XXXX" <sip:+XXX@3tqm7ro6kbs.sip.livekit.cloud>
Contact:  <sip:XXX:5060;line=sr-N6IAzB3Az.PuWlqAM.eyM.y7MGZfM.V7M.qLzBjAWByX3RfY39MQM.y7MGZfM.V7M.qLiBjAWB02MP**>
Call-ID: 30cc406b6e5017df4fc3b6aa0e5137de@10.0.112.192:5060
CSeq:  102 INVITE
Date:  Fri, 20 Feb 2026 19:28:26 GMT
Allow:  INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, SUBSCRIBE, NOTIFY, INFO, PUBLISH, MESSAGE
Supported:  replaces
X-TCN-SOURCENUMBER:  XXXX
X-TCN-CALLID:  XXXX
Content-Type:  application/sdp
Content-Length:  281
User-Agent:  T
P-Asserted-Identity:  <sip:+16094579012@162.222.181.125>

v=0
o=root 892029708 892029708 IN IP4 34.133.237.208
s=Asterisk PBX 1.8.28-cert5
c=IN IP4 34.133.237.208
t=0 0
m=audio 12268 RTP/AVP 0 8 101
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-16
a=sendrecv
a=rtcp:12269
a=ptime:20
I have raised a support ticket as well, added here for visibility. Would really appreciate if you can help check this
Copy code
ulimit -n 50000 && sipp XXX.sip.livekit.cloud:5060 -s +XX -m 1 -t un
I ran a test with above 
sipp
client and the SIP invite timed out without any response.
Copy code
Resolving remote host 'XXXX.sip.livekit.cloud'... Done.                                                                             
  ------------------------------ Scenario Screen -------- [1-9]: Change Screen --                                                                
    Call rate (length)   Port   Total-time  Total-calls  Remote-host                                                                         
    10.0(0 ms)/1.000s   5060      31.61 s            1  143.223.94.131:5060(UDP)                                                                 
                                                                                                                                             
    Call limit 1 hit, 0.0 s period          0 ms scheduler resolution                                                                            
    0 calls (limit 30)                      Peak was 1 calls, after 0 s                                                                      
    0 Running, 2 Paused, 0 Woken up                                                                                                          
    0 dead call msg (discarded)             0 out-of-call msg (discarded)                                                                    
    0 open sockets                          0/0/0 UDP errors (send/recv/cong)                                                                
    0 open sockets                          0/0/0 UDP errors (send/recv/cong)                                                                
                                                                                                                                             
                                   Messages  Retrans   Timeout   Unexpected-Msg                                                              
  0 :      INVITE ---------->         1         5         1                                                                                  
  1 :         100 <----------         0         0         0         0                                                                        
  2 :         180 <----------         0         0         0         0                                                                        
  3 :         183 <----------         0         0         0         0                                                                        
  4 :         200 <----------  E-RTD1 0         0         0         0                                                                        
  5 :         ACK ---------->         0         0                                                                                            
  6 :       Pause [      0ms]         0                             0                                                                        
  7 :         BYE ---------->         0         0         0                                                                                  
  8 :         200 <----------         0         0         0         0                                                                        
                                                                                                                                             
  ------------------------------ Test Terminated --------------------------------                                                            
  ----------------------------- Statistics Screen ------- [1-9]: Change Screen --                                                            
    Start Time             | 2026-02-21    01:24:35.391195    1771617275.391195                                                              
    Last Reset Time        | 2026-02-21    01:25:07.008125    1771617307.008125                                                              
    Current Time           | 2026-02-21    01:25:07.009274    1771617307.009274                                                              
  -------------------------+---------------------------+--------------------------                                                           
    Counter Name           | Periodic value            | Cumulative value                                                                    
  -------------------------+---------------------------+--------------------------                                                           
    Elapsed Time           | 00:00:31:618000           | 00:00:31:618000                                                                     
    Call Rate              |    0.000 cps              |    0.032 cps                                                                        
  -------------------------+---------------------------+--------------------------                                                           
    Incoming calls created |        0                  |        0                                                                            
    Outgoing calls created |        0                  |        1                                                                            
    Total Calls created    |                           |        1                                                                            
    Current Calls          |        0                  |                                                                                     
  -------------------------+---------------------------+--------------------------                                                           
    Successful call        |        0                  |        0                                                                            
    Failed call            |        0                  |        1                                                                            
  -------------------------+---------------------------+--------------------------                                                           
    Response Time 1        | 00:00:00:000000           | 00:00:00:000000                                                                     
    Call Length            | 00:00:31:510000           | 00:00:31:510000                                                                     
  ------------------------------ Test Terminated --------------------------------                                                            
  2026-02-21    01:25:07.005728    1771617307.005728: Aborting call on UDP retransmission timeout for Call-ID '1-45461@192.168.0.104'
@refined-appointment-81829 - Can you please check on this 🙏 ?
@dry-elephant-14928 - is it possible to check why the response for above SIP invite is not being sent please? Apologies for tagging you but this is affecting 100% of our traffic from one of the dialers. Would really appreciate your help Project ID - 
p_3tqm7ro6kbs
Our Agent is self-hosted on us-east-2 region
d
is it outbound or inbound?
p
inbound
d
for this particular dialer, nothing is going through? but others are fine?
p
Yes - Twilio is coming through fine but TCN is not working. I have shared the SIP invite that they are sending to LiveKit
b
@proud-match-31207 where the sipp was failing could you do a dig on 3tqm7ro6kbs.sip.livekit.cloud and tell me what the results are .
Copy code
dig 3tqm7ro6kbs.sip.livekit.cloud
d
are they using Telnyx by any chance? looks like they have an outage: https://status.telnyx.com/
we do not use cloudflare on our sip stack. and other invites are coming through
p
No TCN is a completely separate dialer afaik - https://www.tcn.com/, unsure how telnyx would be related to do that. Apologies if I have not understood this
@bored-accountant-30277 -
Copy code
dig 3tqm7ro6kbs.sip.livekit.cloud

; <<>> DiG 9.10.6 <<>> 3tqm7ro6kbs.sip.livekit.cloud
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2367
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;3tqm7ro6kbs.sip.livekit.cloud.	IN	A

;; ANSWER SECTION:
3tqm7ro6kbs.sip.livekit.cloud. 10 IN	A	143.223.94.131

;; Query time: 45 msec
;; SERVER: 100.88.255.254#53(100.88.255.254)
;; WHEN: Sat Feb 21 01:50:03 IST 2026
;; MSG SIZE  rcvd: 103
I ran the 
sipp
test from India, apologies if that might be causing issues with 
sipp
test? Our actual calls will be from US and routed to our agent on us-east-2
d
understood. we are looking and have not seen anything yet. it doesn't make sense that twilio is working but TCN isn't. our infra is the same
p
Yes - I thought so too. Both of them are on UDP protocol too. Are there any changes that were pushed around 18:40 UTC today? That's where we last saw a good call
r
@proud-match-31207 do you happen to have pcap from TCN side?
p
Don't have pcap - will request one. But I have the above SIP invite that I have attached on the thread, does that help?
r
we are digging. Just looking for as much detail as possible.
Do you see a response to that INVITE?
p
I also tried making a call using linphone to the same SIP address, it seems like it is reaching to the agent, when I ended the call, it didn't receive participant disconnected event though (could be linphone). Attaching the pcap of linphone call I made from livekit dashboard The pcap does show some unseen segment on wireshark
SCL_3rvTRBn83rfb_Nf2vTJNeNV.pcap
r
But that is different thank with TCN right?
p
Do you see a response to that INVITE?
No that is the issue, the response is NOT being sent to the invite.
👍🏼 1
r
let me look at that PCAP. Standby
👍 1
b
+1 to the issue. Spoke to someone from TCN, they mentioned that they are not receiving response to INVITEs For our systems we started noticing this issue after 13:00 ET (18:30 UTC) if it helps narrowing down any of the events that could have caused this.
p
TCN hasn't had any changes recently to any of the call services, there has also been no service restarts in the past 2 weeks, all we can see is it leaves the network towards the Livekit IP. We have other sip transfers landing correctly to different services, so it seems to be isolated ot livekit, Either they have an issue with the GCP route from US central or specifically our IP address, but we have no relationship with livekit directly. Prodigal team are you able to contact support and ask for any traces from their edge?
@refined-appointment-81829 - From TCN team - could this be an issue that can be checked (highlighted in bold)?
l
Hi @proud-match-31207 any updates on PCAP from them? 🙏
p
I have requested for it Hamdan, haven't received it. Will share it as soon as it is shared with us.
l
thank you thank you! sorry for the troubles faced
p
@loud-dawn-97520 - this is the pcap they shared
l
thank you
d
from our side we are seeing traffic from other providers going in fine.. it's unclear why it's not able to see traffic from TCN
p
David - could it be a networking issue?
GCP route from US central or specifically TCN IP address
Does the above PCAP help?
@loud-dawn-97520 - any update?
d
we are not sure.. the call doesn't exist in any of our logs.. so that packet didn't get delivered
can you try switching to ;transport=tcp? to see if it improves it?
yes the pcap helps.. we are all investigating on our end
👍 1
p
can you try switching to ;transport=tcp? to see if it improves it?
Is there a way I can make this change on the trunk side to force it to TCP or would the SIP URI itself needs to be updated to 
;transport=tcp
and tested?
d
yes you can force it to be TCP on the TCN side
they are initiating the request, so we cannot control it
👍 1
p
Got it. Asked them to try using the tcp protocol once.
@blue-cat-31749 - did you receive any helpful response from TCN?
d
just checking in, are you able to convert to TCP from the TCN side?
p
They haven't responded david 😞, followed up again. https://www.cloudflarestatus.com/incidents/kwy3dt82bwbt -> Cloudflare is having connectivity issues too, have asked them if they could be using cloudflare which could be causing network issues
👍🏼 1
d
we just disabled the chicago region.. even tho health metrics looks normal
just in case there are routing issues from TCN to us.. can you check if things are working for you?
p
Checking
r
Keep us posted on if switching to TCP made any difference.
p
They responded, they can't switch to TCP it seems, they are still checking that internally. Anything else that we can try here?
I am testing a call on TCN after david mentioned Chicago was disabled, will report back on this
👍🏼 1
r
If they can reduce UDP packet size that would be helpful test since they cannot switch to TCP
l
Hi, we have opened an incident here and will update our findings there
p
After chicago region was disabled, we are able to receive calls from TCN.
👍🏼 1
👍 1
Will check back on the status page for the RCA. Really appreciate your help team, thank you for the help here 🙌 . Really weird that routing away from chicago helped here, I guess that cloudflare has something to do with this
👍🏼 1
l
@proud-match-31207 just following up on this... one of the primary motivations behind the new "report service outage" workflow, described at https://livekit-users.slack.com/archives/C01KZ61HZRR/p1771597745237399, was to make this process less ad-hoc and easier to triage, identify, and prioritize issues on our side. Even tagging folks from the LK side, it is very easy for us to miss these urgent messages in Slack, and especially since we plan to sunset this channel at the end of the week, I just wanted to make it clear that we highly value these community outage reports.
p
@lively-winter-88668 - thanks for mentioning this. I hadn't realized we can report incidents via cloud.livekit.io and it was a feature till my teammate mentioned it to me. Apologies for creating adhoc slack threads - will follow the above defined guidelines. Thanks again for helping out here.
🙏 1
Apologies for reviving the thread here again folks, would appreciate if there was any RCA around this 🙏 . Thank you https://status.livekit.io/incidents/0xrfd1mrnkht -> The status page for this incident doesn't have it, hence reaching out here.
l
@proud-match-31207 No worries, let me ping the team to ask
🙌 1
2 Views
Open in Slack
PreviousNext
Powered by