https://worldofeuc.com logo
Join Slack
Powered by
This message was deleted.
# citrix-netscaler
s
This message was deleted.
s
Sure you were compromised by earlier CVEs this year?
s
That is my initial reaction as well but we applied the fixes in a timely fashion and even recreated some ADCs from scratch just to be sure.
s
Any session currently connected to the NetScaler GUI or SSH?
s
Yes, but only my session. We have also created a completly new ADC and imported the settings, licence and certificate. The problem reaccurs within a couple of seconds of it being online. The Traffic is all UDP Port 443 from a different public IP now that is unknown to me. Would some kind of brute force attack on the gateway create such a behavior? i.e. thousands of UDP requests creating Output traffic on the ADC?
s
does it happen without reimporting all the settings and only exposing the ADC to Internet?
s
yes. we have concluded for now that it is/was some kind of ddos or amplification attack. when you change the public ip of the ADC the problem is gone but you can still see something from outsides trying to reach the old ip on the firewall.
d
I can confirm this issue, we have the some behavior in our lab and in customer environments. It only helps to disable EDT (UDP 443) on VPN VServer or block that on your FW.
👋🏻 1
s
Thanks for the info, that's really interesting and... weird. Do you have any details on what's the root cause of this? (Greetings from @Marco Hofmann btw 😉 )
😄 1
d
Sorry, I don't have more details about it. But it can always happen because the service is published externally. We have blocked UPD:443 traffic on our FW ;-)
s
Yeah, we did this on our firewalls now as well. It's the first time this is happening so we were and still are a bit alert. Did this happen in the past to you already?
d
Not really. But more and more confirmations: https://twitter.com/msandbu/status/1340762900114386946?s=20
👍 1
s
Any idea how we can check if we are also under attack? Current throughput is 11MBit in, 12Mbit out.
d
You can create a nstrace and find some udp DTLS handshakes. See my attached screenshot for some example.
s
Thanks!
a
Is it really attack not “time bomb” bug?
m
The issue correlates directly with incoming DTLS UDP 443 client hellos. If we block them, nothing visible is left.
👍 1
This is what our "attack" looked like last night:
👍 1
p
If we disable/block UDP 443 every active Citrix session should fall back to TCP without need to reconnect?
d
I am not sure if the user does not have to reconnect his session. Because you reconfigure the communication flow and block UDP 443 traffic just in time. 😉
👍 1
p
Ok just tested to block outgoing UDP 443 on my local client using windows firewall. The session was frozen for about 20 seconds and startet working again without having to reconnect. In director i can see that it switched back to TCP.
👍 7
a
Do we have any “auto” protection against such attacks ? Would be nice to automatically block such IP’s with such behaviors...
m
IHMO that is the job of the firewall, and not of Citrix ADC.
a
Right however check how many people complain about attack 🤔 Would be nice to have one more layer of protection on ADC / NG
d
Block the bad countries on your FW, but Azure IPs are also included as source IPs of the attackers. 😉
g
I’m seeing this too. I’m blocking DTLS on our Palo Alto at this time.
f
We had this problem as well. Started on Saturday. Wouldn't this also affect Citrix Cloud CVAD customers with EDT enabled? I wonder how Citrix is responding to the attacks.
s
We have an open support case but the help/answers are not that helpful. Even with all the new information this is the latest reply:
"As I stated earlier, in addition to setting up rate limited, we recommend to also try extended ACLs. Please refer to the following articles, if you have not already.
https://support.citrix.com/article/CTX127917
https://docs.citrix.com/en-us/citrix-adc/current-release/networking/access-control-lists-acls/extended-acls-and-extended-acl6s.html"
m
@Marco Hofmann and @Daniel Weppeler did you guys try to change the default client hello verify to enabled in the dtlsprofile? If you're using the default DTLS Profile on the ADC
a
This might be a (temporary) solution. Spoke a lot with Citrix PM team today. They also mentioned this setting but there might be more to check later. Let’s see were they come up with…
👍 2
m
I talked with a bigger customer today and they are not seeing an issue (Might just not be attacked) but we also use a customer DTLS profile at them, so that is why i mentioned it 🙂
✅ 2
m
@Mads Petersen As we don't really need EDT, we just blocked UDP 443 for the Gateway VIP at the Firewall level. Issue "gone".
m
Yeah, that works of course 🙂 I was just wondering if it was tried.
d
@Mads Petersen Yes, I have tried this workaround in our environment, but in addition it could also affect the performance of the EDT session. Because every DTLS Request have to be answered with an client hello verify. But the biggest problem is why it is not enabled by default. 😠
m
@Daniel Weppeler Did it work as a workaround? Only the Client Hello is verified right? So once that is done, the connection functions as normal. We haven't seen any negative issues with it on at that customer, but to be fair they are also running some big boxes. But i guess if you're hit with a lot of Client Hello, it would have a negative impact and not in day-2-day operation.
d
The bandwidth is back to normal, but I did not test an EDT session. Sorry. 😄 Will test this tomorrow in some environments with enough hardware specs and test appliances only.
m
And the Verify will not be answered, because the destination IP isn't expecting the request (Guess it's some kind of amplification attack)
✔️ 1
d
Yes, the source IP sends the same request again and again.
m
spoofed request, like the good old dns amplification attack. Thanks for the headsup, lucky this seems like an easy fix (block in firewall - because client hello verify might end up having a negative impact like you said) p.s I would love some packet captures of it, if someone got some they can share (No sensitive data in them please)
m
a
Mitigation with small performance degradation: set dtlsProfile nsdtls_default_profile -helloVerifyRequest ENABLED
m
I'm sorry, no offense, but is this really a solution?
😉 1
a
Might be that Citrix will patch this later, or comes with different/more fixes. But for now this seems to solve the issue! https://twitter.com/AntonvanPelt/status/1341115675134320641
m
Yes, I saw your tweet, but for now (in my very personal opinion) I don't like that solution. The DDOS filled our corporate firewall state tables pretty quick (80% instead of the usual 20%) so we decided for the UDP block. Would this ADC command have helped with out states table? I stick with my solution.
a
Well I spoke with folks that had enabled this setting by default and they didn’t had this issue. So at least, to keep DTLS and EDT enabled this is a good point to start with ;-)
m
Seems like any other kind of protection, where you want to verify the sender is legit. Like Syn flood protection, not sure why it wouldn't be a proper protection.
m
Because I'm not a network guy and I might be mistaken. Right now the more knowledgeable (more network knowledgeable than me) guys like Carl Stalhood and Thorsten Rood seem to agree with you: https://twitter.com/ThorstenRood/status/1341107461219086340
😀 1
a
haha cool! You can of course wait for the official Citrix statement if you like!
😂 1
a
By the way: do we have anything from Citrix regarding this attack? I didn’t see anything so far... did I miss something?
a
Not yet, afaik they are investigating the issue too. Hopefully they will release more info this week
😕 1
a
At least they should let customers know that they are aware of the problem. Many of the customers still may not be aware of the problem ...
s
When I reached out to Citrix TAM & SE, they confirmed there is no official statement yet and asked me to open a case to work on mitigation.
m
@Stephan Sprathoff opened a case for us and it was quite underwhelming. He knows the details as I'm currently not in the office.
a
I really hate situations when vendors are aware and customers are not informed. I’m not talking about solution - just to inform about ongoing situation. I was wasting time many times when vendor was aware many times with hotfix or ongoing fix in CU but didn’t inform anyone. Life... 🤭
d
FYI > Some adc appliances crashes with enabled dtls hello verify feature: https://twitter.com/thorstenrood/status/1341323212324519936?s=21
m
Yes, i have comment on that issue on my blog post, telling the same story.
✔️ 2
c
The DTSL Hello setting consumed all memory on my 12.1 build 60 appliance.
d
Seeing the same memory usage on my 12.1.60 test box as well. Did not apply to production boxes. Also...is there a "master list" of IP addresses/ranges? I am getting spammed from 168.227.48.0/24 and don't see that listed anywhere.
👍 1
s
I don't have a master list but can confirm the IP 168.227.48.254 is part of the attack
d
91.218.66.56 just popped up too.
👍 1
b
Is this DDoS specific to NetScaler or a general UDP amplification attack that NetScaler just happens to be impacted by?
m
I don’t know. Asked this myself.
p
Two VPX’s we have running 12.1 57.18 crashed hard overnight after enabling the hello verify. We’re blocking UPD at the firewall now.
e
Haven't had any customers reporting crashes myself (yet). I'd advise for now to just blocking 443 UDP as intermediate solution.
p
Anyone any idea how the GW Service is handling this in Citrix Cloud?
o
This thread is super helpful ! Thanks all
👍 1
c
So, when we are under attack, our Throughput (Mbps) Out value will be high, is that correct?
✅ 1
m
If your adc is used, your output will be higher. Just block 443/udp as of now, as the CHV workaround is make ADC unresponsive if the attack is big enough
c
I haven't seen any indication that this is a netscaler specific attack, this is just generic UDP amplification.
c
c
yep, email just went out to the security updates mailing list
👍 1
a
Upcoming Enhancements Citrix is working on a feature enhancement in DTLS to eliminate the susceptibility to this attack. Citrix expects to have this enhancement available on the Citrix downloads page for all supported versions on Jan 12, 2021.
👍 3
Does anyone know what this improvement is about? How exactly will it work?
m
Not without breaking NDA at this moment. I know that is a shitty answer, but the best one I can give you
😉 1
p
fwiw - this popped up in our cloud console today
😉 1
c
are the DDOS amplification attacks constant or do they stop /start? seen a few vpx's with high out bound throughput 28Mbps but with no ica connections connected at the time and when i did a packet capture i could see alot of DTLS internal errors with destination to public ips in romania it seems to of stopped now tho. I will turn off DTLS on the GW Vservers tonight
d
They start and stop and come from all over the place. I have been adding IPs and Domains to a firewall rule to block them as they come in as we use EDT and I'm reluctant to disable it. The number of addresses being hit hasn't been unmanageable and hasn't caused us any serious issues thus far. If anybody is interested here is my current list of blocked IPs/Networks: 168.227.48.0/24 91.218.66.56 168.227.51.0/24 168.227.49.0/24 168.227.50.0/24 104.18.31.24 189.126.88.0/24 192.69.202.251 168.228.4.0/24 46.105.98.55 188.212.103.111 117.27.239.0/24 77.81.2.1 45.113.200.5 67.21.86.234 188.212.103.0/24 51.81.116.0/24 151.251.20.75 138.94.36.0/24 207.180.194.0/24 51.77.249.65 188.212.100.0/24 185.101.104.0/24 185.11.145.0/24 91.239.145.0/24 188.212.102.0/24 151.251.28.31 118.166.69.70 103.219.37.30
c
We also seen traffic from 185.101.104.0/24 yesterday when i did a packet capture. I ended up turning DTLS off on the NS GW Vservers. As soon as i did it the TX throughput dropped by like 30-40 Mbps can imagine those with VPX10/25 licenses its not much fun
👍 1
c
@Arthur do you have link for the dtls feature enhancement? I mean is it officially released on downloads page?
d
a
Citrix didn't do any magic or new features. In the new release, Citrix has fixed bugs and functionality now works properly.
3 Views
Open in Slack
PreviousNext
Powered by