This message was deleted.
# citrix-netscaler
s
This message was deleted.
🤔 1
n
The URL however, references cve20234966 and cve20234967, which are completely new CVE's. Woo.
s
n
Good find, I could only find the first one, https://cve.report/CVE-2023-4966
s
It's released now
n
Saw that too, getting very lucky timing on the refresh.
Thank you, was just about to. 😄
s
Just an FYI, I upgraded a HA pair now (didn't force stay primary or disable sync, and the sync wasn't automatically disabled when upgrading secondary).
m
I'm just reading CTX579459 CVSS 9.4 sounds bad, but I don't understand why exactly? Does anybody understand why and care to explain?
m
I learned my lesson from last time. I'm pulling these binaries down before the west coast wakes up.
n
Same, Matt. Soon as it was able to be downloaded, snagged it. Marco, I don't think we know, yet, what type of disclosure we're talking about, best bet is to find it on the CVE-pages (that have no info right now) or wait for someone with more info.
s
Im very curious as to what sensitive information they're talking about. User info/system info?
➕ 2
c
cant download the new 13.1 firmware as site is super busy atm
n
Well....I'm having problems logging in to get the download...lol
f
terrible results trying to access downloads page via the usual citrix accoutns page, but went though the netscaler.com route and was able to pull the download.
s
Upgrade successful on two standalone devices and one HA pair.
m
I'm already patching the firs appliance via ADM service.
g
@Felix Marte Can you provide a direct link via netscaler.com? Can't find it...
n
I clicked the "downloads" link on the bottom of the netscaler.com page and it just 404's
n
Yeah.. the usual race when it comes to this.
f
c
404 for me
n
you'd think as many troubles as they've had with this, they'd bolster the site a bit before making these announcements so it doesn't fall on its face
s
blarg!
n
Well, no need to use autoscaling for something like this.. =p
g
Thank you Felix. I thought you have another link. This is the "standard" which doesnt work atm. Unbeliveable that citrix cannot provide their updates via a working CDN...
💯 2
f
Yeah sorry @Gero, I usually log into my citrix account first to get to the downloads page and that was worse than a 404 error. Figured cutting out some steps would help a little. The download page worked for me straight away and from multiple endpoints, but as others say its a race scenario.
c
Looks like someone hacked citrix.com SSO and then this vulnerability came out.... cant login to citrix.com to download the firmware. Customers who are on 13.1 49.13 version, should they upgrade to 13.1 49.15 or directly 14.1 ? which one is better?
m
My opinion: I would never switch my major release because of a CVE. I'm on 13.1 and I stick to 13.1.
👍🏽 1
💯 3
👍 2
n
Yeah.. and I'll try to steer clear from 14.1 for a while, it's still too new.
g
No need to apologize @Felix Marte 😉 thanks for trying :)
m
With CVE-2023-3519, some orgs found themselves out of luck with 12.1 because the version was no longer supported. They HAD to upgrade to 13.0 or 13.1.
☝️ 1
m
But that does mean something went wrong before, doesn't it?
k
got the firmware from ADM service to upload to netscaler
can copy it from one of those netscalers to the ones that are not in my ADM
farce
j
Yup, I was in @Matt Sliva's scenario. I was hanging on to the last of 12.1s until CVE-2023-3519
j
@nsvegman initial 14.1 release was the same as the latest 13.1 at that time, but agreed it will move forward faster
If anybody needs access to the downloads, hit me up, I can provide a link we use internally at my company
n
has anyone made it through the standard account/downloads route to grab the firmware yet?
j
@Neal Dolson yes, before the article was public. Guess everybody is in a frenzy now
j
I can't even get logged in, just 404 errors
n
Citrix SSO crashed, won't get passed that
c
Same here
n
almost 2 hours after releasing the disclosure and their site is still FUBAR
c
Does anyone know the process on how this vulnerability identification and firmware creation process work? When does citrix know about the vulnerability and when do they create the firmware file?
j
Depends on the channel from which the report comes. Responsible disclosure allows them to work on the analysis and a patch This then needs to be integrated in the test runs and release builds. As far as I know, pretty standard operating mode.
⬆️ 1
j
Notification -> Disclosure -> Patch time can vary. Under the best circumstances, you hope there is no POC, or worse, active exploit in the wild by the time a patch is released
👍 1
m
If I recall the last time there was a rush on downloads, if you used Incognito mode and tried again, you may have some success with SSO working.
n
That worked for me the last time as you said, however this time it just bombs out at sign in every time. I even used a VPN to try accessing from different global regions and it made no difference.
c
@Matt Sliva
n
That being said....I did fiiiiiinally. just get through on my 27th attempt
✅ 2
c
UAT netscaler done with 13.1.49-15 update seemed to go ok altho i did get an SSL error when launching apps about not trusting a entrust intermediate ssl cert. I have a new W11 machine so exported the entrust intermediate cert and imported it into my trusted intermediate CA store and issue went away. I also checked on the ADC and the SSL cert for the GW wasnt linked to the entrust intermediate cert so who ever installed the new cert this year didnt link it correctly
👍 2
c
VPN to Australia and download from there worked fine for me 🙂
j
Looks like pressure is lifting, I'm in and downloaded from California.
🤣 1
m
I'm already done with all 26 VPX with ADM as a service. Took me >90 minutes for everything.
🐎 1
👏 2
c
adm says insufficient disk space after deleting lot of files from /var
m
ADM as a service can clean up disk space semi-automatically
c
This is the first time i m using ADM for upgrading instance. Where in ADM can we find this option?
j
ADM on prem (and if so, what version) or ADM service @Chai?
c
@Jon Bucud ADM on prem VM connected to citrix cloud, if thats what you're asking about. 14.1-11.37
g
Now I can accept license agreement but download doesn't start....
c
same here
g
Same. An error occurred while processing your request. 😞
d
cant download 13.1....
g
I just contacted support by phone. Their technical team is working on it. No lessons learned from last time.
🙄 2
n
I uploaded a copy to my personal onedrive
r
^ Sad that we've gotten to this point were updates have to be distributed via OneDrive. Thank you, Nick. Nice to have the option.
👍🏽 1
👍 1
g
But good that we have such a community 😉
❤️ 3
🙂 1
m
@Chai I ditched our ADM on-prem, so I don't know if the on-prem version has it. ADMaaS has it.
p
8 hours later... 🤦
😂 1
🙈 1
🎉 3
🤫 2
k
that's really bad
just imagine there would be a malicious actor amongst us that would spread out a firmware that's been forked
c
The surge protection and rate limiting must not be working on their own ADCs 🤣🤣
🍻 1
l
So this update was supposed to protect us from a dos and citrix dos’d themselves The irony
n
I'm finally able to reliably download firmware from multiple devices/locations, so maybe they've "fixed the glitch"
j
Same, I'm ADM'ing through our instances as we speak
I don't know why...but i still do one ADM job per HA pair. I know you can mass select them, I just "feel" more comfortable doing a pair at a time.
o
For anyone able to get around the download issues today, any feedback on deployment? Are we all beta testing for Citrix at the same time? 😛
n
Seems fine so far, no auto reboots or crashes 🤞🏽
💯 1
n
fine so far here, 1 standalone and 1 HA pair upgrade to 13.0 build
j
Updated half of my 5 HA pairs and 1 standalone (2 stage, 2nd stage set for Saturday)
o
on my end, I completed a single Nutanix based VPX from 13.1.49.13 to 13.1.49.15 without issue. More tomorrow!
l
I was stuck for about 45 minutes on some SaaS catalog install portion. I had to login again, reboot, and everything was good.
📣 1
s
Same here @Lakatta. When upgrading from GUI we saw the same, had to cancel the upgrade. Upgrading from CLI worked perfectly.
📣 1
o
For what it's worth, I stopped using GUI for any/all NS updates last year, and haven't had one fail since.
👍🏽 1
👍 1
n
We use the ADM scheduled upgrade jobs, works beautifully
💯 2
c
Does it take backup when you create ADM scheduled job? What if it fails and need to revert back?
k
If you choose to, it will create a backup
n
Yes, we have e-mail a PDF of the job, example:
c
ok
l
I am going to finally bite the bullet and use our on prem adm next time.
j
I got brave and did two sites in one ADM job yesterday! (But still did another 4 sites in their own job 🙃 ) I really like the two-stage ADM option, which leaves 1/2 of the HA pair on the older firmware for a configured period, just in case you need to flip the HA pair back to the old FW to troubleshoot.
d
Anyone used cloud service for upgrade?
👋🏻 1
m
I'm already done with all 26 VPX with ADM as a service. Took me >90 minutes for everything.
Quoting myself from earlier in this thread. Just scroll a little up.
s
oh nice @Jon Bucud didn't know that was an option.
👍 1
r
Has anyone seen any problems running ADM 13.0 with 13.1 ADCs/NetScalers? 13.0 gave 30 free licenses, but 13.1 in later versions gives only two. (I have not noticed any problems so far with Web Insight and we don't use ADM for upgrades.)
j
That's the main reason I'm still on the version I'm on Robert... I'm holding onto my 30 "Entitled Virtual Servers"
👍 1