This message was deleted.

Just came to post this/see if there were more info yet booo

Looks bad, on the very first look

unauthenticated remote code execution? yeaaaah.

Did the Citrix Downloads page crash to a 404.html for anyone else?

DDoS via panicked customers

😱

Yay! This'll be fun.. not.

FYI: https://worldofeuc.slack.com/archives/CKHRXATV2/p1688763767838369

The download url doesn't work, someone has a working link or did they pull it back?

Guessing it's just like Neal said, overloaded. I can't even login, getting stuck on SSO. 😛

Citrix should consider moving to The Cloud to help with bursting.

lol 404 on the downloads page

As Matt said above, now I even get a 404 when trying to get CVAD downloads (wanted to make sure if it was just ADC or not), but nope, constant 404 on all.

I was able to get through opening an incognito browser from a device I was previously getting 404, downloading now. Could be just dumb like that attempt got throgh

I just do it via Cloud ADM, works very well

Inprivate/incognito clears the 404 message, but SSO still fails.

yeah probably just dumb luck for me then

Yeah, looking at the text that's definitely a bit harrowing, too. "Exploits of CVE-2023-3519 on unmitigated appliances have been observed."

Yeah, check the thread i linked to. Appearantly the exploit has been on sale for a couple of weeks now.

SMH

Well fuck, seems like ADM is dying now too:

Was finally able to snag the updates too, so some progress at least!

the first upgrade of a 13.0 build completed without issue

Thank you @Neal Dolson! Incognito worked for me too ;)

Got both of my environments running on the new code for the active node. No problems so far

No issues here after upgrading several HA-pairs. We specifically upgraded both active and passive nodes to the latest version because we don't want to suddenly be running an older version in case of failover. "The only way is up"

Smart

I don't know how smart it is, I prefer an env up with a vulnerability (I'll know if there is a failover) rather than all our users not being able to work.

Before upgrading your ADC, it is recommended to perform some checks to identify any modified files. 1) Check your system for the presence of suspicious files/webshells. Adjust the value of the "-newermt" parameter depending on the installation date of your system: find /netscaler/ns_gui/ -type f -name *.php -newermt 20230501 -exec ls -l {} \; find /var/vpn/ -type f -newermt 20230501 -exec ls -l {} \; find /var/netscaler/logon/ -type f -newermt 20230501 -exec ls -l {} \; find /var/python/ -type f -newermt 20230501 -exec ls -l {} \; 2) Check the HTTP error logs for irregularities that may indicate exploitation of a vulnerability: grep '.sh' /var/log/httperror.log* grep '.php' /var/log/httperror.log* 3) Check shell logs for unusual commands: grep '/flash/nsconfig/keys' /var/log/sh.log* 4) Check for suspicious files with the setuid bit: find /var -perm -4000 -user root -not -path "/var/nslog/*" -newermt 20230501 -exec ls -l {} \;

just upgraded one instance now, where some routes are missing after upgrade..

In my opinion reverting back to the older version of the firmware should be a conscious decision (downgrade) instead of something that can happen without warning. In case of security vulnerability fixes, if it's just some bug fixes I'll spread out the node upgrades out to 24h. We have had this happen with a previous build, an upgrade with security fix broke our keytab authentication. The customer was really unhappy and wanted to revert to a previous unsecure version but in the end security prevailed.

Hey @Rink Spies thanks for those commands, curious where you came across them? Whoever compiled them obviously has some intimate knowledge of the method/IOCs

@Neal Dolson NCSC Advisory states that the absence of indicators above does not rule out abuse but is still worth checking. Unfortunately, I don't have any additional information. Hopefully, Citrix will provide additional information in the near future on how to verify the above-mentioned."

ah OK, thanks for that

I've had issues on 2 HA Pairs now, where the secondary node comes up without routes etc.

@Rink Spies are you sure about user root ? 🤔

13.0 HA pair upgraded, so far no issues encountered with that one

For those with route issues, is that 13.0 or 13.1?

hmm..VPN seems to have issues..have not yet trshoot enough to give more info..seems that sessions get connected but nothing works 🙂 Anyone else?

13.1 @HilaryClegg Routes were missing on secondary after upgrade, and it seems a lot of config with it was missing. Added routes, rebooted the instance and gave it time to sync again, and everything synced successfully.

My Number 1 reason for missing config was always expired licenses + reboot.

back to VPN issue: have to do with routes...since I "fixed" it with enabling one PBR and then disable it back and then all good..weird

On 13.0 it seems the load with RefWebUi fail to load the app on mobile phone, Android, so far it happens on Chrome, Firefox on 2 devices.

has anyone found signs of compromise? either positive or false positive?

One reason I have seen missing config besides licensing issues was related to the VMWare hardware version of a VPX being very old on new ESX version. (talking 5.x (10) or older machine version).

It seems that SSL_NOSYNC (https://support.citrix.com/article/CTX136173/how-to-create-unique-ssl-certificates-for[…]address-nsip-when-appliances-are-in-high-availability-setup) no longer works on 13.0 HA. Does anyone have a workaround?

13.1-49.13 Firmware all done on a HA pair no issues. I did run some of the IOC commands before the upgrade and found a few things when running find /netscaler/ns_gui/ -type f -name *.php -newermt 20230515 -exec ls -l {} \; anyone else find anything ?

I heard there was a blog article with IOC info. Anyone know it?

@Sebastian Parelius Thanks for confirming 13.1. Im not seeing any routing table issues with 13.0 91.13. @c4rm0 I did see a false positive when trying to check for signs of compromise. The find php command gave a bunch of results for php files that are part of the firmware. The netscaler I was working on is set to utc time. We started the upgrade at 8pm et which is midnight the next day utc time. Once I adjusted the find command to add two days instead of one, the false positives were eliminated.

Technical analysis of the patch https://attackerkb.com/topics/si09VNJhHh/cve-2023-3519

why does the 'exploitability' say very low?

I have no idea. It also says "it would require network access to the management port", which I highly doubt since Citrix does not mention this and AAA is used for end users auth. Maybe the analyst is not so familiar with the product?

he might be conflating the other CVEs? the one he’s referenced doesn’t need access to the management port

Earlier in July there was a Twitter thread talking about a possible new zero day involving the mgmt IP, but I dont think anything happened with that one beyond speculation, although the timing with this one being disclosed makes the two seem related somehow.

there’s 3 new ones out currently, two which require access to the nsip

NS13.1 49.13.nc: Today I discovered my first small issue. In my Citrix Director Health Monitor, the question mark in the HTTP Request was gone and so the Health check failed and marked the Load Balancer as offline: