https://worldofeuc.com logo
Join Slack
Powered by
What is your procedure for recovering the NetScale...
# citrix-netscaler
a
What is your procedure for recovering the NetScaler VPX configuration from a backup to a clean VM when the original VM is untrusted and its native backup mechanism doesn't support restoring onto a different instance? // published before in wrong channel
👀 1
🙌 1
r
Right now, it's a stand up a "blank" Netscaler, then tell my MCP server to restore the config. It works magically.
👀 1
You can also ssh up the ns.conf to /nsconfig folder, DON'T SAVE, and reboot. That trick used to work.
r
Copy the ns.conf and reboot. Replace all certificates and any credentials (local and anything like LDAP, SAML, etc.) Assume all stored credentials and private keys etc. were compromised and replace them. Don't forget to revoke the old certificates! Even if you've replaced them, the old certificate key pair can be used to impersonate your website if it is not revoked.
💯 3
m
I think there's a new way of doing this with preboot now isn't there?
if you're on cloud or kvm you could do it, not sure if vmware has a way yet
r
Would be interesting as it takes the IP address to copy it in and when you add the IP address it reboots. Not sure why this is necessary though. Isolate the IP (close the firewall port) before rebuild should permit you enough time to login as nsroot/nsroot and copy in the ns.conf. Reset the nsroot password and reboot.
m
I've always wondered if you could crack open the vmdk from the ova file and dump a full nsconf in there and have it boot with all the settings.
never tried it because trying to write to a bsd formatted disk on a windows machine is almost impossible
r
I'm sure you could mount the VMDK to another VM and do that, but why?
m
to see if you could really
probably not any faster than any other method
r
15.1 it would probably be easier. I'd be concerned the temp VM would stamp something on the disk.
m
15.1 will have the xml cloud init stuff
r
Didn't think of that
m
it'll be way easier
r
Haven't messed with it yet. Lab time!
m
I aven't tried cloud init yet but it looks like the way to go
r
I use cloud-init all the time on my lab Linux machines.
🙌 1
m
archy.net/deploying-netscaler-vpx-15-1-on-proxmox-what-actually-worked lol looks like it doesn't work for proxmox at least, but cracking open the disk and putting nsconf in there does
r
Good reading
r
Stephane is still around? Love that guy.
❤️ 1
✅ 1
m
you can preconfigure the nsip and then point nitro at it
that way you can automate the whole thing
and not even have to set the nsip in the wizard before nitro can talk to it
t
techpreview no thank you for production
m
sorry for the hijack Arthur but Excellent reading on the 15.1 stuff, I do have one spun up but really haven't played with it, and have not touched cloud-init. Thanks @Mike Streetz (O_P) @Richard Faulkner
h
@Thomas Poppelgaard TP seems to be better than current production 🤣
d
this is timely. I have a customer that we think may have been partially compromised so I'm not taking any chances - new IPs, new everything, manually copying over ns.conf items otherwise 😉
t
what do you mean partially compromised
h
A little bit hacked not totally
😜🤷🏻‍♂️
t
🤣 its not enough just building a new ns
you need to do a tonz
• Kill sessions on the compromised box right away (on both HA nodes and any GSLB or peer appliances): 
kill aaa session -all
, 
kill icaconnection -all
, 
kill rdp connection -all
, 
kill pcoipConnection -all
and 
clear lb persistentSessions
. • Isolate it from the internet but keep it running, or snapshot it, so evidence survives. • Collect evidence: 
show techsupport
, 
/var/log
, 
/var/nslog
, 
/var/netscaler/logon
, 
/netscaler/ns_gui
, 
/var/vpn
, 
/var/python
, crontabs, 
ns.conf
and a memory or core dump. • Hunt for persistence: webshells (
.php
, 
.xml
, 
.js
in web or logon paths), modified or setuid binaries, new cron jobs, unknown users or SSH keys, odd processes, outbound connections. • Rebuild from a fresh, fixed firmware build rather than upgrading in place. • Review the config line by line before restoring it, never as a blind restore. • Clear the backend sessions: ◦ Log off VDA sessions in Studio or with 
Stop-BrokerSession
. ◦ Reset StoreFront by restarting IIS or recycling the app pools. ◦ Revoke Entra ID refresh tokens. ◦ Kill RDP and VPN sessions on the backend hosts. • Rotate everything stored on the NetScaler: nsroot, local accounts, LDAP and RADIUS bind accounts, shared secrets, and SAML certificates. • Reissue and revoke the SSL certificates and private keys. • Reset passwords for users who authenticated through the NetScaler during the exposure window. • Widen the reset to admins, service accounts or all users only if you find evidence of credential harvesting or AD compromise. • Hunt for lateral movement in AD and internal logs from the SNIP or NSIP, and on StoreFront, DDCs and the backend hosts. • Make sure the management interface (NSIP, GUI, SSH) isn't exposed to the internet. • Forward logs to a SIEM. • Monitor for logons with old credentials and for failed-logon spikes. • Report the incident to the national cyber security authority or CSIRT in your country (for example CISA, NCSC, CFCS or BSI), and notify your IR firm, cyber insurer and legal. Under NIS2 in the EU, the early warning is due within 24 hours.
to be safe reset ALL PWD in entire company
that one is something companies dont like but you dont want a bad actor in your system i am speaking about experience here 2023 helped tonz it happend rebuild ns and then some forgot to reset serviceaccount password and hacker was in again🙈
h
🌊
t
involve forensic team (3party)
i am going to release 1.15 soon with a lot of new things i will be applying ioc by stand without switch as i see people think it does ioc by standard
also people should be aware of hyperscaler azure and aws you cant provisioning any netscaler current if you been compromised and need to build new netscaler. Netscaler been removed from azure, aws 🫪
r
Isolating the box will end all backend sessions if you do it right. Rotate EVERYTHING. Only the ns.conf is re-usable and even that you should review to be sure nothing was added etc. Resetting passwords should not be as hard now as it was a couple years ago. Everyone should be rotating their service account passwords regularly anyway (Like LDAP etc.).
Yes, the hyperscalers now you must create a vm and import the disk.
h
It's still hard, almost had a fight around changing user passwords, everything else was fine even old legacy svc accounts that people said hmm don’t know if it will survive this
r
LDAP service accounts etc. I don't think I'd even use the same account. Replace username and password.
t
if things are compromised then they are in…. so its crazy not resetting all users pwd
you would be surprised what i have seen services accounts for ldap on ns that is domain admin🙉
h
Jup
d
Well, yeah. I've already sent that instruction to them to do so.
🙏 1
r
But let's remember the CVEs had compromises where they could get in and they had DOS capabilities which didn't allow any access. Compromise is the concern. If your SAML crashed your NetScaler and it rebooted it doesn't mean any data was lost.
t
revoke certificate is also something many dont like doing
d
but I'm having them do that anyway
h
Seen enough of misconfiguration as well service account privileges, wildcards etc it's a fire pit
r
Most commercial certificates when you re-issue automatically revoke the old certificate so that is mostly for the internal certs.
t
hehe never forget 3 y ago some i worked with had to revoke 40 certificates they didnt like it
😵‍💫
d
it's a bit strange - the IoC says theres a suspected perl fingerprint but I can't find any other indicators.
h
True have a revoke about 72 hours or so with digicert
d
fortunately this one is really simple - two gateway URLs and a Storefront. Done.
t
nis2 in europe is also crazy u have to report within 24 hours to cyber in your country
d
I tell you what tho - this has exposed a lot of bad and assumed practices. Customer thought they were blocking every IP outside the US (Not that that's safe in and of itself) for one. The other was realizing that patching thru Console is fine but the scripted actions weren't being done in a lot of 'em.
r
The biggest item to me has been Enable IP Reputation. So many of these would have been a non-issue because the TAs were known and blocked already.
I even had a few customers who were using an external WAF and these IPs were still getting in. Not sure if their external was dropping as they should have been, but since everyone is Platinum Edition now, turn it on!
👍 1
t
requires platinum license
yeah you said it :)
what do you mean everyone is?
not everyone is
i seen env where customer only buy enterprise ver
r
No such thing now. Everyone event the express freeware is Platinum. UHMC, CSPO, and Platform all are. Where else are you getting a license>
t
there is also other licenses
r
LAS doesn't permit anything else. At least not any I have found.
t
private cloud lic
but you are right richard most should have the feature
d
agreed - IP Reputation is something I thought was in the WAF for Gateway so there's egg on my face for that as well. I think it was prior to the license update to platinum or something and it didn't get activated? Problem is I don't really live and breathe this stuff
t
ip reputation should come up and be enabled by default or big red flags :)
it would help a lot richard you got a good point here
maybe citrix can fix this💪🏻
✅ 1
r
I am going to let them have a piece of my mind on their handling of this CVE situation. To say I am displeased is an understatement.
s
for IP rep, in scenarios where your netscalers don't have any egress, aka citrix connections allowed in/out, but traffic originating from netscaler can't get out, is ther a recommended solution here? Standup a proxy? Would be nice if the ip rep could use the netscaler console/agent to handle DB updates.
n
At Citrix Connect Copenhagen yesterday they said they thought that the team had handled it well, I did chuckle to myself a fair bit but I couldn't be arsed to push.. haha.
😀 1
d
I mean - it could have been a lot worse...
🥲 1
m
I am with you on the IP Reputation bit @DJ Eshelman. I'll be adding that on our side as a standard......still catching up with the licensing changes.
r
@Steve Noel you can configure them to use a proxy server to download the Webroot database. In fact, that is probably more secure.
s
yeah, think that's the only option for now, would be nice to have it handled by Netscaler console, but hey, least there is an option
👍 1
r
I had said that as well. Everyone is connected to Console for licensing, let's push the BOT updates and Webroot database that way.
💯 2
Open in Slack
PreviousNext
Powered by