This message was deleted.
# citrix-netscaler
s
This message was deleted.
j
check again, I'm currently installing it
r
OK yep, a quick refresh and it's showing up now.
k
quite the thing again...
c
82.41 got RTMed today so it's probably just taking a minute to propagate through the CDN/etc
m
We just upgraded two HA pairs at different costumers, I didn't have any issues while my college got a console error "ssl3_get_record wrong version number" which resolved by enabling TLS 1.2
s
was your prior version the latest before today? or is this that HA bug that was introduced a few versions back?
d
Another day, another ADC vulnerability πŸ™„
Take Note! It's not just upgrading to fix the SAML issue, you need to also add the relayStateRule as per https://support.citrix.com/article/CTX316577
m
The device with the SSL error was upgraded from NS12.1-60.19.nc, the other one was on 13. Indeed, we also added the relayStateRule.
k
does anyone have sample configuration available if we have set up ADC as a SAML SP and the IdP is Azure AD (with or without ADFS)?
I think I can figure it out myself also, but if someone already did this, it'd be so much easier just to copy paste πŸ˜„
d
Anyone configured the relayStateRule and got it working with the Workspace App? I'm getting the same as @Mads Petersen is reporting on twitter - "RelayState in Response does not match with rule in Action. Please contact your administrator
m
This is what i'm seeing in my debug with using Azure iDP. if i decode the Relaystate from a Browser it will start with: ns_policy=auth_azure_samlhttps://gw.netscaler.dk/ Now if i do it from native workspace app, it will look like: ns_policy=auth_azure_samlwv=1&&https://gw.netscaler.dk/ And even a AAA.LOGIN.RELAYSTATE.CONTAINS("https://gw.netscaler.dk") does not match the workspace
d
Have you started with Citrix Support yet? I will log a ticket in the morning.
m
I have not yet personally, doing my own troubleshooting, but i know others have done it already.
k
@Mads Petersen I have also tried (SP end) with no success. We have configured Azure AD as the IdP, which also "pays a visit" to ADFS for password entry. I've tried the following with no success:
set samlaction <http://saml_srv_gateway.company.com|saml_srv_gateway.company.com> -relaystateRule "AAA.LOGIN.RELAYSTATE.REGEX_MATCH(re#^https://[a-zA-Z0-9]*\.microsoftonline\.com/#)"
... and :
add patset patset_customer_azuread_adfs
bind patset patset_customer_azuread_adfs "<https://login.microsoftonline.com>"
bind patset patset_customer_azuread_adfs "<https://fs.customer.com>"
set samlaction <http://saml_srv_gateway.customer.com|saml_srv_gateway.customer.com> -relaystateRule AAA.LOGIN.RELAYSTATE.CONTAINS_ANY("patset_customer_azuread_adfs")
m
I've tried a lot of weird combinations of regex/pattern set. I think the issue is related to how the relay state is different based on browser/workspace app. I have an weird idea that the && which is added in the relay state for workspace app, breaks the encoding. I plan to play around with encoding, but we have Citrix PTEC the next couple of hours, so that will tonight
k
oh, got it working
set samlaction saml_srv_gateway.customer.com -relaystateRule "AAA.LOGIN.RELAYSTATE.REGEX_MATCH(re#^https://[a-zA-Z0-9]*\.customer\.com/#)"
so the entry in the rule needs to match the SP URL, not the IdP side
j
I've got a handful of 12.0's out there, completely missed the EOL last Oct. I'm diving into change logs/notes as we speak, but any 12.0 -> 12.1 gotchas come to mind immediately?
k
@Jon Bucud, perhaps a separate thread on this one?
πŸ‘ 1
m
@Kari Ruissalo (WyW) does it work with native workspace app? I assume you're testing with a webbrowser, because yes if you're SAML SP, it will be the URL that the client enters, not the iDP. I've tried this regex also AAA.LOGIN.RELAYSTATE.REGEX_MATCH(re#^https://[a-zA-Z0-9]*\.customer\.com/#) and workspace app still breaks
k
@Mads Petersen, I'm getting this error on WSApp too, it's working in browser
m
Yeah, that is the issue i'm reporting. I've created a support case on the workspace app issue, so i'm working with Citrix on it now. the problem is related to how the relaystate is, there is a difference from a browser and the workspace app
k
thanks @Mads Petersen, please share what you'll find. I don't love this.
I can't remove the relaystate rule from GUI nor from CLI
only way I've found to get rid of it is to unbind the saml policy, remove the policy/action and add them back
m
That issue i haven't seen, and i've added/removed the rule a lot of times today
k
yeah... just need to do an ADM job to clear that one out
did you try the pattern set approach @Mads Petersen? ... with that you can define either ASCII or UTF_8 charset. One approach would be to capture traffic with browser and wsapp login and try to analyze from there what's the difference.
frankly, I'm exhausted and not that eager to go down that road at least today πŸ˜„
m
I've already capture the traffic and i know what the difference is, the problem is that the SP FQDN is shown in both relay states, so a simple contains fqdn.tld/ "should" work. I might do a little more troubleshooting tonight, but it's been meetings back to back all day, so i'm not sure if i want to do more today. I've provided citrix with captures and information, so i might just wait to see what they get back with
πŸ‘ 3
k
oh ok, thanks for sharing
I hope they'll update the article regarding this as well
s
thanks for all the information, will keep an eye on this thread.
m
Anyone see Henry's post about 82.41 breaking XenMobile? https://twitter.com/hereshenry/status/1402593293561704448
m
Yes i've seen that one, but i also seen he fixed it with a traffic policy
k
One more thing of the CVEs; are these raised in well known forums like nist.gov, or similar (at least the Finnish security officials haven't raised this as an issue yet)? I'm also concerned of how high the severity on these is?
m
The DoS is rated as medium, it will requre you have layer 2 access, and some more stuff i can't say here because of NDA - i wouldn't schedule an emergancy upgrade because of this one The SAML is rated a high, but it does require a phishing attack to happen beforehand, so the user would need to click on a link or something, to visit an endpoint at the attacker beforehand, so it's not an easy exploit. So even with a way to steal a valid user session, it does require other bad stuff to happen beforehand
πŸ‘ 1
πŸ‘ 1
d
Reply for follow πŸ˜…
k
thank you @Mads Petersen!
j
We had also several deployements for our cuustomers were we updated and applied the change. In some situation some could logon and some could not. We currently made a copy of the action (as we could not change the existing one)
d
Just had customer who got the following message after updating HA cluster (Version 12.1) "Unable to establish connection with the secondary. Command propagation failed". Then I remembered that he automatically turned on Secure HA (TCP 3008). After I checked on the internal service that the certificate is bound correctly, I activated TLS 1.2. Voila and the sync runs again.
πŸ‘ 1
j
We did some test with saml tracer in Firefox (fyi published apps were started via a different portalsite, Workspace365) , (the relay state is encoded in base64) and found out that in our case it contained the following info
"ns_policy=your_saml_action_name<nul>/Citrix/StoreWeb/Workspace365AppLauncher.html?appname=AppName"
so this did not contain the hostname but something different. so we need to expand our expression probably with something like
|| AAA.LOGIN.RELAYSTATE.CONTAINS("/Citrix/StoreWeb/Workspace365AppLauncher.html")
We have to schedule maintenance with the customer to test this. When I know more I will post it here.
Unfortunately this didn't work. Did also tried a regex re#^.*$# which should match everything. But it didn't work. It did work if we connect direct to the gateway url. But in cases it was launched from another page it didn't 😐
We only had a very short time window to test this case
Also noticed that when it "works" and that session is expired and you refresh the page, it wil trigger a new saml logon and then the page wil also receive the message "RelayState in Response does not match with rule in Action. Please contact your administrator" message
k
We're eagerly waiting for a solution for the Workspace App side. I'm also a bit afraid that if we get the Windows WSApp working, we'll face issue with OS X, Android, iOS WSApps... no news on the support case yet @Mads Petersen?
m
All i can share is expect good news soon
k
We patched one production environment (12.1-60.19 -> 12.1-62.23) on Wednesday. Yesterday everything worked just fine, but today we started getting an error on Workspace App (see screen capture). We haven't implemented a relaystaterule for this environment due to WSApp -problems. As a precaution I always update just one node and wait for a couple of days before upgrading the other node, so I just HA failovered to the node that wasn't yet patched and everything started working again. The error message is in German and we're located in Finland. I have no idea why the error message is in German, but Google translate results in "demand can not be completed". So my guess is, that this originates from StoreFront and is "Request cannot be completed"
d
Yes smells like StoreFront. What is in the Event Log on your StoreFront Server ?
k
I'm logging in just now to see
... this will cause a world-wide shitstorm, if SF can't request new certs from the FAS after the update.
πŸ‘ 1
d
Have also converted several customers with SAML. Without Rule of course. They are all still running, without any probs
k
are you running FAS?
no errors in StoreFront logs... my FAS user cert has been issued today
(I have one week validity on the certs)
j
Is your FAS server certificate still valid, could be a coincidence? Also check your FAS eventlogs (Application)
k
I did check the logs, but there was nothing
It wasn't just my user that had the issue... I'm just about to delete my user cert from another similar environment and then see if it causes similar issue
j
Also we removed the relaystate expression for some of our customers. And they are functioning without issues. Waiting on more info currently
k
@John Billekens, did you find a way to remove the relaystaterule expression without recreating the saml actions?
j
no, temp create a new action (click the excisting one and select add, it will use the same settings) just change the name and remove the expression. Click add. Then in policy change to the new action
πŸ‘ 1
when the right fix is there you can change accordingly and change the policy back to the original action
and remove the temp action
k
that's what I did too... not optimal...
not related to expired FAS cert, at least in another env
so it's likely an environment specific issue
this new issue doesn't seem to relate to the firmware upgrade either, as it's now happening with the non-patched box too
πŸ‘ 1
either workspace app reset or then recreating my SAML action did the trick
it was about the SAML action recreate
πŸ‘ 1
originally it didn't have relaystaterule, but when the problem started occurring, I changed the rule and even that I failovered to the old version the setting somehow stuck there even if it was not visible in GUI nor CLI
... however, when the issue begun, there was no relaystate rule defined, so it couldn't cause the original issue
j
Via the vendor of Workspace365 portal we got the following expression and this seems to work in our case.
AAA.LOGIN.RELAYSTATE.EQ("<https://portal.domain.com/>") || (AAA.LOGIN.RELAYSTATE.LENGTH.EQ(0) && HTTP.REQ.HOSTNAME.EQ("<http://portal.domain.com|portal.domain.com>"))
not sure yet if it works in all cases, but it was an improvement.
k
can't try this today anymore with WSApp, but looking forward to see how it works, thank you!
πŸ‘ 1
j
Np, as said earlier, I have not yet had time to test it thoroughly but wanted to share it.
k
I bet someone will test this in the US timezone today and report back πŸ™‚
j
Confirmed by Citrix engineers that this adjustment is secure against the CVE-2020-8300
m
Firmware out that fixed the CWA bug. β€’ The authentication from Citrix Workspace app fails when Citrix ADC is configured with SAML authentication and relayStateRule. The browser based login is not impacted. β€’ [ NSAUTH-10517 ] Was not expecting it today, but tomorrow πŸ™‚
And i can confirm it works πŸ™‚ If you haven't already seen it on my twitter, atleast on firmware 13.0-82.42, i dont mess with 12.1 anymore. πŸ™‚
d
But I still enough. The latest 12.1 version also fixes the error. πŸ˜‚πŸ˜‡
πŸ’― 3
k
what kind of expression did you end up using?
j
The new version solves the issue with CWA, but it's still recommended by Citrix support to use this expression (n our case):
AAA.LOGIN.RELAYSTATE.EQ("<https://portal.domain.com/>") || (AAA.LOGIN.RELAYSTATE.LENGTH.EQ(0) && HTTP.REQ.HOSTNAME.EQ("<http://portal.domain.com|portal.domain.com>"))
m
I'm using a regex match rule
AAA.LOGIN.RELAYSTATE.REGEX_MATCH(re#^https://[a-zA-Z0-9]*\.domain\.tld/#)
k
thx guys, I'll check with support myself too. Just for laughs if we get a third one πŸ˜„
πŸ‘ 1
j
Fyi if you are using a aaa auth vserver for access to cs/lb vserver. Add all fqdn's to your expression for each site, add |`|| AAA.LOGIN.RELAYSTATE.EQ("https://authsite.domain.com/")`
πŸ‘ 1
d
I have a client where multiple users have bookmarked the ADC page after they've logged in, ie the URL is https://gateway.domain.com/citrix/storeweb and if they use this bookmarked URL we are seeing the "RelayState in Response does not match" error. Debug shows the relaystate as "ns_policy=samlaction/citrix/storeweb/", I've tried multiple versions of the Relay State Rule with regex as per Citrix doco, minimal versions eg AAA.LOGIN.RELAYSTATE.CONTAINS("storeweb") and more but unable to come up with something that works. Anyone else seen this and can help me out?
k
We're seeing the same exact issue with web browsers as @Daniel Marsh and the only way around the issue seems to be to either use incognito browser mode or empty the browser cache
d
I've been given another expression to try by a Citrix engineer - have not tested yet, I'll be doing this tonight:
Copy code
AAA.LOGIN.RELAYSTATE.EQ("<https://example.com/>") || (AAA.LOGIN.RELAYSTATE.LENGTH.EQ(0) && HTTP.REQ.HOSTNAME.EQ("<http://example.com|example.com>"))
Just saw this got posted as well πŸ™„ Symptoms or Error After following the steps inΒ CTX316577Β some users see error "Relay State and response does not match with roll in action". Solution Complete the following steps to workaround this issue: 1. Do not use bookmarks, only type the Gateway URL or Load Balanced URL directly 2. Clear cache and cookies https://support.citrix.com/article/CTX319685