This message was deleted.

I'll be updating a secondary datacenter instance later this morning (VPX) will let you know if I run into that as well.

Updated ours here last night. No issue encountered.

Updated 4 pairs last night 1/2 way, 2 stage ADM. No issues here. I assume you tried the typical clear cache or bypass cache?

You are able to login to the CLI?

yes, i can get to the CLI and the admin gui. end users getting nothing

custom themes?

Yes but it’s never been an issue in the past. And we’re using fas and saml with azure auth

I assume IDP initiated flow still works and that's how users are getting in now? Anything jump out in Browser devmode -> Network when attempting to load the page? FWIW, I'm also SAML+FAS, but Okta as IdP; RfWebui based custom theme. No issues on yesterday's 13.0 build

Uncaught TypeError: this._cleanUp is not a function

at Object._handleAuthenticationError (ctxs.webui.min.js209262) at Object.c [as error] (jquery.min.js214498) at Object.complete (ctxs.core.min.js22165) at m (jquery.min.js117159) at Object.fireWith (jquery.min.js118387) at c (jquery.min.js192449) at XMLHttpRequest.<anonymous> (jquery.min.js202172)

Nothing in that error specifically, but I'd wager it's in one of your nFactor schemas or custom themes (more likely the schema IMO) based on the spinning wheel / dev mode screenshot hung on doAuthentication.do. Since you're SAML, are you doing something like a no_authn EULA first, SAML, Group extraction? If so, are they getting the EULA? I know here, that "page" is given to me since it has branding...and that has caused breakage

Also make sure your certs are still present, especially the Azure cert that does not show up in cert bindings but shows up on the auth server.

i've seen cert binds drop on upgrade, great suggestion on checking that saml action/server

cert bind drops are often due to problems with ha file sync... you've installed the certificate on to the primary box and when you upgrade and do a ha failover, the certificate vanishes. Running this command prior to upgrade can be helpful or reveal problems

force ha sync

Now both nodes are exhibiting the behavior. one patched, one not patched. I suspect this is due to me forcing failovers and a bad config overwriting a good one. now to figure out how to restore a previous nsconf

I assume you've made a backup before you start the upgrade?

We backup every 72 hours via ADM

Then you can try to restore the upgraded node back to its origin state before upgrade. After that you can check the behavior over that reverted node. If that works fine try to force a HA sync again. After normal behavior you can try to upgrade one node again and check the working of it.

thanks. ADM is a little confusing. i have two nodes in this HA pair and ADM shows backup files for each node (IP) with the same timestamps. not sure if it matters which one i try to restore from

Important is to take the backup file closest to the timestamp of the start of the upgrade. Also the backup file needs to be matched of course with the node.

Thanks so much to you guys who replied! The upgrade reverted our configuration for the login schema. Obviously they think their out of box config is better than your own.

time for beer!

Are you using a WAF on the gateway? Ferroque posted a blog about it being a new signature with the recent firmware and it caused connection issues. It was 13.1 in their scenario, but I thought it might be worth mentioning. https://www.ferroquesystems.com/resource/issue-citrix-workspace-app-html5-client-fails-after-netscaler-13-1-b51-15-upgrade/ Edit: added link

No WAF