seen a couple of these in the past day since the latest update

During boot after upgrading? 13.1 or 14.1?

Not sure what the customer did, but i had to reload the kernel manually - they were doing the 13.1 update

Lovely

I had that happen to a customer on their MPX units. I have taken to heavily recommending customers with mpx/sdx devices to make a clonezilla images of the hdd before it is even configured, just in case they need to recover it back.

One of our SDXes reported a failure during the upgrade, accompanied by a note in the browser saying that it had failed after "an hour and forty minutes" or something similar. (We were only only 20+ minutes into the upgrade.) We were able to restart the upgrade and it ran to completion. We rebooted the other SDX before attempting the upgrade, and that one had no issues. Upgrading on of our VPXes from the SDX failed, I believe due to recurring disk space issues, and we recovered from that. Finally, I upgraded one of our MPXes and it cratered. Upon rerunning the upgrade, the node reported that it was missing its Perl files, so per Citrix article https://support.citrix.com/external/article?articleUrl=CTX582458-when-upgrading-netscaler-getting-error-bash-installns-usrbinperl-bad-interpreter I copied those files back. Now I am getting errors like this: "Can't locate loadable object for module Fcntl in @INC (@INC contains: /usr/local/lib/perl5/site_perl/mach/5.30 /usr/local/lib/perl5/site_perl /usr/local/lib/perl5/5.30/mach /usr/local/lib/perl5/5.30) at /usr/local/lib/perl5/5.30/mach/POSIX.pm line 11." Does anyone have a solution for this? I fear I may have to call support as I've been unable to fix this.

Would re-running the upgrade replace these files?

Hi @Nick Pylarinos, I did try that with various NetScaler versions, but the "missing" Perl (?) wouldn't let me complete. We had let support expire on that NetScaler pair because its datacenter no longer requires NetScalers, and I've asked to decommission the pair to be safe. Right now, the active node is fully updated and I've set STAYPRIMARY and STAYSECONDARY respectively to pin service to the primary.

I had a client that got in that state. We winscped into the working one and copy over the perl directory to the secondary, and that fixed it.

Thank you, @Jeff Riechers, that did the trick! The article I had found (now added above) only mentioned /var/perl5/*bin*, so I had only targeted that folder for repair. Per your comment, I broadened my search to the entire /var/perl5 directory and found many missing files and folders. I copied those missing files and folders over individually not knowing whether or not WinSCP would respect symbolic links. I then ran the 13.1-59.19 upgrade and it completed without issue.