Our Network team migrated to a new Palo Alo firewa...
# citrix-netscalerc
c4rm0
08/30/2024, 4:25 PMOur Network team migrated to a new Palo Alo firewall last week due to the old Palo Alto FW being EOS (Existing FW config and NAT Rules went across ) and since that change (Issues started next day) our external users to the Netscaler Gateway are experiencing connection drop out issues where the HDX connection is dropping 5-10 times daily which means session reliability kicks in (Session interrupted box appears) . It then waits up to 5 mins (Session reliability timeout) and then reconnects back to the suspended session . I see the Session reliabily event in Citrix Cloud Director for the affected User and the VDA the user was connected to logs the session reliability event as well. The users all appear to be using different ISP's and are from different parts of the country and its only external users who use the Netscaler Gateway who are impacted. Connection Speed using a internet speed test and packet loss for the affected users seems ok and they dont have any uploads/downloads running and its both wired and wireless connections impacted. The Users on the inside via MPLS/LAN who connect to storefront directly and launch Apps are fine due to the direct HDX connection between endpoint with CWA and the VDA. I have gave the ADC (MPX 5905) a check over and generated a tech support bundle and uploaded to CIS and no issues reported. I also checked the performance metrics/ logs ns.log and newnslog /Network interfaces and LACP channels and Rate limit packet drops and no issues found. The ADC has been on the firmware 13.1.53.24 without issue for 2-3 weeks before this issue started. Anyone come across issues with Palo alto firewalls and HDX connection drops ? anything i can get the firewall team to check ? anything i can check ADC side apart from packet captures ?
r
Rob Zylowski
08/30/2024, 4:28 PMCould they have enabled packet inspection during the changeover
c
c4rm0
08/30/2024, 4:29 PMI did ask them if packet inspection is on they told me SSL inspection is off
r
Rob Zylowski
08/30/2024, 4:30 PMI have mainly seen that issue when either the palo firmware is old or the firewalls are overwhelmed at times
Rob Zylowski
08/30/2024, 4:30 PMI assume neither woudl be true of your new deployment
c
c4rm0
08/30/2024, 4:37 PMi would presume that is the case but will get them to check the PAN OS version and performance metrics
c4rm0
09/05/2024, 4:57 PMso the palo alto guys came back and from looking at the packet capture from the firewall beleive its because the client is connecting on udp 443 which is failing because its not open on the firewall and its then failing back to tcp 443 which is then causing the problem. I explained to the palo alto guy that is normal behaviour with Citrix adaptive transport if the external firewall doesnt have udp443 open the client will try on udp443 and if it fails it will fall back to tcp443. The weird thing is the old palo alto firewall didnt have udp 443 open so would of been using tcp443 and didnt have issues. As a test they have opened udp 443 on the firewall for the source ip of an affected user and so far the connection hasnt dropped once (I confirmed in Director and ctxsession its deff using EDT/UDP). As a test for myself i disabled Adaptive transport from the default of preferred on a test DG which would force TCP but i still had the connection drop outs. I also set the reg key HKLM\SOFTWARE\Policies\Citrix\ICA Client\Engine\Lockdown Profiles\All Regions\Lockdown\Network\UDT\"HDXoverUDP"="Off" which again forces TCP but again i had the connection drop outs
c4rm0
09/05/2024, 5:00 PMI also tested the Netscaler GW directly by connecting to the NS GW VIP in DMZ from a windows server in the same subnet (Bypassing firewall) by using windows host file to point to the DMZ IP of the NS GW and it was rock stable and didnt get one connection drop and that was using TCP
c4rm0
09/05/2024, 5:08 PMThey also confirmed that Packet inspection and the PAN OS is same as the old FW