Has anyone seen a NIC just go completely broken in...
# citrix-vad
m
Has anyone seen a NIC just go completely broken inside a guest on xen? Out of 2000 W11 25H2 VDIs maybe like 5-10 will go belly up every day I can add a new NIC and that one is good, but the original NIC is completely unresponsive This freezes up - PS C:\it> Get-NetAdapter "Ethernet 2"|Restart-NetAdapter This also freezes up - PS C:\it> pnputil /remove-device "XENVIF\VEN_XS0002&DEV_NET\0" Microsoft PnP Utility Restart the VDI, fresh one comes back fine Ticket open with citrix but I'm not holding out a lot of hope
s
Haven't ran into this. Latest xenserver OS patches? Is your host firmware all up to date on the NICs/BIOS/etc..? Any paticular host you are noticing these freezes on? Any recent xenmotions happening? VM/Host resources look good at the time of freeze? Storage? Dom0 have enough resources?
m
latest os, tried the latest xen tools and the last version, firmware is up to date, random hosts, no xenmotions, 1 TB RAM per 50 guests and only partially in use, storage is a Pure X20R4 SAN and everything else is happy with sub 1ms latency. No issues on any other guests when it happens, and if I restart the VDI its totally fne.
s
these MCS/PVS VDAs? Guessing you combed through event logs to see if anything stands out?
m
MCS
Nothing interesting in event logs at all
its gotta be a crash somehow of the nic's individual xennet.sys driver but why and how to prevent it .. really not sure
s
this just started happening? Windows Patches? ANy security agent updates?
m
timing is a bit unclear as I have been battling a differnet issue, fixed that, found out folks were confusing this issue with another one. still poking at a box I found. removed everything third party live and its still busted, but that may not be a reliable tevhnique
also considering it could be npcap 1.60 crashing it, may have to change the gold image, not sure if anyone has seen that
s
everything else on the VM works as far as opening local applications and all that, it's just the NIC and network items that don't work?
m
yep just the single nic. the new nic I add is fine
existing nic says its plugged in and up but no arp
s
can you xenmotion it?
you can't just xenmotion the vm (compute/memory)? Storage would stay the same
m
forgot I have functional pools now. yeah no change
s
xentools isn't accidentally kicking off an auto update or anything? guessing citrix is going to make you kick off some dumps when this happens again
m
yeah nothing in the logs about that. there's a debug ps1 for xen tools and got the logs off to citrix but they're ... slow it happens at least 5-10 times a day so not hard to reproduce trying the 1.88 version of npcap to see if it helps. there is a known issue with 1.60 and vmxnet3 which isn't xennet but its something
s
can't temporarily remove it all together to prove it out? Does it matter if a user actually logs into the machine as well? or does it happen regardless if it's being used or not?
no one is doing anythign funky with GPOs and nic settings? had that happen once where some network keys were getting replaced in the registry which caused issues...not sure if you can take a handful of machines and put them in an OU with blocked inheritance
m
I've found empty ones that are busted. No, I build those. A specific workflow requires pcapping certain data from the nic to an app for compliance/legal reasons, so .. down the slow rabbit hole
s
i bumped into a similar problem, but for us its vsphere., and the issue was VMXNET driver appears to be corrupted. we switched our nic to E1000 to and it worked. perhaps reinstall Xen equivalent of the vmware tools, guest driver.
s
@msaks - whether the VDIs having issue switch to APIPA address?
m
We have/had something similiar on vSphere.
m
It was npcap breaking the nic sporadically, npcap 1.60 came with calabrio smart desktop connect, but there's a flag to not install it, which is fine if you don't do call recording at the client level.