Hi Guys, since a XenServer patch (we could confirm by checking the logs but things got worse overtime), our boot time has degraded significantly to a point that the Controllers shut down the VMs. Our 4 PVS, 2203CU5 (had the same issue with CU2) have been working well so far, we use XS8.2, UEFI with W16. The more VMs we reboot the worst it's getting, up to 14min. 400 VMs to reboot for 4 PVS in 3h, 200VMs per 2 PVS as we have 2 sites (I increased the reboot duration from 2 to 3, no major diff). I've pushed latest patch to XS including the network driver, we use Dell. No major difference. Our target PVS 2203 CU4 (rolling out CU5) doesn't use latest Xentools, we will have to rebuild, it's not so straightforward. I can't see any perf issue om PVS,CPU, memory, vDisk is correctly cached is RAM, not an high disk usage. Network has 2*10GB bonded, all good with Network. 1Gbps usage when I boot VMs, no queue building up. I removed the AV as it has given us trouble in the past. No XS PVS caching. I'm running out of options, any ideas please?

Inside the PVS server, what network bandwidth are you seeing? I assume the PVS servers are XS VMs as well? So the 10Gb NICs are bonded in Dom0?

Thanks. The max I see inside the PVS which are XS VMs is 1Gbps. Yes the 2 * 10Gb NIC are bounded at the XS/dom0 level, LACP used.

I have some questions: • Is XS using the PVS accelerator to cache PVS vdisks ? • How many targets per XS host ? • What do you mean 2203 CU5 does not use the latest Xentools? The images have not been updated with the latest XS tools? • Are you seeing more retries during boot and how are you boting is it an orderly startup or boot storm (all at once)?

Hi Carl , thanks for your help! 1. No it doesn't, never used it. 2. Per XS host, I have 16 VMs/target. The XS host has 128 vCPU (2 AMD pCPUs,768GB of RAM, local SSD for the PVS WC) 3. I mean the XS tools is not up-to-date on the PVS target, it runs 9.0.42 4. Only 2-6 retry during a boot. 5. The reboot is handled by the Controller. 200 VMs for 2 PVS servers to restart over 3h. I don't find the reboot handled correctly (even if the boot time degraded since those patches), it doesn't help. For example, I can reboot 52 VMs under 5 min 21 sec, 16 VMs per host, 3 hosts on the same site so served by 2 PVS servers. Worst boot time is 5min21 reported in Controller (167 sec, 5 retry in the StatusTray). So ultimately it is not so bad, still worst than pre-match but the way Controller handle the reboot and how it shutdown VMs if it doesn't register after 20 min make things much worse.

Can you take a look at a running target and make a note of the amount of data transferred during boot using the PVS tray app? I would suspect that if you divide that by 1Gb/s times the number of PVS servers, the time we be about right. The XS PV Nic driver in the PVS server should be able to drive more than 1Gb but you might need to ensure you have the latest PV drivers installed in the PVS server. It wont be as high as the 20Gb/s available, but it should be > 1Gb/s.

End of the day it was a type of host, Intel CPU that had a bad patch and after updating to latest patches+ drivers for a few hosts, not all, everything went back to normal. So those hosts were most likely disturbing PVS and slowing down everything, snowball effect!

THanks for the update

At the end, the root cause was the wrong AV policy applied due to a bug. Making sure the correct policy with exclusions and no scan fixed the issue.