This message was deleted.

hey @Leee Jeffries kicking into a thread here ....Network looks clean, no retries or anything nasty - something narky is telling me the accelerator could be throwing a spanner in the works. Re Numa, I'm not all that savvy with the best way of carving up XenServer CPU...10 Sockets with 1 core per socket is what they are currently playing on

Defo check networking performance. XenServer always has issues, especially with 10Gb. Anything peaking on the host and XenServer performance tab?

CPU is definitely the concern i think - it is sitting around 80-95% on the VM, and the host is thumping around 70-75% - just have no idea why (flying blind without monitoring)

Aaahhhh okay. As long as the host has headroom you should be good. It's not overprovisioned which is a no no on XenServer from my PoV.

yeah we left plenty of headroom for the narky Dom0 🙂

I'd change to 6vcpu's / VM , 1 socket, 6cpu's per socket in that configuration personally. Preferred NUMA config is 1 NUMA or 0,5 NUMA. I can't see the logic for the 10 socket config unless I'm missing something. In this config you'd be relying on XS to schedule cpu since it is overprovisioned, which makes for higher cpu queue lenghts etc.

hey @Lucas Van den Broeck - the driver for the larger config was GPU based for the most part - customer wanted to have heavier GPU footprints per VM as the utilisations of that GPU would vary across users (some just use office, but others might do a bit more intense work that hits the GPU,) Logic was having a larger GPU profile meant we had more flex in the VM, which of course meant awkward CPU sizing as a result. Initial view was to have a 4Gb GPU profile, with 7 VM's at 6 CPU Spec and sacrifice 4Gb worth of GPU...waste wasn't an option given the price of GPU

What is the result of the top command on XenServer ? Does the CPU is consummed by VM or Xen process ?

Since you didn't specify the bios settings I'm just throwing it out there, might already be ok but -> make sure the profile is on Virtualization - Max performance and not on the default HP settings or on Virtualization - Efficient. Will get way more juice out of the cpu's like that.

@James Kindon are you using the T4-8A profile? I would check the GPU Frame Buffer usage with nvidia-smi to see if your not running out of Framebuffer which will lead to high CPU usage and slow performance.. Are you using the latest CH8.0 hotfixes and did you enable the optimalisations stated in this article: https://support.citrix.com/article/CTX258320

@Julien Destombes I will dig into it tomorrow onsite and see what i can see... @Lucas Van den Broeck Static High Performance my friend, i have been bitten over years with anything other than that

@James Kindon Static High performance will disable the ability to use TurboBoost. I normally leave it to OS controlled so CH can figure out which C-state is the best (and utilize Turbo Boost)

@Rody Kossen i have that hotfix installed and all my VM's were created post - just checked the flags and they all look to be OK. I will check the FrameBuffer tomorrow and see what it looks like - i have asked customer to go and buy RDAnalyzer. Interesting around OS Controller - I had my ass handed to me back in the day with vSphere deployments and OS controlled, have the default stance of setting high and running for the hills but i didn't realise it got rid of turbo boost

@Rody Kossen I am interessed by your bios config. Could you share it ?

Here is a Citrix article about CH and Turbo Boost: https://support.citrix.com/article/CTX200390

that is very interesting @Rody Kossen ...now i have more stuff to go and read through 🙂 thank you

With Dell it is: Performance per Watt(OS) Tobias Kreidl wrote a great article about it: https://www.mycugc.org/blogs/tobias-kreidl/2019/03/07/tale-of-two-servers-bios-settings-affect-apps-gpu

hmm i fail with a "failed to get get cpufreq parameter" when trying to validate

Yes, that means that C-States are disabled and so the Turbo Boost is also disabled

@Rody Kossen just to confirm also, yes i am using that specific GPU profile

With a T4-8A profile you can handle 25-35 users per VM depending on the workload

argh, does that mean that the GPU actually limits how many users you can fit onto a box? Even if they are just doing basic workloads?

Yes, every user claims a certain amount of Framebuffer (depending on Resolution/Monitors & Apps). If you're out of FrameBuffer it will swap it to Memory which is much slower and consumes CPU. You can easily check it with NVIDIA-SMI, it reports the amount of Framebuffer in use. If it is below 90% than you're fine

toying with that now - does rdanalyzer show this too?

Yes RDAnalyzer will show you those metrics

For HP Servers the BIOS settings should be “OS Control Mode”, Turbo Mode should be enabled. You can check if Turbo Mode works with: “xenpm get-cpufreq-para | fgrep turbo” on the Xen cli. You have 24 Cores, so let’s assume you can over-commit your physical cores with 1.5 ratio. So, 36 cores (without Hyperthreading!) gives you 6 VM’s with 6 vCPU’s. Why 6? NUMA or half of NUMA! Now here is the problem: You can’t configure the 8A profile anymore, because you have too many VM’s. Check what the frame buffer shows in relation to the CPU with GPUProfiler. https://github.com/JeremyMain/GPUProfiler/releases

@Rody Kossen That is incorrect, Turbo is not tied to C-states. C-States is core idle/parking. Turbo is tied to the P-state. C-state should always be disabled on a hypervisor.

@Joe Citrix states otherwise: Using all the available C-states gives the processor most thermal headroom to enter Turbo mode. Hence the default configuration that uses all available C-states should give the best performance for processors supporting a Turbo mode AFAIK: disabling C- states breaks P-states as well for CH. But this can also be a Dell BIOS thing that if you disable c- states the P- states are not controllable via OS

How does disabling C-state break P-states? The are different power-saving technologies. When a core is a state other than C0 then it’s not executing. During P-state the core is executing.

I understand the technologies behind them. I'm only saying that I had issues in the past with Dell hardware and bios settings. But as said, Citrix states that C- states should be enabled for Citrix hypervisor.

I’m not sure which Citrix article you’re referring to but for a general workload or a Citrix workload on a hypervisor C-State and P-states should be disabled. Sure, you don’t get turbo, but you get all of you cores running 100% and you don’t incur and performance penalties of the c-state or p-states. Turbo works great for a select number of workloads that need value clock speed over core-count.

This one https://support.citrix.com/article/CTX200390 ( I mentioned that article earlier in this thread)

And unless Citrix changed the CPU scheduler in XenServer then turbo isn’t going to give you what you expect.

I've seen good results with turbo mode. See also an article from Tobias:https://www.mycugc.org/blogs/tobias-kreidl/2019/03/07/tale-of-two-servers-bios-settings-affect-apps-gpu//www.mycugc.org/blogs/tobias-kreidl/2019/03/07/tale-of-two-servers-bios-settings-affect-apps-gpu But I understand that experiences can be different but this is what I saw

I just read that Citrix article and it’s riddled with inaccuracies.

I‘m glad that there is a discussion going on about this topic. I NEVER really figured out what’s the best BIOS configuration for Xen. Maybe now, so please go on... 🤓

Joe, do you have any articles that maybe represent your experiences and your recommendations?

I enabled them and enabled turbo mode. My CH experience is 90% CAD related where clock speed matters.

So it really depends, potentially on use case and hypervisor? So coming back to James use case? As hes using shared sessions?

@Rody Kossen Tobias’ article is synthetic benchmark that can immediately benefit from a high clock speed. He’s also running VMs specs that are less than hardware which allows for cores to be parked in favor of clock-speeds.

True, that's why I also said that my experience is with cad. So not many VMs per host and utilising mainly 1 core.

@Leee Jeffries Yes, use case makes a huge difference. CVAD/XenApp is a thread monster so I would value high core count at 100% vs lower core count at 130%.

Love the convo guys, very cool to understand the drivers behind your decisions. I'd love to see some testing score to compare the settings in the specific use cases you've mentioned.

But @Joe if all cores are utilized it shouldn't matter if you enable c- states as it will remain in the highest state.

You would think, but no. It doesn’t work that way. Cores still get parked and you still incur the latency waiting for them to wake up.

Thanks for the awesome input, lots and lots to go and play with 🙂

James, one thing we've found on XS / CH with large VMs and vGPUs that has benefited us is to pin your VMs vCPUs to the underlying pCPUs. Doing this imposes some limitations on the environment - you can no longer migrate VMs between hosts, for instance. The other thing we do is make sure that all VM pinning fits within NUMA boundaries for our processor architecture. Lastly, we pin DOM0 CPUs as well. Our current processors are 2x12, Intel, with HT, so a total of 48 "cores." We put 4 x 10 vCPU VMs per host, pin each vCPU to pCPU (within NUMA boundaries, it's a pain), then pin the remaining 2 pCPU "cores" on each set to Dom0. Doing synthetic testing with LoginVSI netted us ~20% increase in performance doing that.

@Tim Riegler that’s my exact spec at the moment - will definitely look at your model as we have no shared storage and simply carve up the host.. looks like chrome is causing havoc and fslogix backed storage fell over too. What’s you cores to socket config on your 10 vcpu vm?

We do 10 sockets, 1 core each, 58 GB RAM (static, not dynamic), Grid K2s @ K260Q (so, 4 VMs / host, and 2 Grid K2s / host). We also pin the Dom0 vCPUs to the "remaining" pCPUs from our 10 cpu count (remember, 12 cores per socket + HT). So 2 Dom0 CPUs pinned, then 10 VM CPUs pinned x 4. This has netted us the best performance overall. For Chrome / FSLogix, we redirect some things to local C:\ (so RAM cache), which seems to help. We also limit page file (4GB in our case, although when we enable WEM memory optimization we increase that to 20GB, on local SSD). No shared storage anywhere - everything is to local SSD on the host, or RAM cache. No PVS accelerator, either - we don't find it useful for only 4 VMs / host. Also, we use ControlUp to monitor the PVS RAM cache, and ended up increasing ours to 12 GB. If you are seeing slowness, make sure to check that and your page file - you may be paging like crazy.

as a follow up to this - things have settled down a little with a chrome update of all things, we are still working through a load of little things here and there but once again chrome is just plain nasty in all sorts of unexpected ways. Next steps are to get onto 8.1 release as 8.0 isn't wonderful, and look at pinning as per @Tim Riegler guidance - many thanks for the awesome input and ideas from everyone

We force a few extensions to make life a bit easier, and use WEM's processor / RAM controls to keep it in line. Specifically, The Great Suspender extension is fantastic. There are definitely times when Chrome wants to rule the world, though. WEM to the rescue for those!