World of EUC

If you ever find this, please let the rest of us know. Easily the worst part about Studio.

Import-Module .\Citrix.GroupPolicy.Commands.psm1
New-PSDrive -PSProvider CitrixGroupPolicy -Name LocalFarmGPO -Root \ -Controller localhost
Set-CtxGroupPolicy “policy” computer -Priority “1"

<https://github.com/ryancbutler/Citrix/blob/master/XenDesktop/GPO/Citrix.GroupPolicy.Commands.psm1>

you could also do -user instead of -computer, I’m not sure it matters

this was on a 1912 ltsr farm too, so that legacy module still works

Gonna create a blank policy tomorrow just to basque in the glory of moving it to the top in under 12 minutes.

lol, it took me more than 12 minutes to figure out how to do it, but now that I know…

Plz like and subscribe! <https://twitter.com/O_P/status/1435745794812891139?s=20>

I remember having to do this at a former employer that had a Studio policy for every location they had that mapped the printers based on client ip address.  They had 700+ locations...

yeah, it’s never fun, I’m amazed it’s taken me this long to work out how to do it in powershell.

This has genuinely made my week. Had the same scenario as <@UVBEF2GQG> above and was re-evaluating my career choice as i clicked...waited, clicked...waited.

I’m sure someone like <@ULJM9DQF2> can wrap the whole thing in GUI with powershell and have it show you every policy and priority number and let you edit it from a table, but that’s beyond my powershell skills at the moment

It is something I’d love to see though, and happy to help test it if someone does create it

Great news James ! Thanks for the share ! I hope you are fine !

Yeah, had this issue this week with 1912LTSR
`Start-Process -FilePath "C:\Temp\CitrixWorkspaceApp.exe" -ArgumentList "/includeSSON /silent" -Wait`
It never return, it's on my to-do list.

This is working as a work around at the moment:
```$Installer = Get-ChildItem -Path $Path -Filter "CitrixWorkspaceApp*.exe" | `
    Select-Object -First 1 -ExpandProperty "FullName"
$FileName = [System.IO.Path]::GetFileNameWithoutExtension($Installer)
$params = @{
    FilePath     = $Installer
    ArgumentList = "/noreboot /silent /includeSSON /ENABLE_SSON=Yes /AutoUpdateCheck=Disabled EnableCEIP=false EnableTracing=false ADDLOCAL=ReceiverInside,ICA_Client,SSON,AM,SELFSERVICE,Vd3d,Webhelper"
    Wait         = $False
    NoNewWindow  = $True
}
Start-Process @params
Do {
    Write-Host "Waiting for process [$FileName] to complete."
    Start-Sleep -Seconds 10
    $Running = Get-Process | Where-Object { $_.ProcessName -eq $FileName }
} While ($Null -ne $Running)```

Here is what I do:

`$Options = @(`
        `"/silent"`
        `"/FORCE_LAA=1"`
        `"/AutoUpdateCheck=disabled"`
        `"/EnableCEIP=false"`
        `"/ALLOWADDSTORE=A"`
        `"/includeSSON"`
        `)`
        
`$inst = Start-Process -FilePath "$PSScriptRootRoot\CitrixWorkspaceApp.exe" -ArgumentList $Options -PassThru -ErrorAction Stop`
        `if($inst -ne $null)`
        `{Wait-Process -InputObject $inst}`

I used <https://www.bujarra.com/citrix-mobile-receiver-setup-url-generator/?lang=en> with <citrixreceiver://createprofile> for sending configured iOS receiver app accounts to my users. It looks like it's not working with iOS workspace app, anymore. Workspace App is starting, but nothing happens. Any infos about syntax <citrixreceiver://createprofile> is deprecated since the switch from receiver to workspace?

I have done it the same way, applied to Citrix top OU where all the VDAs reside and enforced the GPO. No issues, works as expected. 

citrix has never supported in-place OS upgrades

<https://support.citrix.com/article/CTX212452|https://support.citrix.com/article/CTX212452>

I'm not sure how I did not know (or check) that. But that's what a lab is for right? :slightly_smiling_face: Will be building a new box, but went through troubleshooting anyway. Uninstalled SF, reinstall wouldn't work at all until after I added .net 3.5 to the server (?) and then worked through the last process here to get past an error and get a backup config imported. <https://discussions.citrix.com/topic/386394-export-storefront-configuration/>

I've seen this error on one endpoint where wfshell.exe crashes after session launch and I was not able to resolve it.

I can see on RDSH servers few times per day. 

Citrix Support demanded procdumps with page heap enabled but I was not able to collect these. 

image.png

Truthfully we find that its still just number of cores that matter the apps will always use more power as it’s available. Our rough rule of thumb is 5 vdi or 10 sessions per core but that is for a task worker and of course your milage will vary. I personally think your old environment is the best way to know. Calculate sessions or desktops per core and use that with the new number of cores. 

Our current XS hosts have two 16-core CPUs, and right now memory is the big limitation. We have capacity for 62 W10 VMs per host as is.

Still curious to hear how others are going about this, though.

So that sounds about right for the 5 per core minus a gew since you ste using 3 not 2 vcpu

the amount of ghz is also important, you don’t want to go for a slow CPU even if it has a ton of cores

We're looking at Intel 8368s, so 72 cores per blade. Cisco UCS B200 M6's.

Using that screenshot I posted as a guide, it looks like maxing out each blade would yield 142 VMs.

I think our goal is going to be closer to 85%, but still, that would allow us to stay at 6 total chassis' in the new hardware.

I can't believe I didn't ask this sooner, but does the rule of 5 and 10 still apply? It's a dated article, but if that's even remotely accurate, this hardware would be more than capable of handling 142 VMs per blade.

Did you read my posts?  Yes but that always is just an estimate. Again what you see now is more representative. 

Actually, I just lied - I think I missed that post somehow, lol. It has been a long day.

What kind of apps do your users run? I think our internal CRM app is very resource intensive (Web app in Chrome) and they have Gmail, Slack, and Jabber. A 4x12 VM is the lowest we could go to get acceptable performance from users, and we were asked to give them 8 cores (Not needed - cpu ready what!?).  We're running a proc I don't care for right now - Xeon Gold 5220 (18 core 2.2GHz) and I while the core count is ok, I'd rather have more GHz for VDI. We're about to switch to a 6226R due to supply chain issues, and it's a 16 core 2.9Ghz, so we'll see if that makes much of a difference. I also want to try some of the higher core count procs to see about better density and scaling up instead of out.

We’re lucky in that the biggest workload is going to be Office 365 and Edge. A lot of these users access our EMR from within that desktop, but it’s hosted in XenApp, so I really don’t have to worry too much there.

FAS GPO is apployed to FAS server as well confirmed in registry

managed to get it working as its got a proxy set for system account via winhttp netsh winhttp i had to enable bypass proxy for local addresses and it now works

May I ask what the use case is for host to client redirection? why you are using this instead of BCR? - like to understand before I (try) to help you get it to work.... haven't done a lot with host to client redirection myself

Customer runs a rather heavy graphical program and the application contains hyperlinks. To spare some server-side compute resources, they'd like to open these hyperlinks on their local clients rather than on the server. Modern browsers tend to eat a lot of RAM

Exactly, good thought.. let me check docs and so on.. see if I can get something setup to test this.

Search the comments at <https://kolbi.cz/blog/2017/11/10/setdefaultbrowser-set-the-default-browser-per-user-on-windows-10-and-server-2016-build-1607/> for ServerFTA.

Thanks <@UM531NUFJ>, I added the following on to the registry, but it still doesn't seem to work:
`Windows Registry Editor Version 5.00`

`[HKEY_LOCAL_MACHINE\SOFTWARE\Clients\StartMenuInternet\ServerFTA]`
`@="ServerFTA"`

`[HKEY_LOCAL_MACHINE\SOFTWARE\Clients\StartMenuInternet\ServerFTA\Capabilities]`
`"ApplicationDescription"="Citrix ServerFTA"`
`"ApplicationIcon"="C:\\Program Files (x86)\\Citrix\\system32\\iexplore.exe,0"`
`"ApplicationName"="ServerFTA"`

`[HKEY_LOCAL_MACHINE\SOFTWARE\Clients\StartMenuInternet\ServerFTA\Capabilities\Startmenu]`
`"StartMenuInternet"="ServerFTA"`

`[HKEY_LOCAL_MACHINE\SOFTWARE\Clients\StartMenuInternet\ServerFTA\Capabilities\FileAssociations]`
`".htm"="ServerFTAHTML"`
`".html"="ServerFTAHTML"`

`[HKEY_LOCAL_MACHINE\SOFTWARE\Clients\StartMenuInternet\ServerFTA\Capabilities\URLAssociations]`
`"http"="ServerFTAHTML"`
`"https"="ServerFTAHTML"`

`[HKEY_LOCAL_MACHINE\SOFTWARE\Clients\StartMenuInternet\ServerFTA\shell]`

`[HKEY_LOCAL_MACHINE\SOFTWARE\Clients\StartMenuInternet\ServerFTA\shell\open]`

`[HKEY_LOCAL_MACHINE\SOFTWARE\Clients\StartMenuInternet\ServerFTA\shell\open\command]`
`@="\"C:\\Program Files (x86)\\Citrix\\system32\\iexplore.exe\""`

.. did it on a running PVS provisioned worker though, not sure if the settings need to be configured in the golden image?

No niin... just finished testing.. did enable the policy, created the XML file, connected that to the GPO setting and powered on a multi-session win10 desktop. (don't have ServerOS running but will test that as well)
worked as expected _(hiding the fact that initially I tested on a single user OS and it didn't work_ :disappointed:  _damn, I need to learn to read_ :wink: _)_

let's connect directly.. you can logon to my environment and test as well.
I didn't set any registry settings in the VDA.. got asked where to open it... picked chrome and it opened at the client side

Oh. Ok, need to check. I already "left the office" for today, but I'll check this tomorrow.

let's connect.. I don't have 1912 running, no 212R2. but perhaps someone has internally..

brief update on the public side as well; the config works on 2019 based VDA, but not on 2012R2

The GPO and XML file basically take care of the registration of ServerFTA to open http and https:// links. <@ULMMMRQA2> told me you do get the prompt when clicking a URL, so that part seems to be working fine. You also need to enable the Citrix polciy "Host to client redirection" and make sure it applies to your user on those 2012R2 boxes.

I can confirm it does work, I'm doing the same. A customer has Navision that contains links to documents stored on sharepoint. Because the endpoint is always signed in and configured for Office 365, it made more sense to open those links on the endpoint. There are still running 7.15 LTSR.

So, trying to remove 2 delivery controllers from studio as i'm trying to decom them

i'm a full admin, and i get "you are not authorized to perform this action"

well, running it from a delivery controller seems to have done the trick

Will this help?
<https://social.technet.microsoft.com/Forums/windows/en-US/e2632b6e-76b9-4640-85b9-698fb55199d8/cprogramdatamicrosoftcryptorsamachinekeys-is-filling-my-disk-space|https://social.technet.microsoft.com/Forums/windows/en-US/e2632b6e-76b9-4640-85b9-698fb5519[…]microsoftcryptorsamachinekeys-is-filling-my-disk-space>

We discovered a similar problem in the user profile but in the locallow folder.
In our case it is OneDrive that is causing this issue.
Scenario:
user1 logs on and does not configure OneDrive
user2 logs on and configures OneDrive --&gt; no issues
user3 logs on and configures OneDrive --&gt; lsass.exe creates a folder in c:\users\*user2*\appdata\locallow\microsoft\crypto\tokenbindingkeys\microsoft\crypto\rsa\*&lt;SID user3&gt;*
We are working with MS support on this.

Onedrive is installed on the CVAD Farm as well, but only enabled for a small group of users. Procmon shows that several processes are accessing the RSA Files, like MSOIDSVC.EXE, MsMpEng.exe and lsass.exe. digging more into it right now.

Solved it by updating to UPM 1912 CU3, as the following issue was present in CU2:

 Folders created before enabling *Delete excluded files or folders* through the *Logon Exclusion Check* policy but excluded by the *Exclusion list – directories* or *Enable Default Exclusion list – directories* policy might not be deleted. [CVADHELP-16439]

<https://docs.citrix.com/en-us/profile-management/1912-ltsr/whats-new/cumulative-update-3/fixed-issues.html>

Could be the FastReconnect issue:
<https://www.mycugc.org/blogs/brandon-mitchell1/2019/09/22/not-so-fast-reconnect>

it didn't helped.
Happens also with VDA version 2106.

Any other ideas?

We had to deactivate echo cancellation on the terminal servers -&gt; problem solved

In the end the problem was that the local citrix workspace crashed.
On the local PCs we use the newest citrix workspace app 2108. After some research we found out, that it wos a bug that was already fixed in Citrix Workspace App 1912 CU3 - see attachment

Afterwards we configured follwoing registry key on the terminalservers:

• 64-bit: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Citrix\ICA Client\Engine\Configuration\Advanced\Modules\ClientAudio
• Value name: EchoCancellation
• Value type: String
Value data: False

Hmm.....what a weird issue, good catch though :slightly_smiling_face:

<@U02E1R7JK55> I'm not following, You modified the reg value on your RDSH servers even if the reg value is related to the local Workspace App client?

I created the reg value on the RDSH servers.
<https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/policies/reference/hdx-registry-settings.html>

mhmh now after a few days without problems the local citrix workspace app crashes again. now we try to install the ltsr version

We just has an update session on HDX+SDWAN, I’ll see if I can find a good product team contact. I personally have done WANOpt for sat links in prod and it did help a lot, but that's long before we introduced some of our fancier tricks. If you do have multiple links you may be better off using packet dupe on standard these days. 

Yeah that's what i'm trying to glean.  In some cases they do have multiple links, but afaik they're all in the rural bits

EDT/UDP uses more bandwidth. If the link is low speed as well, that might have adverse effects.

as i understand bandwidth isnt a huge pipe but not tiny/ has sdwan underneath.  Its just latency and loss.  Waiting to make contact with customer  so I can get access to all their gear

I've used SD-WAN Standard and Premium (WAN-OP) in the past when I still worked for a customer (I work for Citrix now).
But my view on it is that WAN-OP is only good if you want to squeeze every last bit out of your connection, especially if there are other traffic flows like SMB / email etc, as you can cache quite much of that with WAN-OP. So if 2 users access the same file it only get's send over once. For pure HDX, the bandwidth savings are much less.

For high loss it's a bit of a different story, I've seen quite some good results in the past with SD-WAN but that was mostly because there were multiple links where only one of them showed packetloss (and SD-WAN moved the flow over to another link). But in the end, SD-WAN should resend the packet if it has been lost. As it can see the separate ICA Channels (like Audio or keyboard/mouse) it could do packet duplication on it for example.

I would recommend to test both EDT with Loss Tolerance as well as SD-WAN. SD-WAN is really preferred if you have more than 1 WAN link and you are able to do packet duplication on the different ICA Channels.

Thanks all for the contribution.  BTW I didn't hear back from Muhammed so still keen to know from the HDX engineering team what the current state of play is

<@ULJMUKVMK> I'm part of the HDX engineering team as well:blush:

looks like through the installation wizard I can click OK on the error message and the installation goes on but through command line, it just fails.

Not a hope. A really old app? Have you already tried to move it to 64-bit? What's the concurrency for usage?

a really really old app... requires MS Host Integration Server 2006, which I can't get to install the 32bit version either.

If I could get HIS 2006 x86 client to install, but it keeps installing the x64 version. So my only other option is to possibly go back to Win7.

64bit i know works, never tried 32bit server 2008R2

Probably your easiest path is Win7 32-bit. How does your app integrate with HIS? Can you just copy the registry and folders from 64-bit install to the 32-bit locations, or vice versa? Sometime you can make it work this way.

Could you use Win10ltsc2019-32bit? I had success with 1912ltsr and some old 16-bit apps. 

No, user OSTs should NOT fill up the write cache. Anything that's in the container shouldn't hit the write cache, as it's not writing to OS Drive.

Cheers <@ULJMUV893>. That was my understanding also. User must be getting a local profile so. 

give this a launch when the user is logged in, to confirm.  that's usually my goto first step.   If the stop light color is not 'green', then there is an issue.

<https://support.citrix.com/article/CTX232495>

Hi. We have had the same issue with a customer. Not with Outlook though, but with another software. We've disabled these for the customer about a year ago and it hasn't had any user impact at all.

Thanks <@U01KDDAJDRR> I've had the settings turned off since I posted this last week and no complaints so far.  What's really interesting is that I have a dev environment running the same OS, same VDA version, same Outlook version and same Citrix policy settings and I could not reproduce the issue in that environment.

The only impact it has is that when using mobile devices (such as Ipad's etc), the keyboard will not automatically appear when selecting a text field. You would have to manually start it. But apart from that there is no difference, but it could be a bummer if you have users who access Citrix through touchscreen devices.

Yes, it is a weird issue. I've had two different applications not work when this policy is enabled.

Yeah I suppose its a bit of a pain to press the keyboard button every time on a touch device but luckily I don't think we have any users using such devices, at least not regularly.

I was going to say that it didn't seem to make sense to have to destroy and recreate the MC, but the article  does explain it, because it records the storage location in the hypervisor connector, so that kinda makes sense.  they should probably make a way to do the migration through studio, which would update that info

Right, my thought is that it's not exactly an unusual scenario. But the process seems overly complicated for persistent machines.

I did exactly this and created a new MC and imported the migrated persistent machines by manual. This should be supported, too, as a redeployment makes no sense for the users :man-shrugging::skin-tone-2:

I done this before as well. There is even a script I seen to do this somewhere in slack.

Oh here it is
<https://ninaronline.co.uk/2021/09/17/how-to-migrate-xendesktop-vdas-between-catalogs-on-the-fly-with-powershell/>

You diamond <@ULLH5JGFR>. That's outstanding. We opened a support case with Citrix, to get their official word on it (not something we've done before), but this looks like it will do the job. There's no reason why it wouldn't work with Cloud, right?

LOL, Not postive about cloud man. Its a new area for me and i am learning the do's and donts.

Ah, it even mentions Cloud Connectors. Sweet!

Ahh nice, I havent read it all the way yet. I just seen it and said, OH COW. I need this. Bookmark

<https://stevenlemonier.fr/how-to-create-an-mcs-catalog-in-powershell/>

Thanks <@U010KKUE4HH> I have done some of this before. I guesss I am wondering if there are any issues around that? Is it that Standard SSD is just not included in GUI or is there a reason for that.

I have no experience with Citrix in Azure, sorry Dima :disappointed:

Pretty sure that was an enhancement so you won’t have it in LTSR - 99% sure - can check to confirm 

<@UKJTJD4KA> no doubt. that being said would creating this manually create any issues? BTW, your article and code about changing the identity disks inspired this

So confirmed it’s cloud only, ltsr won’t do it. You won’t be able to change them as MCS uses on demand provisioning - you won’t have anything to change when machines are off :grimacing:

but in the prov scheme there is a place for disk type

Just read your first post again - you already knew the ltsr bit :dizzy_face:‍:dizzy:

You can try it I guess, I’m not sure the code is up to date to recognise that value but you can try

CustomProperties             : &lt;CustomProperties xmlns="<http://schemas.citrix.com/2014/xd/machinecreation>"
                               xmlns:xsi="<http://www.w3.org/2001/XMLSchema-instance>"&gt;&lt;Property xsi:type="StringProperty"
                               Name="StorageAccountType" Value="Standard_LRS" /&gt;&lt;Property xsi:type="StringProperty"
                               Name="UseManagedDisks" Value="true" /&gt;&lt;Property xsi:type="StringProperty"
                               Name="ManagedDisksPreview" Value="false" /&gt;&lt;Property xsi:type="StringProperty"
                               Name="LicenseType" Value="Windows_Server" /&gt;&lt;/CustomProperties&gt;

In this situation, I would try another web cam as as a test. This will help me determine if something in the virtual channel changed in the verison.

Then as a test, try to force it as generic and see the outcome as well.

Yeah, I have suggested that they test another webcam, however they are a bit reluctant as it works in their old WS2016 image, so they can't see why it shouldn't work in the new WS2019 image.

As far as I can see there is at least one change in the virtual channel, the Optimized webcam mode feature which no longer compresses the H.264 content, but instead sends it directly to the webcam. I have tried disabling that feature using the mentioned article by Citrix, but it didn't change anything

I thought I read somewhere that 4k webcams aren't supported, can they change it to only work as a 1080p cam?

<@ULLAD12P2> if you can find any official statement from Citrix, please send it my way

<https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/devices/webcams.html>

you could try messing with the high def registry keys to see if it will go higher, but only zoom supports 4k webcams anyway to my knowledge

<https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/policies/reference/hdx-registry-settings.html>

Like even if you look here right down the bottom Teams says it's only going to use the camera in 1080p mode... <https://www.microsoft.com/en-us/microsoft-teams/across-devices/devices/product/logitech-brio-4k-webcam/37>

I instructed my customer to configure the webcam to a 1080p resolution on the laptop and then test again

I was thinking about going 4k for meetings and virtual conferences but support is so limited I ended up just doing 1080p with a dslr and a better lens

even 60fps 1080p is too much for most platforms

I was just thinking...the webcam doesn't present us with any issues when connecting to the old WS2016 image with VDA 1912

something did change in the way it's doing webcam redirection after 1912, I had a bunch of stuff break on server 2019 and could never work it out

like webcam redirection to webex just completely broke in VDA 2012

The other thing you could look at is a plugin for whatever app they are using, like Teams has HDX redirection, Webex has a windows plugin that handles redirecting the webcam, I think Zoom has one too

We are still investigating, but at the moment, nothing out of the ordinary is found in the event logs

Is the same happening when the camera is connected to the laptop vs the dock?

They have other webcams, no issues with those

maybe the channel is not big enough for such resolution?

According to Citrix there is a 1080p limit on the webcam resolution in the session

I don't know if the other webcams with no issue are 4k though

I have this exact same webcam. Never had problems running it but i run with server 2016 / 1912 LTSR CU3.

Might be a shot in the dark (I think citrix automatically downscales resolution higher than 1080) but in Logitech Brio software you can adjust what  resolution to use for the webcam. Maybe try setting it to 1080 to see if it makes a difference?

Good point <@U01KDDAJDRR> I'll suggest that to my customer

Maybe this is helpful (Multimedia section):
HDX features managed through the registry | Reference (<http://citrix.com|citrix.com><https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/policies/reference/hdx-registry-settings.html|)>

Thanks <@UMB4XQE79> I did find those and we might look at them at some point

I tried it when it first came out.  It's pretty basic...If I recall, it can only do a test launch once per hour I think? We use Scoutbees from ControlUp now so I really have no need for Citrix's implementation.

It is a shame to propose such a weak solution and it is still called Premium licensing. It could be really good feature but Citrix as you can see is not interested. I'm disappointed :pensive: 

Report crash to Citrix Support - they will try to fix in 2010 as it seems to be software bug. 

Hello Arthur, thanks for the response. Fix in 2010?

I have no idea what kind of bug do you have. Procedure is very clear: report to Citrix, they will collect check if fix already existing or will create new one. Without reporting to Citrix maybe this bug will stay forever. :man-shrugging::skin-tone-2:

Adding more CPU to existing VM may imply CPU co-stop and CPU ready depending on the load on your hypervisors.
Nothing to fear for RAM as long as you have sufficient memory available on your hypervisors.

I would add more VM instead (specially if you use MCS/PVS)

Thank you Steven so you recommend me to keep 4vCPU and additional VMs?

I’m with Steven, would much rather keep the VM config as they are and expand the VM count.

Would be good to enable WEM CPU optimizations. It will improve user experience 

Yep. Keep 4vCPU and add more VMs!
And use WEM as Arthur recommends

Included in Advanced and Premium licensing 

what does director say for average ram and CPU consumption? and max. If you have plenty of headroom already, no point scaling up, scale out instead. For 2019 published desktops I've not come across too many places where the density gets above 15 users per VM no matter how much cpu/ram you throw at it

<@ULLAD12P2> This has been my experience as well.  If you have the headroom, make your life easier and scale out.  It's just easier.

My searching has me thinking the answer is no, but there isn't a convenient list out there that shows a breakdown of everything enabled at each VDA functional level is there?

That blog link is a 404 for me, but the docs site for Storefront has a smartcard auth section that should help: <https://docs.citrix.com/en-us/storefront/1912-ltsr/configure-authentication-and-delegation/configure-smart-card.html>

<@ULPS6HFAT> I just updated the link. 
I read this doc 3 times and did not find what is wrong in the configuration 

I would think if you followed that article, you would at least see the browser trying to hit storefront and something in the storefront logs, are you certain your stas are setup correctly and the session profile is pointed to the proper storefront url?  also if there is more than one certificate on the smart card, is the proper certificate with the smart card logon key usage being selected?  have you looked at the netscaler debug logs to see if auth is successful at the netscaler?

Yes. I ran the VDA cleanup util on it last night. Didn't make a difference, so I deleted it last night, and going to try to reprovision it again.

Does the hostname on the provisioned vda look correct?  I had a similar issue recently.  After cleaning up the VDA, I reinstalled using the iso and my problems went away.  <https://support.citrix.com/article/CTX131430>

im not sure that viable anymore but reinstalling from iso resolved my issue

Huh, interesting. If it's still happening after provisioning, I'll give this ago. Thank you!