World of EUC

Yes, All the time when it is running as a published app or running in a remote MMC.

image.png

Anybody have any tricks to only blocked HDX Webcams? Not Generic USB but the Optimized channel.

I can follow this, but it will shun the mic as well.
How to Configure Default Device Access Behavior of Workspace app for Windows (<http://citrix.com|citrix.com><https://support.citrix.com/article/CTX133565/how-to-configure-default-device-access-behavior-of-workspace-app-for-windows|)>

IMG_0604.jpg

Are you creating a new catalog or are you updating an existing one?

Did you follow this <https://www.citrix.com/blogs/2021/01/27/citrix-tips-citrix-app-layering-and-nvidia-grid/amp/|https://www.citrix.com/blogs/2021/01/27/citrix-tips-citrix-app-layering-and-nvidia-grid/amp/>

unless XenServer is different, you will have to add your GPU after provisioning - at least that's how it works with vSphere, I've only done PVS with vGPU in XenServer

they had an existing XS Pool with GPUs, new hardware has been deployed, new Pool created (because server and GPU hardware is different), attempting to deploy VDAs into this new Pool.

I havent done this with xen server maybe you need a new catalog. Im off this week but ill ask the engineers next week. 

Otherwise, if vsphere: <https://www.logitblog.com/change-vmware-vgpu-settings-powershell/|https://www.logitblog.com/change-vmware-vgpu-settings-powershell/>

Screen Shot 2022-10-31 at 1.02.51 PM.png

You say its new hosts.  Does the hosting configuration have vGPU configured?

Hmm you did the first two things I would have done. Let me think on it a bit.

Another tool that may help is this.
<https://support.citrix.com/article/CTX222592/citrix-application-experience-cax-troubleshooting-tool>

i've seen some weird shit where I made a bat file that would launch the printers control panel module first, then the app, and it launches.

how did you figure out you needed printers cp?

i'm sure ANY control panel module would have worked, that was just one the client was already using, so..

just trying to give you some "hail mary" ideas

I guess a procmon may help, Although it like looking for a needle in a hay stack

Long shot....
published CMD on the Same server that the free app is on. Launch the CMD, then call explorer via cmd line. The navigate to the free app via cli and open it from commad line. Just curious if something is timing out

dang, I suppose it looking for taskbar to be loaded behind the scenes?

yea, so wonder if there is way to load it via bat

<https://www.epa.gov/water-research/storm-water-management-model-swmm>

I feel for you.  I really do.  :slightly_smiling_face:

i always find myself trying to get all these different "Craplications" working

One thing I learned is not to say the word "Citrix" it just breaks there brains. Say Terminal Server lol

can alleviate some of these weird published app crapulation issues

Would that address issues with missing start menu or taskbar?

you can give it a shot and see if you can get it to run isolated - might help point you in the right direction

or run a procmon trace and see what it's bitching about when you launch?

yea procmon is next. but I will see if we can get it app-vd. Not sure why I did not think of it. getting old :disappointed:  Thank you for this suggestion.

Yea I didn’t even think of app v either. Community power

Anything Is worth a shot in the app land 

Ppl on their github recommended older ver. That one worked. Sounds like latest one added icon to taskbar and thats what caused the issue.

It’s “being investigated” is all I can share on that one - sorry <@ULJM98FME> I completely forgot to reply :man-facepalming:

I figured it’s just a matter of time.  Unless they want VMWare to be the only partner

Hmmm that’s very strange. Studio and connection strings point to the new name?

Yeah everything is pointing to the new name and SQL server

What’s interesting is that before the move everything was pointing to the single name, but apparently it’s been still writing to the 3 databases still.

Figured this out, apparently we have to change it studio in addition to the powershell commands 

We ran a script to update all the DB connections through powershell 

Studio was still pointing to the old server, I guess it was an oversight 

I see, sorry I thought you did that already. I literally just wrote a blog on this. I was going to send it but I’m finalizing things

We need to move all our SQL DBs before we can upgrade to 2203 on the backend, have to double hop with new Server 2019 DDCs 

Adding ie4uinit.exe to LogoffCheckSysModules helps, but this is a new behaviour

Finalize.ps1

Here's mine from my last place. W10, non-persistent, running Defender for AV. Image built in App Layering.

No, they won’t get the new version until they reconnect 

Hi <@UU04UTPUN> - Seen this? <https://worldofeuc.slack.com/archives/CK6TCV3JN/p1663877818539989>


<https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/1912-ltsr/whats-new/cumulative-update-6/fixed-issues.html|https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/1912-ltsr/whats-new/cumulative-update-6/fixed-issues.html>

We received user complaints for a similar issue with access (WS2016, VDA 2203 CU1) seems to be random, case has not been yet escalated to my team

Very weird issue.......there was the virtual channel allow policy  enabled  i think in 2109 onwards maybe it is messing up the clipboard virtual channel. I would set it to disabled and test

This is interesting. I have not had any reports of this from clients, but myself just yesterday in Excel (Office version 16.0.15726.20174) had a popup that the clipboard was not available when cutting a cell to paste into a new location. I basically ignored it as a one off issue. VDA 2203, Windows Server 2019, Seamless Apps, Workspace App 22.10.2.21 (2210), FSLogix 2.9.828.50276, Virtual Channel Policy blocking configuration disabled.

Will be keeping eyes and ears open for more reports of this.

Thanks, just not sure what else to do here.  I can replicate it on demand too.  <@ULM6WD9LY> that setting is disabled.  Not sure I'll get anywhere with support on this one without a lot of back and forth.

I thought I was going nuts, I’ve been having clipboard issues too, like having to copy the same thing multiple times before it would paste, I thought it was just my machine

Using MCS?  The template/master image is used for the hypervisor properties - disk, gpu etc. But new accounts will not inherit anything from the master image in terms of AD properties - attributes, security groups or anything else.

If you’re applying things based on computer security groups you’re going to have to automate that yourself or do it based on OU instead 

I don’t know of any limitations around instance types for mcsio - but hey, weirder things have happened - did he have a specific instance type mentioned?

It seems the write cache disk is only visible inside explorer (with a drive letter) with instance that have a temporary disk (ex: D2dsv4 or D4dasV4)

Strange - the write cache was always a separate disk in azure which would could be set to persist or not persist until recently. Are they playing with “Use non-persistent write-back cache disk” settings? Maybe a new release has broken a flow somewhere 

Just use the web versions on RDSH :shrug: big problem if you have add-ins tho

Yeah that's the issue. Plus that's a total process change for users.

Obviously this isn't immediate, 3 years until 2025, and Server 2022 is 2026. I was just asked the question today and hadn't seen they were dropping it.

MS are very keen to remove RDSH as it's one of the (two?) Windows Server features that require a GUI. They want Windows 10/11 for client and Server Core for server, long-term, I guess. Whether that means they will have to allow MU Windows to run from on-premises or hosted tin, remains to be seen

could be write cache drive filling up.  You're doing 'Ram with overflow to disk'?

There is no monitoring, so it stays an assumption but at boot there is 40GB free space at the overflow disk

and you can't get to the d$ drive when it's hung?  are you persisting your eventlogs?

Logs are persisting indeed on D, have not tried administrative shares but services,eventvwr and powershell are not connecting

after you reboot the VM do the eventlogs indicate anything?

No, nothing. On the machine that is still going since this morning, the has not been overflowing to the vhdx

you have proper AV exclusions on the VDAs and on the PVS servers?

so it's happening to multiple separate images?

Yes, automation rolls out the images, so tried two complete different installations

do the VDAs seem to freeze at the same time, or roughly the same time?

No, it is at random, some more times per day, and today 1 runs without problems

i've used folderchangesview to continue to export all c writes to an excel file on the d drive, that way you go back and take a look.  Not 100% its an overflow issue, could also be maybe something scheduled or automatic windows updates kicking off, or vmtools geting upgraded or something like that

would think that stuff would show up in event logs

ur using BISF on the seal, t persist all event logs, are you just doing application/system?

That was also what we thought. Due to the start after db issues, we are going to restore databases from backups. Will let you know tomorrow

you have offline support enabled? PVS DB issues shouldn't affect the VM performance after it's already up, but maybe the stream service is freaking out, would think that woudl show in eventlogs too tho

i agree that the clock still moving, but everything else unresponsive feels like an i/o pause (clock runs in memory).  i/o pause for pvs...is either streaming disk is no longer streaming...or can't do writes (wc disk full).

how long do they take to go from boot to unresponsive?

Seen the same thing twice before.  Once was sccm discovering the box and trying to update filling wc.  The other was a scan kicked off on all the vms, which killed io, but that was shown on pvs servers CPU usage being pegged

There was a good thread that included some WC filling troubleshooting info in it not too long ago.  Let me see if I can find it.

<https://worldofeuc.slack.com/archives/CK6TCV3JN/p1662139403761449>

It appeared to be Windows Defender who hung up the servers. Exclusions are in place..

good find, care to share how you came to the conclusion?

<@ULJMUV893> , credits to my co-worker. He saw high disk latency, and first reaction is disable anti virus haha

Morning. Anyone running Citrix FAS on-prem with the Netskope agent on the MCS master? it's causing issues for us with SSO apps to 365. Also, we have the option selected for in-session FAS/smartcard cert and no kind of cert appears in the local user cert store? TIA

We stopped 2209 after the policies were randomly not applying. I think they have a mitigation now, but the security risk of that was BAD. Like external users can copy and paste / drive redirect like half the time

yea i reverted 2206 because of that issue but have been testing the fix and so far its working properly. I can also see the policies are applying to my session at the moment.

<https://support.citrix.com/article/CTX472865/cvad-version-2206-2209-gpss-fail-to-apply>

the fix

doesnt Studio policy override GPO when it comes to disconnect/idle settings? Thought that changed with the latest CVAD version.  not sure if you have any conflicting policies.

hmm now that you mentioned it, i do recall something about that in patch notes.

of course I cannot find anything about that now lol

<https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/policies/reference/ica-policy-settings/session-limits-policy-settings.html>  Take a look at the "note:" at the bottom

Interesting on this - Disconnected session timer – Multi-session ( I assume this is newer ?) I alwasy did it via GPO, but this is nice add on.

only doing the controller side upgrades this weekend.......keeping the desktop agent at 1912 cu1 to start and tackling that next

<https://worldofeuc.slack.com/archives/CK6TCV3JN/p1654035681089039>

I would read through that and just make sure.

As to your other question, you can jump to 2203 from any 1912 build, so you're good there.

Our infrastructure is still 1912 CU5, but we've been pushing 2203 CU1 VDAs all over the place.

ok cool thanks guys!  i'll read over that  link as well.....correct me if i'm wrong but i THOUGHT that loginui issue was only w/ the upgraded desktop agent wasnt it?

this weekend we were JUST doing the controller servers (db obviously) and maybe SF

desktop agents eventually here in the next couple months

i should have mentioned we are 100% persistent MCS VM's too

We started with our persistent MCS, and so far I think things have been fine. My own VM has been solid.

<@URVNXB96C> I honestly don't recall anymore.
But read through this and see if anything applies.

<https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/2203-ltsr/known-issues.html>

cool thanks guys! main thing was confirming if we could do the direct jump which i was pretty sure we could but being paranoid :slightly_smiling_face:. from what i've heard the backend upgrade has been pretty solid and issues mainly stemmed from the new desktop agent so gonna ease into that after we knock this one out.   i wonder if there are any big changes from SF 1912 to 2203 but we'll see :slightly_smiling_face:

Make sure your 2203 CU1 ISO is from August 24 or later.  Citrix reissued CU1 quietly; the initial release had a problem with its RPM.DLL.

Virtual channel allow list is enabled by default

yup for sure i made sure it was CU1 because of that.

Are you swapping controllers or upgrading Citrix in-place?

so what would the virtual channel allow list affect? not sure if we use that right now

Depends on your environment. Webex, Jabber, Bluejeans, etc.

<https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/secure/virtual-channel-security.html>

upgrading in place.....we had our original controllers on server 2012 so a month or so ago we added 2 additional server 2022 controllers to the cluster , then removed the 2012 so as far as i understand we should be able to direct upgrade

Windows Media Player, Dragon, Epic, Zoom... anything that uses a custom virtual channel would be affected.

IMO, I see no reason to use the updated policy unless your security team was forcing your hand.

cool ya i was gonna roll with what we have and see how it goes if we need to adjust after

Here is an example of the Virtual channel list.
Windows Media Player:
CTXMM,C:\Program Files (x86)\Windows Media Player\wmplayer.exe

Nuance Dragon Medical One (your SoD.exe may be in a different location, as I used the copy file install method and not the MSI):
PSPSBEX,C:\Program Files\Nuance\DMO\SoD.exe
PMICIIB,C:\Program Files\Nuance\DMO\SoD.exe
PSPMIX,C:\Program Files\Nuance\DMO\SoD.exe
NUCAAUX,C:\Program Files\Nuance\DMO\SoD.exe
NUACOM,C:\Program Files\Nuance\DMO\SoD.exe
NUAPLAY,C:\Program Files\Nuance\DMO\SoD.exe
NUAREC,C:\Program Files\Nuance\DMO\SoD.exe

Cisco WebEx Meetings:
CSCOMT,C:\Program Files (x86)\Webex\Webex\Meetings\meetingshvdagent.exe

Newer WebEx-Teams
CSCOTM,C:\Program Files\Cisco Spark\dependencies\teamshvdagent.exe

BlueJeans:
BJNBLUE,C:\Program Files\BlueJeans\current\BlueJeans.exe
BJNLOGS,C:\Program Files\BlueJeans\current\BlueJeans.exe


Zoom
<https://support.zoom.us/hc/en-us/articles/4412351225101-Configuring-the-Citrix-virtual-channel-allow-list-policy-for-Zoom>
ZOOMHDA,C:\Program Files (x86)\ZoomVDI\bin\Zoom.exe
ZOOMHDC,C:\Program Files (x86)\ZoomVDI\bin\Zoom.exe
ZOOMHDS,C:\Program Files (x86)\ZoomVDI\bin\Zoom.exe
ZOOMHDV,C:\Program Files (x86)\ZoomVDI\bin\Zoom.exe
ZOOMHDX,C:\Program Files (x86)\ZoomVDI\bin\Zoom.exe


Sources
<https://www.citrix.com/blogs/2021/10/14/virtual-channel-allow-list-now-enabled-by-default/>

That was collected from Slack and from Nick and Myself

And the Webex unified client isn't entirely accurate.

I left before I could test, but there is a new one. Webex support couldn't even tell me what it was.

Ahh I just got an updated Webex this morning.

Cerner support is pretty good, though, so I bet they could tell you.

CSOMT may be the missing one. It's not in my list.

Yea, if you contact Cerner, they will have it.

wait, hold on, we have Webex here and I know the logging is enabled. let me check

Most vendors have updated this list bynow

awesome thanks guys for the help!   for those of you that did the 1912 to 2203 upgrade, about how long did the upgrade take for you?

I have only done the 1912 CU5 -&gt; 2203 CU1 VDA upgrade, which was pretty quick.

Webex VDI.pdf

i setup a 2hr downtime window so i figured that was more than plenty

Webex's app and how it deals with downloading versions in the user profile is still straight up garbage.

I guess it depends on your footprint <@URVNXB96C>.

Just the delivery controllers? If so, how many?

2 delivery controllers so probably looking at those 2 controllers, 2 storefront servers possibly (just upgraded those from 7.15 to 1912 last week) and probably the 1 license server (thats quick i know)

IMO, 2 hours should be fine, but you might want to bake in some extra just in case shit happens.

that should be enough time. 15-20 minutes a DDC, SF is like 5-10 minutes

FWIW, Citrix TAM just told us today that we should go from 1912CU1 to CU5 prior to the 2203 upgrade to avoid rare database issues

<@U02LGMH26JZ> The Docs are misleading then lol.

it seemed more of a CYA type of thing for a single issue that has cropped up

Click on Site Test, get to test 265 of 266 and sit there for 10 minutes

<https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/2203-ltsr/upgrade-migrate.html#versions-you-can-upgrade>

I wonder why they say this. I heard of folks doing both to be honest.

• Virtual Apps and Desktops 1912 with or without CUs, up to and including the latest CU


right, their documentation says you dont have to but like he said after the fact they probably had 1 ticket and a weird rare issue so now they are covering their butt

<@URVNXB96C> just backup the DB, and do the  "oh shit" snapshots just in case

yeah, I've seen the chart, and they said it _*should*_ be fine, but we're healthcare with the single largest in-house Epic installation so CYA lol

yup for sure.....we have sql backups every night at 9pm and gonna do controller/server snapshots

We opted out of hosted Epic to host it ourselves lol

I don't know anyone who has Epic host their stuff yet. My last company was at least talking to them about it, but I would bet against them doing it.

~40k concurrent users, split among 4 farms, with active-active failover (semi-automated) between 2 datacenters

we weren't about to trust that in their hands

<@URVNXB96C> sorry for hijacking your thread :joy: good luck with your upgrade

thought of one more question.......considering we're MCS persistent,  is it recommended before you start the upgrade to reboot all desktops and put delivery groups in maintenance mode to prevent any connections during the upgrade?

SF upgrade from 1912 to 2203 keeps failing w/ a pre-req failure so gonna put a pin in that part o this upgrade until next week so i can look into it more

havent found any useful info in the log just yet

Citrix-DeliveryServicesRoleManager-2022-11-06 20-57-12.log

thought maybe i'd have better luck w/ the SF upgrade after i did the controller upgrade but same issue.....this is the log i'm digging through now

yea, i think every single environment i've ever setup from 4.5 and on is "ctxdatabase"

The only thing I can find is to look for EventID 2002. Is that it?

That's all I know of, as well. You could use PS to query VDA event logs for those entries to compile a list if you wanted to see what was being thrown.

say hello to loops :slightly_smiling_face:

Apps, Delivery Groups, Catalogs, Policy Filters, Desktop Restrictions, AppGroups - That is a nice product enhancement request though to be able to see that in the console

I think I have a script that outputs all the different tags being used and the desktops assigned if that's what you're looking for?

Pretty much yeh, looking to find where all these tags are used.

Cool I can dig that up tomorrow and send what I have

ExportTags.ps1

alright this is what i found a year or so ago and seems to work......if you edit the script to input your delivery controller fdqn and the tag value it will output a grid....when you click OK it will show all desktops w/ that tag.   the first grid will actually show you all tags in your environment but to show desktops for that tag , you need to input that into the script and re-run....i'm sure theres a simpler way to add them all but i didn't dig too much into it

Awesome thanks!
Will give it a go on Monday!

Hi, maybe this helps? <https://support.citrix.com/article/CTX463756/2203-or-2206-server-os-vda-may-experience-bsod-on-tdicasys>

I ended up opening a ticket with MS and also with Citrix to check logs and debug files.  After a month of waiting, MS responded to try the November updates instead.  So far so good, been up and running for more than two days now without any further BSOD or CPU freezes.  Not really sure at this point what the cause may have been, but glad to be over this hurdle and be able to resume patching again!  :slightly_smiling_face:.  The initial debug logs showed driver compatibility issues, although this was a new VM from a master template with nothing else installed, so the only drivers were MS and Xen.

I typically request that they service account that is used to make the DB connection have sysadmin rights. Then use that for the upgrade. There are other ways, but that is a sure thing.

Just re-read that, Sorry. You stated that already.

I am not aware of a script to check. Other than seeing a screenshot, or remoting to the SQL server and seeing. Maybe there is one, and I just don't know.

Ask your DBA to give you a screenshot for support just in case you need to call. Not that you will, but it’s a nice way to say give me proof without him thinking you don’t believe it lol

yup sorry!  upgrade finished in under 20 min w/out an issue!  at least the license server and DC's at least......Storefront servers weren't having the upgrade for some reason

I don't anymore as I usually just let BIS-F do it (check it out) and I exclude BIS-F from removing 'processors' as sometimes I'm moving stuff between hosts with different procs and I just don't want to worry about new hardware detection

Its recommended to remove all ghost items in Device Manager for PVS and MCS.

Troubleshooting HTML5 -&gt; nFactor GW -&gt; SAML SP -&gt; FAS -&gt; VDA (Win10 Pooled desktops) failing to connect.

Workspace App through same GW -&gt; SAML SP -&gt; FAS - VDA flow to same desktops work, but HTML5 fails with timeout, hanging on "Connecting".

Spider sense says VDA can't find its way back to the HTML5 client session.  I see port 8008 in event logs of brokered VDA fire up and wait for connection, but never establish to connection through SNIP to client.

(Non nFactor gateway with LDAP+RSA can connect to same desktops with HTML5 fine)

Any pointers on where I should be looking to troubleshoot?

Have had all different environments blow up with time change in the past.  Now, we build all systems in UTC and allow TZ redirection.  Haven't had an issue since

In the past have had to redo the image after the time change to resolve strange behaviors.  Have not had any of those issues since moving systems to UTC

That applied to nonP only... But included pvs and MCS images

<@UK89L1Q02> Did you ever find out WHY this happened?

samesy, always saw it more with PVS, and also app layering...not so much with MCS.  Usually recmmend new builds on Sunday, or if your doing manual pvs, new versions.

why... it seemed to be the changing of the time by that hour (especially in Fall), seemed to cause issues.  Our best guess was when the time jumped to before the system logged it booted.  if we let them sit for an hour or 2 they seemed to calm down, but being a 24x7 shop makes it very difficult to let a system site idle for that long.
the problem was never as prolific in the spring time change.
to avoid the pain of redoing an image on a sunday after the time change... we just moved to UTC about 3 or 4 years ago and have never had the issue since.

Yep - but only at one customer... MCS machines didn't auto-update time when daylight savings rolled over, I had to fire up the master, manually update the time on it and push the image out again to resolve it.

PVS, yeah pretty regularly. MCS, not so much.
Moving to UTC across the env currently so hopefully that will be the last of it. 

Check this one…
<https://de.scribd.com/document/235836342/PVS-Timezone-Issue-v1|https://de.scribd.com/document/235836342/PVS-Timezone-Issue-v1>

Hope it’s still accurate because it’s quite old :)

have a copy that doesn't require a login?

VDA Cleanup utility, then fresh VDA install for me - especially with 2203.

That's what I was leaning towards too. Any particular reason why especially with 2203?

We saw some weirdness with 1912 CU5 &gt; 2203 CU1 simply upgrading it.

Using the VDA removal tool first has been a 100% success rate.

That said, Ops found issues with 2203 CU1 (haven't even explained it to us yet) that halted our rollout. Sticking with 1912 until early next year.

Hmm I'd be curious to hear more about that. We have planning sessions this week and we were possibly going to do the vda upgrade in the last 2 months of the year

Yeah, so would we. We're too busy to inquire more since it's not our task.

Our environment is massive and incredibly complex, though.