World of EUC

image.png

I remember disabling this via the following regkey, not sure if it still works.
*Key*: HKCU\Software\Microsoft\Office\16.0\Common
*Name*: PrivacyNoticeShown
*Type*: REG_DWORD
*Value*: 2

Doesn't help unfortunately. Compared to FSLogix Profile Container, the only difference should be some filebased action. NTUSER.DAT is stored in the userprofile and should make no difference.

The official error message is Photo viewer cannot open due to insufficient memory

<https://twitter.com/endresenfredrik/status/1458401127385423879?s=21&amp;t=HzM4FcUhM6YYIgUf6FzKLQ|https://twitter.com/endresenfredrik/status/1458401127385423879?s=21&amp;t=HzM4FcUhM6YYIgUf6FzKLQ>

That looks to be the same as what my customer is getting

Do some testing before, that gave me weird issues, some picture in email signature was displayed inverted

Going to drop it into a users hive first - need to get someone to report it first...

Like everything when you need something to break - it never does

Yes on my side they were reverted too also in thumbnail view in explorer, had to revert it back

Can also confirm the registry change fixes the issue, but also ended up giving me issues such as colors inverted on pictures in Excel sheets, outlook, etc.

I ended up deploying a third party picture viewer instead since the legacy photo viewer is kinda crap

This is actually good insight Patrick - thank you for that

<@UU04UTPUN> - One of our other customers had a similar issue and it was to do with photviewier having issues opening photos taken with some mobile devices (andrioid/ios) and not handing the file metadata. They had to move to imageglass instead, cheers

photos taken with mobile devices, we installed this <https://www.copytrans.net/copytransheic/>

I upgraded from 1909 to 20H2 without issues, and we're currently on 1912 LTSR CU3 with PVS.

We were on Trend Micro Apex One at that time, and only recently switched to Defender. I'm also using App Layering.

Are you using PVS or MCS? So we use Deep Security for all VDI and Apex One for physical boxes.

We are moving away from PVS right now. We are replacing our 1:1 non-persistent desktops with app layered hosted shared desktop servers (2019).

Yes. So initially we found that PVS and app layering with UEFI bios and / or paravirtual SCSI didn't play well with a BDM, so we were gonna have to re-engineer to use PXE or TFTP. Given the size of our environment that was a bad idea, app layering was only being done on our newly built xenapp boxes and HSD boxes, so we went with MCS and haven't looked back.

We are almost ready to retire PVS for xenapp, but we still have 4400 desktops to replace with HSD, so for now we got this 20H2 boot issue.

We've traditionally been a PVS shop btw, I wouldn't have thought we'd leave it ever tbh

Which VDA version are you using <@UKJTJD4KA>?

2203 <@UNBRPVD4G> it's not a dominant issue, just once that we noticed in troubleshooting some nasty image crashing in Azure - landing more towards the problem being triggered on a disconnect and reconnect - but not 100% confirmed. Was sure i had seen this combo happen before a couple of years back in Azure with ServerOS

<@UKJTJD4KA> not specifically on GPU no, but interestingly enough Shane and I have been seeing this randomly on that POC we are doing you are familiar with where we don't have any N series. We haven't been able to place our thumb on it but have recently upgraded that to 2203 VDA to see what happens. Some users on newer VDAs now so we will see.

Let's talk a bit on it tomorrow night for our weekly EUC  meet up mate

Sorry <@UKJTJD4KA>, all our GPU workloads are on-prem at present, so I haven't experienced this as yet. The OPEX costs for us to move hundreds of machines to Cloud does not make sense at this point in time given the size of the VMs (including GPU) needed.

That's ok - it was a shot just to see :slightly_smiling_face:

app-v.PNG

Guys, has anyone had issues deleting and App-V package from Citrix Studio with 'Application not Found' (when its showing in the console). Here is the error, I've never used App-V within Studio before but asking for a customer facing this issue, cheers

Citrix has been awful with emails lately, I got 50 for the same thing the other day

Another 20 emails just dropped into my inbox....ffs.

I hear DING DING DING, Then I turn my sound off. Now my Cell phone is  vibrating like crazy. haha

Nobody can't say they don't communicate to us anymore :slightly_smiling_face:

Image from iOS.jpg

One of my 10 accounts got a subscription added to it and I still haven't had time to log into them all and figure out which one 

Because they couldn't be bothered to include that info in any of the 50 emails 

new outlook rule to send emails from <mailto:knowledgecenter@citrix.com|knowledgecenter@citrix.com> straight to trash can

ok. this is fucking ridiculous.  I guess i'm turning off my email for the day.

108 mail so far.. I wonder when citrix will become flagged as spam sender? :slightly_smiling_face:

I received 108 mails. This morning (Denmark time zone) I have received 8 mails

<@ULJ5UE820> Did you really? :slightly_smiling_face:

<@ULJUX26GG> had 120 I heard, so „No“ :)

I didnt face any such issue yet with WS2019 VDA 2203 LTSR

Thank you for posting this. We just upgraded our app layered image for 2019 hosted shared desktops with 2203 LTSR VDA. Let's see if we find something similar. I think we will know more sometime early next week since we don't have a ton of usage in Dev, but I guess if it's a memory leak we should see it sooner...will check either way

It continues in our environment. Granted only deployed to two production servers that are identical. Not sure that I can deploy to more until we figure out what's going on here. Citrix hasn't responded to the ticket yet.

Installed on a fresh build and seeing the same symptom, but smaller amount of memory used per new session compared to the previous systems. Still waiting on Citrix to provide any useful followup.

How much memory per session on average were and are you seeing?

Testing on quiet servers with no other users logged in:
Original server build around .5 to .8 MB per login session
New fresh build server around .1 to .2 MB per login session

:slightly_smiling_face: Leaking that much per session on svchost for TermServ.

Well on that note how much memory does your typical user session use? And what's your user density and vm sizing?

In that one process, which is all I am focused on at the moment, it uses a bit more while the user is logged in, but not a great deal more probably 1 to 1.5 MB per session. Not sure what user density and vm sizing have to do with this particular issue. The servers are sized appropriately when memory isn't being leaked away.

The svchost.exe process for TermService should not use 10.23 GB of memory. This is after just 2 days since a restart.
What is special about my environment? This started when 2203 was installed on the servers.

I was just curious. For learning purposes. I know it's not related.

As an MSP, for us, the answer is almost always, it depends. We do sizing per client, but generally limit user load to 20 per machine and CPU and Memory are somewhat application dependent. There is a formula that is used, but I don't do that portion of things. We adjust as needed based on real world performance as needed.

Found the culprit.
&lt;&lt;username&gt;&gt; tried to open custom virtual channel CUEPUX

This is the ControlUp custom virtual channel. I knew that virtual channels were going to be blocked by default in 2203, but hadn't put in the policy to allow them as I was hoping to test an only allow as I found them and may get back to that, but for now they are all open again.

I have also observed this same msg in System Event Logs for other virtual channel.. CTXMM and CTXCAM

<https://www.deyda.net/index.php/de/2021/12/21/saml-authentifizierung-zwischen-citrix-microsoft-mit-azure-mfa/#Troubleshooting|https://www.deyda.net/index.php/de/2021/12/21/saml-authentifizierung-zwischen-citrix-microsoft-mit-azure-mfa/#Troubleshooting>

Maybe this helps

So we saw this error before, but honestly I THINK it was a red herring we chased for no reason. We had some persistent desktops that would log a user off and you would see this event. We investigated FAS, the GPOs, SF, VDA side--and couldn't find any correlation.

in a nutshell, we weren't having issues with any other xendesktops, so we noticed the error (VDA side) came right after the session was terminated. So I wonder if that error is a bug / not meaningful.

Could it be something like this.  The error implies that the VDA cannot talk to the FAS server<https://rakhesh.com/citrix/s104-identity-assertion-logon-failed/>

Is the vda and the storefront in the right groups on the fas. Do u test it with set the groups to domain computers. So no machine are excluded ?

Yes, I checked that. It's Domain Computers and Domain Users. And StoreFront seems to correctly authenticate with FAS.

I tried adding a separate AD group with the VDA's but that didn't give any change.

Are u define the gpo and set it to the vda and sf server ?

Yes, GPO is applied to both the VDA's and the SF servers.

I also see the registry settings that the GPO creates.

Is the vda trying to connect to the FAS server you expect.  The GPOs are just to tell the different systems which servers to connect to.  FAS does keep an index of the servers and if the index is nto the same between the vda and storedront you can get errors but not this one.

Yes, the event shows one of the FAS servers I specified in the GPO.

Do you have a firewall between the VDAs and FAS servers?

No, but the FAS servers are in a different subnet so the traffic is routed. I asked them to check with the networking department. Thanks for the link you sent earlier by the way.

Do a tracert from the VDA to FAS to see how far it goes. But if Port 80 is open, that is strange its through a Access Denied error. Nothing in the FAS event Logs?

That's the annoying thing. If it's an access denied, you would expect an error on the FAS side. But there isn't.

You can do it from the VDA if you have access 

If the VDAs cant get to the FAS sever there woudnt be an error on teh FAs servers

He said Port 80 is open from FAS to the VDA right? I see it above. I assume he checked that and verified it's open. The Error message above is because the VDA can't fetch the Cert from FAS. Which I know you know the Rob, because you're the man at Citrix. :slightly_smiling_face:

But if it's an access denied, where do I need to provide access?
And if there was a firewall or routing problem you would expect the 'no endpoint listening' error (<https://support.citrix.com/article/CTX269399>).

I've fixed this when adding the VDAs and Cloud Connectors, Storefront Servers Computer AD Account to the Windows Authorization Group in Active Directory

No. In Active Directory you can find this ad group

It seems a colleague of mine solved it. The Delivery Controllers and the VDA's had the same FAS servers in registry, but in a different order. Even though they both receive the same GPO.

When correcting the order it worked again.

No, the client had 2 policy's in place. In one, address1 corresponded to server1 and Address2 to server 2. In the other policy it had address1 to server 2 and vice versa.

So when we had the same order, address1 and server 1, problem solved

did you upgrade the VDA? or did fresh new installation

That would be the VDA Commandline Helper Tool. Citrix retired it and ripped out the KB article for it.

You can still download it from the Supportability Pack, though I have no idea what versions it supports: <https://support.citrix.com/article/CTX203082>

Sounds like this may be newer than that one: <https://www.dropbox.com/s/zcqxc8icz24scz1/CtxVDACmdLineHelper-May21.zip?dl=0>

Yeah, that last link shows CVAD 2103 as the latest for the tool.

I wonder if it supports WEM Cloud Service.

The WEM Agent tab does have a space for the
WEM cloud connector address.

CCV1.17.png

The Citrix Cloud (DaaS) doc script V1.17 update is ready for testing. Let me know if you want to test.

.json configfile, copied via GPO or WEM in %appdata%

Yeah, I preconfigure the .json file and copy it to the user's Teams folder at logon.

So you both do a replace? In our case, it seems more desirable looking at this to do an edit and change the parameter. I guess it would reset their preferences each time? Like dark / light mode?

I use WEM to run a PowerShell script (RunOnce action) to configure Teams the first time, but they can change preferences after the fact.

You can also use this code in a logonscript to disable it:
$TeamsConfigFile = "$ENV:AppData\Microsoft\Teams\desktop-config.json"
If (Test-Path $TeamsConfigFile) {
    (Get-Content $TeamsConfigFile) -Replace '"disableGpu":false', '"disableGpu":true' | Set-Content $TeamsConfigFile
}

If you run Teams optimized, I don't see many reasons for turning off GPU tbh.

Specifically the ICA RTT is like 20ms, then 500ms, then 80ms or something way off every time you refresh. It was concerning that it may be a real issue, but it SEEMS to not be legit

Have not seen that so far. We are on Director 1912 CU2 and testing VDA 2203. The 2 production servers have pretty stable connected users in fairly close physical proximity and generally are getting sub 20ms ICA RTT. As far as I see, it's pretty consistent there.

I am seeing something similar, but with CVADs

Removing the german languagepack has helped already a lot of times on a SF Server :grin:

yep. however it is supported and should work.
Getting this error at startup. Restore from last week did not work either.
Aktualisieren des gemeinsam verwendeten Schlüssels fehlgeschlagen: System.NullReferenceException: Der Objektverweis wurde nicht auf eine Objektinstanz festgelegt.
   bei Citrix.Fma.Sdk.ServiceCore.KeySharing.PeerKeyManager.QueryPeers()

Sometimes I think Citrix is doing this intentionally to bring us to DaaS.
What a crappy SW.

<@UMCH8SMQB> which version of Storefront are you running? I suspect 1912.3.X?

Were there any changes made to Storefront prior to the issue? Servers added etc?

<@UNBRPVD4G> Version: 1912.0.3000.9 on Server 2022.
Absolutely no changes, no Windows Updates, nothing.
Pretty small site, so I ended up installing a new server without German LP and SF 2203 to bring people back to work..

"Dont use other OS languange than Englisch and no LPs when deploying Storefront!"
Storefront: "Hold my beer".

again a big shutup to the persons who created or maintain storefront locaization.
This is just hilarious, cant find words.
New System, English OS, English Storefront Components, Cant add a Server to the deployment with the following error:

Thanks for nothing.

Did you delete the VDA in question in Studio or jus AD?

I think I have seen something similar where I ended up deleting the VDA in both Studio, AD and PVS and then created a new AD object via PVS and the imported the new VDA in Studio

I can try deleting it through the PVS console to see if that changes the behavior.

It's just weird because I have done this before with success. Looking at Get-BrokerMachine, the last connection to this VDA was... January. oof.

I've had this when using PVS, yes do a proper clean-up at all levels, AD, target in PVS console, and recreate it using the Wizard. I've also noticed that a couple of reboots was needed then to see the blank/SID name to be replaced by the actual machine name after that.

Seems I had not had enough coffee, and neglected to actually remove it from the Machine Catalog. Did that, and all is well again.

I blame Citrix for naming this option "Delete" instead of "Remove from Machine Catalog".

because the Delete wizard also has an option to delete the machine from the HV and in AD.

Sounds like adding a "Remove from Machine Catalog" option would solve my issue, then.

I haven't deleted a device in years, so I honestly would not have guessed what the Delete option looked like.

When I remove a device that is used in PVS. I do this.
1. Remove from DG and MC
2. Remove from AD
3. Remove from PVS Console

But if you asked me how to do it with MCS, I would be like ummmmm (in the background doing a goodle search ) lmao

with MCS just do it from the MC, it will remove the machine from everywhere.

I seem to remember that, but it's been a long time. Good to know, Thanks Olivier.

Yeah, I have never touched MCS outside of Citrix training.

We have had issues with MCS not removing the account from the identity pool, so we go to delete the device from MC and it won't allow it, there is a PoSH command to unlock that identity. But PVS is more straightforward for sure.

Every time we would run the PVS wizard it was pretty much an experience with "ok, wait guys, here it is, be very attentive" and taking like 1 minute between each step. "So far we are fine.... so far we are fine..."

I really don't regret that in the MCS environment

Most ridiculous thing ever, bringing PVS into Azure.

If we go to Citrix Cloud/Azure, I would want PVS... :slightly_smiling_face:

This is called "Windows File Explorer Addiction"

PVS is an amazing product... but people just get addicted to it, and don't want to give up on it when it's not the use case.

I love PVS and I can do so much with the vDisk. MCS I used in 7.6 about 6-7 years ago. It was ok, but I have way more control with PVS.

<https://virtualfeller.com/citrix-workspace-visio-stencils/|https://virtualfeller.com/citrix-workspace-visio-stencils/>

I just use PowerPoint rather than Visio these days. Dan's got the same set of icons available for PowerPoint too, I just grab those

PowerPoint sometime it's easy and flexible

Not supported rarely means not functional. If you need Citrix support at some point and they discover an unsupported situation they won't help you unless you upgrade.

Not entirely correct. They won't help except if you prove this is not related to the « unsupported layer ».
In this example, you have an issue when hosting VMs on Vsphere 7. They won't support you unless you prove you have the exact same issue on VSphere 6.5. In that case, they will.

I have quite a few 2012 boxes that have no issues with vsphere 7. 

What you are trying to achieve is quite illogical, especially with FSLogix.

Look for "Enable multi-session write-back for profile containers" with UPM (<https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/policies/reference/profile-management/advanced-policy-settings.html>) and disable Session Sharing (but I really don't recommend that)

Yeah I guess its just a glaring oversight on my part. Looking into other options. I mean worst case we have a DG of XenApp boxes

Best option really, make your published appson "silo" application servers

So then it's just an FSLogix issue, if it's on on silo servers, look above for UPM to become a supplement to your FSLogix sessions and work as a write back system on FSLogix disks.

This is normal, If you have a XA Session that is a desktop, and the same user launches a publihed application it will  not launch on the same server. Look up Session sharing details. However it's not supposed to jack up the session

So our desired architecture is we have 2500 users or so with a hosted shared desktop (2019 Server multi-session host), and we also hosts our published apps on that same DG. So we have another 1000 users that will launch published apps from a physical device or persistent desktop, but there are users in the 2500 HSD users that need to launch legacy apps from that same pool their desktop sits on

I see, so you have users that open up a Shared Desktop aka Published Desktop, then they also open up published apps? Are they opening the published application from within the Shared Desktop? Or are they going back to workspace and opening the app from the source where they clicked to open the Published Deskop? The reason I am asking, is if they are in the published Desktop and using Workspace in the share desktop to open published applications vPerfer would help you.

From within the desktop, so logically it may not work at all

Ok, So here is the ticker. What profile type do you have set in FSlogix? I ask because I set Profile type to 3. The first session was RW disk, and the seconds launch for the same user will be in Read mode.

Yeah we have it that way too. I will double check

Also, the shared Desktop they are on. The application that they are opening from Workspace app isn't already on the Published desktop?
I am trying to understand why they would go into a published desktop, then open a published app that is hosted on the same machines? Forgive me if you already said this.

Yeah my logic keeps going in a circle and I see more and more why you wouldn't want to do that. We changed vprefer policy and now all the apps launch locally as expected/ desired. We will have some siloed apps that will need to be in a different machine catalog and some legacy apps in the same situation too. So I think I just wasn't thinking this through at all originally. Thanks for the help all.

No worries man. I have done this as well before sometimes we just try to put out something to make everyone happy, we forget about making it easy on us. Glad I could offer some help.

we're mcs persistent w/out using profile management (for the most part) so may not be the same kind of comparison but a first time logon for us usually is about 30 seconds and each after is between 15-20 seconds (sometimes a little more , sometimes a bit less).    are you stopping the clock when you can fully interact w/ the start menu i assume?

i'm sure you have this already but on a dual screen setup we found one of the biggest culprits for longer login times was the Acrylic Background blur deal  Windows 10 brought in (i think in the 1809 build?).  disabling that helped quite a bit

I should mention, this is non-persistent PVS on 1912 LTSR CU3.

And yeah, <@ULJM9DQF2> tweeted about that long ago, which prompted me to make the change.

cool yup i remembered you were non persistent and pvs so not apples to apples here

I run Citrix Optimizer, rip out _all_ UWP apps, use WEM as much as possible, have efficient GPOs, etc. I think I'm doing everything that I can think of to help with this. New hardware, which I'm getting this summer, should help, and I've heard rumors that 2203 is faster (<@UL5TD2QM9> I think mentioned this?).

No user GPO, all in WEM, logon time = 20 seconds.

I swear I've tested no user GPO, and it didn't really cut down the login time. I can certainly revisit that, though.

Please do. The waiting time we can see is the "FSLogix Apps Service" which is normal, last 4 seconds or so. Then Explorer kicks in, takes 15 seconds to make the desktop operational.

I've looked at our FSL logs, and didn't see that much of a delay. Also ran Guy's COntrolUp logon analyzer script which didn't really reveal anything glaring.

I know FSL is a little slower for the logon than CPM, but I just cannot go back to CPM as it stands.

20 seconds from clicking on the published desktop to seeing your desktop on a brand new profile?

Okay, so connecting to an existing profile

yes, brand now profile you have the Active Setup etc

That's still 7-10 seconds faster than my setup.

Defender is actually responsible for adding roughly 10 seconds to my logon.

We have McAfee, so I can't say about that

My guess is that you'd get your logon down to 15 seconds without McAfee, lol

We just ditched McAfee in our enterprise. All Defender now.

If you run into issues with your VDA, post up here. I went through a lot of bullshit to get things working nicely with Defender.

but from the click on Desktop icon, in StoreFront or CWA, we get to 20 seconds before the desktop being operational

it seems like Windows would go to synchronous GPO/GPP processing sometimes

I just removed all of that, so it never happens

I think 20 seconds would be perfect. Can't see anyone complaining about that. Not sure I can get there, though. Certainly not on my existing E5-2697A hardware.

I don't have any GPP stuff anymore - all of that is now in WEM. Moved Folder Redirection to WEM, too, which cut nearly 20 seconds off the logon.

For sure, folder redirection was quite something for us too, I've removed that as well

I just moved it from GPO to WEM. Single best move I made.

Indeed. But we have instructed our helpdesk team to prepare the "new" user profiles

so our benchmark never include a brand new profile

not such a high price to pay, our users keep their profile for months if not years before they need a reset

I'm going to throw that into the hat as an option, let leadership shoot it down. I like that idea.

I have only had to kill one FSL profile since going live.

and they were a crazy user who bastardized everything

Well, especially with WEM, because WEM is a post-logon thing so with a brand new profile you must have a lot of things not in place

WEM should run while you're already in your desktop session, though

yes it does, but I think that your very first logon, on a brand new profile, might not be the best

I'm an App Layering user, too, so I can't move some of those registry keys to the default profile.

I was an App Layering user too. Trying to convert them in my new place. :slightly_smiling_face:

Honestly, I can live with an extra 10 seconds or so for the initial profile creation. It's the next logon and others that I think are the target here.

Well, using different techniques, like extreme UPM optimization, we would achieve a 30 seconds logon even at the time

but with FSLogix, M365 and the "go big on profile" things have changed a little bit

MS doesn't do us any favors. Each build of W10 gets slower. Go-EUC has documented this consistently.

yeah, that's why I've decided to go on full WEM, nothing on MS

We're going to new Cisco blades with Intel 8362s, so I really hope that alone will alleviate most of this issue.

Our current hardware is just dated, and also affected by those Intel bugs from back in the day.

Meltdown? Not still sure if it's the name of the fix or the issue.

Profile streaming was great because you just download the NTUSER.DAT and deal with the rest later

We used Profile Streaming when we were on UPM. It was great, for the most part.

OneDrive alone is reason enough that we can't use it anymore.

yeah the only issue was with just mapping drive and printers

I use WEM for drive mapping, too/ :smile:

which is the main things I've noticed to take time in logon, and yes as you say it's all in WEM now for us too, and no issues

Printers... we're lucky there. Citrix Universal Print Driver only, and we map any printer the user has mapped on their client.