World of EUC

VDA Registration Issues: The Never Ending Story......

image.png

I fixed the issue on my lab! Old GPO pushing an incorrect ListOfDDCs key. However, client is still noting this:

Do they use a proxy? Check there is a local exception on the machine: netsh winhttp show proxy

Might also get the cloud connector checking tool as it could be a missing cert as well

I ended up building a second cloud connector, same issue. I'm doing a re-build of the win 2022 master today. We are trying a new process, where we house golden images on Azure, convert them to VMware / Nutanix /etc as required. Everything tested fine on my VMware home lab, but there might be something that's conflicting with this client's environment.

It was firewall! <https://support.citrix.com/article/CTX133769|https://support.citrix.com/article/CTX133769>
Blocking the port used to communicate with AD as per the above KB

Is Notepad running on the exact same session as the company application when doing the test? Are you using the universal print driver, or do you allow the mapping/installation of the native driver?

Give me a second I'll upload it to my Github repo

<https://github.com/kaspersmjohansen/Install-Printer-Drivers>

At line 1 you have to specify the print server name/names

I'll work on some documentation for the script

how do you run it, is is a computer startup script?

Either as a script in the golden image og as a startup script/scheduled task

Cool, however if you have SEP that probably won’t work

Printer Drivers need to be installed prior to crazy SEP. E.g. if my customer have just 1 new driver we need to rebuild the master image (unless you wanna do uninstall/reinstall). Might be type 3 vs 4, but back in the days a used a similar script with PVS

Good to know. SEP is crazy as you mention :)

I think that since the PrintNightmare security update, our only option is to find a Type 4 driver type and trust the print server via GPO.

I mean preloading the driver in the gold image might not even work at all...

Make sure you have the latest uniprint DLLs also.  

Xerox Global driver is great, is only a Type 3 tho, sorry.  Makes builds easy.

<@U01DU4TSUNS> I think SEP == Symantec Endpoint Protection

CU3 is pretty old, I'd check the known issues list. 

We also had these issues with CU3, CU4 and CU5. Go with at least CU6 on your VDAs...far more stable!

It is weird that issue happens only after patching and it is so sporadic. A reboot fixes the issue. But they are all rebooted nightly. From last 3 days 2 servers showed this behavior. Today there is no issue on any of the servers. Other than the desktop service information in event logs there is nothing.

Ghost session?  Check for machines with session but no username.  

Despite no username, Citrix will still try to reconnect, there's some non-visible stickiness in there. 

What does the event logs say on the launch times in the delivery controller?

Thank you <@UV3TVUW0K>, we found a session with no username but we are able to identify with endpoint name. So I have checked the VDA where the User wants to connect and the VDA was not registered. I rebooted VDA and everything works well again. <@ULLH5JGFR> the Delivery Controller Log has no entry of this issues. 

I've been running Trend Micro in my environment since the OfficeScan days, and now Apex One, and have never experienced that issue, personally. Mine are all W10 VMs (7.15 previously, now 1912) on XenServer, though, so it's not a good comparison.

We had this issue but on 1912 LTSR CU3.  There are additional exclusions that were unceremoniously added to the AV best practices doc that address this specific problem.

We are also running Windows Defender in the affected VDAs.

<https://docs.citrix.com/en-us/tech-zone/build/tech-papers/antivirus-best-practices.html>

Virtual Delivery Agents
Files:
• `%SystemRoot%\System32\drivers\CtxUvi.sys`
• 


We identified that as the change using the wayback machine

<https://web.archive.org/web/20210813115801/https://docs.citrix.com/en-us/tech-zone/build/tech-papers/antivirus-best-practices.html>

You can see here that file was not listed under "Virtual Delivery Agents &gt; Files:" in August

And that change is not included in the "Document History", so even if you subscribed to the RSS feed, you wouldn't have been notified of the change.  :face_with_rolling_eyes:

They need to move all their documentation into github like MS is doing

I ran into something weird like that where workspace app would disconnect the session if you tried to paste an image into it, was the weirdest thing

Well, that's certainly something to look at. We're working on getting session recording set up, to see if there's anything like that happening. That's good to know though. I guess sometimes it's not the obvious, but the truly bizarre.

Citrix will tell you that you need CDF traces... :slightly_smiling_face:

I'd consider Receiver logging if it's the same users. Easy to implement.

Yeah, I think that's where we're at, to be honest

This, essentially?

<https://support.citrix.com/article/CTX286820>

I've never used that method, but I'd be interested in hearing if it works.

This is what I was referring to: <https://support.citrix.com/article/CTX132883>

Cool, we'll try both and see which works better. Not sure how comfortable they are with editing the registry, tbh. Carl Stalhood has this also:

<https://www.carlstalhood.com/workspace-app-for-windows/#receiverlogging>

Yeah, I usually just enable the logging myself on their machine remotely. Asking a user to click on a batch file is going to far here.

I'll give them the choice. They have their own "IT"

you have EDT enabled?  Same netscaler gateway for users that work and that dont?

Sorry, we're using Citrix Cloud. No EDT yet, can't seem to find the time to get that in place. Would that be beneficial?

it 'can' be.  Right now it's prob one less thing to turn off though.

"Session sharing does not occur by default between published desktops and any published applications running inside those desktops."
How does this work if you've also set "Restrict Remote Desktop Services users to a single Remote Desktop Services Session"? Does the published app just not launch?

Published Apps and Published Desktops are different resources, Session Sharing won't work unless you have a single server in DG :slightly_smiling_face:

good luck without pcap :slightly_smiling_face:
p.s just here to see the solution also

Use pktmon, it's built-in :slightly_smiling_face: <https://docs.microsoft.com/en-us/windows-server/networking/technologies/pktmon/pktmon>

Sounds interesting. How to check whole network patch to verify traffic and delay on each device in network path ? :thinking_face:

You can't do that easily, not with these tools. I would recommend to contact a networking specialist to debug your network stack

```Powershell: Test-NetConnection &lt;ip_address&gt; -p &lt;port_number&gt;```

<@ULLAD12P2> you'll get TCP connection test but no tracert unfort.  There is a cmdline switch for tracert but it's only for ICMP, shame.  Surprised how hard it is to do a TCP connection with a defined TTL!

Can anyone tell me if the consult then transfer function, for a mobile incoming call in Teams is working from a Win2016 VDA ? <https://support.microsoft.com/en-us/office/transfer-a-call-in-teams-b7f40f14-e083-46b9-b739-68038c8f73a0|https://support.microsoft.com/en-us/office/transfer-a-call-in-teams-b7f40f14-e083-46b9-b739-68038c8f73a0>

Anyone using the Browser Redirection Extension for Google Chrome and experience issues with high CPU since the extension update at 19 november 2021?
Issue only occur at some sites when opening the site from a shortcut or launch at logon.

I notice not exactly the same but user is frozen on Preparing Windows.
I can see all process (even WEM log shows that all actions have been precessed).
User cannot be logged out (even by admin). The server need to be restarted. 
It is a very random issues (cannot see the problem for weeks).

Do you published apps or desktops? In my environment it is desktops.

Hmmm, sounds very annoying :slightly_smiling_face:

Gonna try the 2112 VDA and see if that makes a difference

Not EDT based? Sounds like an EDT special 

Have you read this KB ? <https://support.citrix.com/article/CTX312452>

Yeah, Graphics Engine is running here though and we have no antivirus

Definitely only affects Server 2022 - 2019 is fine, so is Windows 10

I had this after an in place upgrade, I cleaned the VDA and reinstalled it

I've rebuilt fresh four times now :disappointed:

<@ULJMUPWHK> I get your same behavior with the same few remediation options. I've notice the issue only occurs on VDA 2019 and 2112 but 2106 is ok. We're running the latest release of FSLogix.

2112 VDA install fixed this for me, sorry forgot to update thread

<@U02MF1TENCQ> Hello,
I don't have FSL but I use Folder mirorring acceleration. I upgrad to VDA 2112 with same result.
The problem seems to come from LogonUI.exe (when the session hangs kill this process make unfreeze the logon)

Same type of problem
<https://docs.microsoft.com/en-us/answers/questions/621859/rds-problem-reconnecting-to-a-session.html>

<@UL5TCSW3D> I was hoping 2112 would have solved the hang for me. I'm still in early testing on a small UAT catalog. 1 out of 6 people had it. I'm sensitive to it for sure.

I make a posh script to be run as schedule task with a system account to unblock users

Function Get-QueryUser()
{
    Param([switch]$Json) # ALLOWS YOU TO RETURN A JSON OBJECT
    $HT = @()
    $Lines = @(query user).foreach({$(($_) -replace('\s{2,}',','))}) # REPLACES ALL OCCURENCES OF 2 OR MORE SPACES IN A ROW WITH A SINGLE COMMA
    $header=$($Lines[0].split(',').trim())  # EXTRACTS THE FIRST ROW FOR ITS HEADER LINE
    for($i=1;$i -lt $($Lines.Count);$i++)
    { # NOTE $i=1 TO SKIP THE HEADER LINE
        $Res = "" | Select-Object $header # CREATES AN EMPTY PSCUSTOMOBJECT WITH PRE DEFINED FIELDS
        $Line = $($Lines[$i].split(',')).foreach({ $_.trim().trim('&gt;') }) # SPLITS AND THEN TRIMS ANOMALIES
        if($Line.count -eq 5) { $Line = @($Line[0],"$($null)",$Line[1],$Line[2],$Line[3],$Line[4] ) } # ACCOUNTS FOR DISCONNECTED SCENARIO
        for($x=0;$x -lt $($Line.count);$x++)
        {
            $Res.$($header[$x]) = $Line[$x] # DYNAMICALLY ADDS DATA TO $Res
        }
        $HT += $Res # APPENDS THE LINE OF DATA AS PSCUSTOMOBJECT TO AN ARRAY
        Remove-Variable Res # DESTROYS THE LINE OF DATA BY REMOVING THE VARIABLE
    }
        if($Json)
        {
        $JsonObj = [pscustomobject]@{ $($env:COMPUTERNAME)=$HT } | convertto-json  # CREATES ROOT ELEMENT OF COMPUTERNAME AND ADDS THE COMPLETED ARRAY
            Return $JsonObj
        }
        else
        {
            Return $HT
        }
}

Start-Sleep -Seconds 10
$sessions = Get-QueryUser


foreach ($s in ($sessions |?{$_.Sessionname -like "ica*"}))
{
    $process = Get-Process -Name LogonUI | ?{$_.SessionId -eq $s.id}
    if ($process.StartTime -lt (Get-Date).AddSeconds(-10))
    {
        Get-Process -Name LogonUI | ?{$_.SessionId -eq $s.id} | Stop-Process -Force -Confirm:$false
    }
}

Still occurring with my VDAs. Le sigh. We'll see if Citrix support has any ideas.

KillLogonUI.ps1

Hi All,

You can run the following Powershell script with a schedule task and a logon trigger. Since I put the script the problem went away. The script will kill LogonUI.exe if the process hang for 30s which will unfreeze the user logon process.
A log is created to follow the problem and the user impacted.

I had this issue 9 months ago and gave up on it.  If you figure it out, let us know.  :slightly_smiling_face:

doesn't even work.. you can't disable ADAL and use MFA

This is a workaround, it should work but I haven't tested it myself:

if (-not (Get-AppxPackage Microsoft.AAD.BrokerPlugin)) { Add-AppxPackage -Register "$env:windir\SystemApps\Microsoft.AAD.BrokerPlugin_cw5n1h2txyewy\Appxmanifest.xml" -DisableDevelopmentMode -ForceApplicationShutdown } Get-AppxPackage Microsoft.AAD.BrokerPlugin

Implement it as a login script/scheduled task

It fixes/resets the AAD Broker Plugin which is responsible for activating Office

for some reason it doesn't work with published apps

RDS RemoteApps also suffers from this issue, so it's not a Citrix issue

I think console from 127.0.0.1 is a connection from a hypervisor console. Ignore me if it's a physical PC.

thanks for your input. A little bit "hard" for helpdesk to troubleshoot these sessions. It cant be determined if a user is present in office and works (at console) or a session is just disconnected via citrix. Not in our case cause no user will sit in DC and work but generally ;)

There is an agent thats needs to be installed with the vda

Is it integrated in the VDA installation or is it a different package?

It's in the Features options for the VDA installation

so I need to update the VDA install on the gold image to enable MCSIO

MCS/IO is a tick box when you install the VDA also make sure you got Temporary storage configured in hosting connection as well

I have reconfigured the VDA but MSCIO will not stay enabled. Do I need to do a proper uninstall/reinstall?

isnt it the case that deployment will fail wenn mcsio is not installed? Did you create MCSWCDisk by yourself, was it an existing machine or was it created by studio?

I have enabled the RAM cache and disk cache in the Machine Catalogue. The MCSWCDisk was created, mounted and formatted automatically for each machine. But no writes are redirected to the disk because the MSCIO driver/feature wasn't installed initially with the VDA. I've just run the VDA change/modify setup and added the MSCIO feature, but it doesn't work, the MSCIO is still missing if I run the VDA change/modify setup again. So I am guessing I'm good to uninstall the VDA properly, run a VDA cleaner if there is still any out there, and reinstall the VDA.

Even with a proper VDA uninstall / clean-up utility / reinstall and selection the MCSIO driver in the Features page, it doesn't work. The MSCIO driver won't be installed. I will try to find a way to install it via a cmd line maybe that can help.

you can try:
/components VDA /controllers YOUR_DDC.fqdn.local /disableexperiencemetrics /enable_hdx_ports /enable_hdx_udp_ports /enable_real_time_transport /enable_remote_assistance /includeadditional "Citrix MCS IODriver" /masterimage /noreboot /quiet /virtualmachine

This will presumable happen with the next LTSR release

upported operating systems:
• Windows Server 2019, Standard and Datacenter Editions
• Windows Server 2016, Standard and Datacenter Editions
• Windows Server 2012 R2, Standard and Datacenter Editions


Given the M365 support statements you might want to look at 2019 anyway?

Yeah, I was kinda hoping to get some more native MSIX capabilities, but I can just slot AppVentix in anyway

FSLOGIX Rule applied to office16\root\* and change build numbers of OS :smile:

Are your VMs running on XenServer? Isnt it the case that - if power managed by studio - the shutdown is initiated by the VDA, not as Hypervisor action (that would possibly initiate xenagent.exe)? If Xenserver, do you have tools auto update enabled?

Tools are able to receive the updates from Windows Update, yes. I always thought Powermanagement is done trough the hypervisor connection? At another customer (always on Xenserver, different Version) I've set -ShutdownDesktopsAfterUse $false and there the cmdlet has the expected result.

The only thing that comes to my mind is that the user is connecting to 2 different farms.  The first farm has a Workspace App HTML version, e.g .: 21.0.1 and the second one is 21.5.5 and on the second connection the browser uses the client already loaded and does not load another version (21.5.5)

Does anyone have the opportunity to test it?

Monitoring panel to see only green lights (joking) and I started a morning script to get a status of my infrastructure (uptime, certificate expiration, registered/unregistered, powerstate unknown, check for some event in Windows Event)

what kind of information would you like to see in the morning or any time you open Citrix cloud console?

What should be at a glance there?

1. my services?
2. Monitoring data of infrastructure (like Steven says, uptime, certs, reg/unreg,power)
3. customizable, widget-like dashboard?
thoughts?
want to know so I can harass PM's :wink:

Are any users yelling? If not, than sleep more.

I'm not the type that waits for users.  I try to be as much proactive as possible. :man-shrugging::skin-tone-2:

<https://github.com/StevenNoel/Citrix-Morning-Report>

Nothing.  Problem Management efforts go into engineering 'correctness'.  Lucky enough I guess that we don't have anything needing proactive inspection.

<@UPWELNJGZ> is experiencing issues at a customer as we speak

Saw it on a customer... sems its working again ? (Edge)

Yes seems to work again. Some users needed to restart their session.
But strange, restarting the session didn't work an hour ago with the same user.

Users had to run this once "msedge --disable-extensions" - close edge and hereafter start Edge as normal worked fine

Yes seems that for some users it's still not working after restarting the session. Thanks for the hint with the switch.

So the only resolution right now is to disable extensions? Just got our first few tickets on this.

We enforce extensions, so the only thing I can think of doing right now is blocking the extension until it's fixed. No chance users will launch Edge with a command line.

The strange thing as I said... User tried running the command "msedge --disable-extensions" once and hereafter, they could use (Start Edge) as nomal

Any chance somebody submitted a ticket with Citrix for this? I just asked our contact about it.

We also just removed the extension using GPO for all users since this was the quickest way.

<@ULC9J5XPB> No, we haven't submitted a ticket yet. Our plan was to try reinstalling the plugin again using GPO, but right now we'll leave it disabled.

They have properly added a bug by mistake or something as the extension has been updated today
 <https://chrome.google.com/webstore/detail/browser-redirection-exten/hdppkjifljbdpckfajcmlblbchhledln>

I think I'm going to go that route. We only use BCR for a handful of sites, so the impact is likely going to be low.

...and that's what I'm doing, as we continue to get tickets.

<@URAD0CDBK>: Did disabling the extension fix it for users who previously had Edge crash? I'm wondering if they're out of luck until you run that command line.

After disabling the extension Chrome/Edge was working immediately.

<@URAD0CDBK> which version of Chrome/Edge are you using?

I'm on 94.0.992.47, FWIW. I also didn't see the issue personally. Neither did our other Citrix admins.

Hmm..let me just test with the latest Edge version

We're having various versions (91.0.4472.106 and 95.0.4638.54)
Doesn't really seem to be realted to a specific version.

Hmm...for some reason I am not using the latest Edge version in my lab. I am however not experiencing any issue with Edge 95.0.1020.53 64-bit

One of our admins is on a call with a user now and said that disabling it didn't fix Edge. Gonna try the command line route to see if that helps.

Think we just got this issue as well had a report 30 users report edge is crashing will check it out.

Did anyone create a support case for this? So I can pass it on internally (I work at Citrix)

Not yet, but I passed it onto one of our reps, who usually does create a ticket. Worth passing on internally, though...

to remove you just removing the extension id / update url from these Chrome/Edge polices "Configure the list of force installed apps and extensions/Control which extensions are installed silently" ?

I'm removing it from our *Control which extensions are installed silently* policy.

FOr the "cannot be installed" policy, we use *** so that it blocks everything.

3 for 3 on our tickets for this resolving it. Just waiting on the last 2 users to get back to us.

I just saw this, we've been dealing with this behavior all day.  Clearing out the users chrome roaming data also seemed to be a work around. Anyone gotten ahold of citrix support yet?

just noticed this thread ... experiencing same issue in PRD here (and finally disabled BCR)
Case created @Citrix ... we'll see

<@ULXAFK8BU> <@URAD0CDBK> can you share with us on your current chrome version? I think there are some issue with chrome update. if you can try remove the extension or repair the extension and see if the issue cleared up or not. If you have a ticket open with Citrix please PM me.

switch to the beta branch 5.4
<https://chrome.google.com/webstore/detail/browser-redirection-exten/pajinnhelbahlkakdibkhfcnnoflloff?hl=en>

<@U02P1J9TEJE> We have different customers using different versions. One used the most recent version 96.0.4664.45 and another one used 91.0.4472.106 and 92.0.4515.159

Also, we have noticed that not every user got this issue even if they were working on the same TS.
Removing the extension using GPO resolved the issue immediately. Until now we were not able to reproduce the issue by reenabling the extension.

FYI : Citrix case @ engineering ... seems they are aware

<@ULDHJB8BB> Do you see beta branch 5.4 worked for your case? did you try repair the 5.3 extension?

A customer of mine just reached out to me and the issue persist even with version 5.4

It seems as though the easiest solution for Citrix would be to simply revert to the previous build and work on a fix in that beta channel...

According to the 5.4 release notes, this is exactly what they did

I was trying to figure out what was going on. After reading all this, I had to remove the Extension that we push out via GPO.

<@ULDHJB8BB> <@URAD0CDBK> <@U014B0DC77S> we have a KB published to address the BCR extension upgrade issue.  please let me know if you have any feedback <https://support.citrix.com/article/CTX335487>

The doc references a repair? Is that the fix other than removing it?

It looks like a disable and re-enable could fix it? The doc says this as well.

For those using GPO an uninstall/reinstall should work

yeah looks like I misread the article my bad

latest update on case : there is a v5.4 in the beta store available for testing ...

Citrix has an official article about this issue
<https://support.citrix.com/article/CTX335487>

The article isn't clear enough. Do we still need to remove that directory before re-enabling the extension? Or is simply disabling and re-enabling the extension the fix? Our users can't (and won't) manually repair the extension, so step 2 is out of the question.

As I understand it you can user either the GPO way to disable it or deleting the registry key

Some report they can then re-enable the extension to resolve the issues, others report the the issue persists even after performing the steps on the article

Man, I need to finish my coffee... for some reason, I thought this article was telling us that we can re-enable the extension afterwards.

I disabled the extension yesterday via GPO, which took care of the immediate issue.

Same here, disabled the extension, waiting Citrix to confirm 5.4 is released without his issue.

Disable it or remove it. Then add it back, then it should work. At least it did for me in my lab.

Did you not remove the old extension data location in the profiles?

No, I just did the GPO method of removing it. I just removed the GPO extension, GPupdate /force. Made sure it was removed in the browser. (Assuming it removed it from the Profile) Enabled it back and it worked.

Hmm, gonna see if more people can confirm that. I can't make waves right now (healthcare, new hospital opening).

No, I am waiting to put it back in Prod myself. I am not confident 100% until I can get others to confirm as well.

I was never able to replicate it in my QA environment, which is why I worry a little.

Yeah, it didn't affect everyone here. We received fewer than 10 tickets.

Was it crashing on startup for you, or when you navigated to a site that used it? I just launched Edge, and the extension was at 5.1. Immediately updated to 5.3, but still works.

It opened, then displayed a message stating it crashed. Then about 4s-6s it would close.

Okay, so I still can't get it to crash then.

I am going to put it in my Test Environment in 10 minutes. If that works, I will try it will a couple of users. Then go from there.

I hate not being able to test this. Almost like it's a hope and a prayer when I turn it back on.

we had it in our prod environment but i didnt see it in my lab

Nick, if you want to put some hours on the board. I can come it and make it work in your environment. My going rate is 85hr. But for you Ill do 50hr. HAHA jk.

I worked in a Hospital environment. I feel for you man. Dr's are so mean when you work for them

I wish I could say we weren't impacted or wasn't able to see it in prod - we've been getting wrecked for the past two weeks with related issues.

That sucks man, Sorry to hear that. I know that feeling when you go to bed with a pit in your stomach.

Well, if you get to sleep depending on how big the issues are.

I just check the extensions page every day to see if the updated date changed from 11/30 (it has not): <https://chrome.google.com/webstore/detail/browser-redirection-exten/hdppkjifljbdpckfajcmlblbchhledln>

So, does everyone still have BCR disabled? The extension has yet to be updated in the Chrome store.

Too close to end of year change freezes for us.  If that weren't the case - it will probably be disabled for us until we feel we have a solid reason to reintroduce it.  The impact the BCR extension issues had on our enterprise during November is really going draw more scrutiny for us in the future if we do reintroduce.

Same here - we've been in a change freeze since before Thanksgiving.

We go into freeze Monday, but yea I have it still disabled for now.

I may have to re-enable this before Citrix updates the plugin, and if I do, I'll probably use WEM to delete the BCR extension folder as a RunOnce task just to be safe.

Anyone re-enable this extension yet? If so, did you do anything before re-enabling it? I think I'm going to run that WEM task this week and enable it next week.

Hmm, one odd thing I just noticed is that the version folder in the extensions folder used to be "5.3_0". Testing out my WEM script, and upon launching Edge, I see both "5.3_0" initially, then a new "5.3_1" folder, with 5.3_0 deleting shortly thereafter.

Read comitted snapshot is a option on the SQL Citrix databases as per my screenshot . The databases get setup as part of the site setup via studio or via the DB setup scripts as far i am aware you cant configure the site and setup the databases via the command line using xendesktopserversetup you can only choose components and use /nosql for not installing sql express

Ok... I went wrong in thinking that it creates a read only file on the DDC. But looks like it enables an option on the DB server. This is where I was confused. Is it needed for a citrix site for 400-500 users?

its always been best practise to enable it on citrix DB's

im sure this was already tested, but I would verify network connectivity just in case.

yeah, and make sure that DNS is resolving orrectly

Yea, we verified connectivity, they actually have another environment on the same subnet and that one was able to connect without issue

We tried using IP instead of host name just to check, same error

Nope, time is good. I'm sending something to our network guys to have them check the firewalls… again, lol

can you connect over other ports? like with a UNC connection?

do you see traffic from the source address in the fw logs of the license server?  Is the traffic reaching the license server and being dropped or is it not reaching the license server at all?