World of EUC

Hi Folks. Vad 1912cu1, server2k19, fslogix profile container. I’m testing onedrive files on demand. How do you recommend to  config onedrive cache? Do you use the Storage Sense policies?

i tested it with 1912 VDA /Workspace 1912 (client) and 7.15.2000 DDCs and worked pretty well.  The DDCs arent really needed to make it work other than Studio policies have the option to enable Teams optimization.  the 1912 vda SHOULD have the HKCU key for it already enabled though so as long as you have the MSTeamsRedirSupport key set to 1  under HKCU\SOFTWARE\CITRIX\HDXMediaStream  you should be fine

you use GPU? If not you arent going to get that 24 FPS anways. Its more in the range of 8-10

Have seen lags with enabled EDT recently. Does it make a difference when connecting via TCP?
High latency involved? Try to disable the video codec and see if this is making any difference. 

No GPU on these. Looking at RDAnalyzer, it's jumping between 6-15fps as I work.

We actually disabled EDT a while ago, so that;s not in play anymore.

I wonder if setting the minimum fps to 10 would help.

one interesting thing i read recently was they recommend the "view window contents while dragging" to enabled for better performance.

I do have that enabled because without it, it's impossible to see overlapping windows.

I'm actually going through things right now and am looking at disabling the animations, fades, etc.

Worth mentioning that I'd love to smack whoever thought to include all of these settings (UserPreferencesMask) in a single binary string.

Well, this has opened up old wounds. Even with the EnableVisualEffect setting, I can't get the updates UserPreferencesMask settings to stick past the session. Guess I'll be doing some ProcMon work next week to see what's changing it back.

What VDA are you using? I was testing the 1912 yesterday with 7.15 ddc and since HW acceleration is on with 1912 graphics was horrible, had to revert. You might need to disable it with HDX policy.

<@UL96CM0E6> HW acceleration only does something when you have a GPU. With a GPU it's a massive improvement and as it doesn't impact no GPU scenarios it's on by default. 

<@ULC9J5XPB> can you share a screenshot of RDA? So I can take a look at your settings, latency and bandwidth. 
Also, what's the CPU load on the machine? 

<@ULMD920DC> A GPU certainly helps but even without a GPU you should be able to get decent framerates. Here is a comparison that I did between a CPU only machine and a GPU machine running a 4K YouTube video:
<https://youtu.be/GX0tDG3MWr0|https://youtu.be/GX0tDG3MWr0>

This is a 1912 VDA. I’ll grab a shot of RDA on Monday. In terms of CPU load, it’s basically idle. This is a 4 vCPU/8GB device running on XS with a very light workload.

image.png

<@UNBRPVD4G> (BTW, this utility is outstanding)

Nick what about this <https://support.citrix.com/article/CTX217373>

I've had that set via GPO for a while now. I actually tested changing the HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\VisualEffects\xxxx *DefaultValue* keys, and it seems to work. Not sure if that's the route I want to go, though.

It's just weird that I was able to set this stuff before 1912, but now I can't. The funny thing was it took a lot of testing to get it working in 7.15 - had to use two WEM external tasks. First one was a RunOnce, which set it to one thing. The second was a run every logon task that set the permanent value. 2nd logon would be good, and from then on.

Well, I'm looking at it from two angles. The Visual Effects settings, and Citrix policies. I'm not sure that I actually need to change the policies.

I guess I am trying to understand what where you able to set before 1912?

assuming your meaning the actual VisualEffect not sitcking in 1912 VDA?

HKCU\Control Panel\Desktop\UserPreferencesMask

Basically, this: <https://www.tenforums.com/tutorials/6377-change-visual-effects-settings-windows-10-a.html>

Are you testing with the gateway or storefront?

I've been pounding away at this for hours. Ironically enough, I just moved _SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\VisualEffects\VisualFXSetting_ from HKCU to HKLM, and now the effects are set to Custom. Hoping that my WEM task is now going to work.

The initial complaint is from an offsite admin coming in through the NetScaler, so the policies are a little more bandwidth friendly.

Its Ironic that you mention that, I was going to ask you if the HKLM will over ride the setting. But I was thinking the setting might still be store in the HKCU setting and stay because its per user.

I at least want to set the defaults, so if this works, I'll make my WEM task a RunOnce in case a user wants to adjust it to their liking.

I'm like 90% there. I think I need to adjust my WEM value, because two items are still checked off that shouldn't be.

&gt; So you added the VisualFXSetting as a Dword ( lets say to 3 for example) in HKLM. Then do a WEM external task to run a Script to import the regbinary into the HKEY_CURRENT_USER\Control Panel\Desktop (userperferencesMask) Runonce for default?


Reason I am asking is at times I ponder if I need to do this.

If it retains the setting, it'll be a RunOnce. If it doesn't, I may just force it.

Feel free. If I get this working as expected, I'll send you what I did in total.

Wife: "Why are you still working?" Me: "Because of some jag at Microsoft, that's why"

I can definitely commiserate with the wife asking why I'm still working. Sorry I can't add any value to the discussion otherwise, lol.

Same here man. I going to save this thread and come back in the AM. She just said "I am waiting on you" lol

I don't like leaving this unfinished when I think I'm so close, lol

Hours later, just 10 more minutes..... Them midnight hits

The solution to this may not be pretty, but if it works in the end, I can live with that.

I assume when the admin changed them or u changed them his graphic lag was gone? Well for the short time I mean

I need to have another conversation with him on the actual details, because when I asked him about changing these settings, he said it definitely helped. So I'm not even sure if his actual issue was anything related to graphics or video performance, but responsiveness with simple tasks like menu animation, etc.

Makes sense, it will help with clicks and rendering for sure.

I can watch 720p content on YouTube via XD, and it's not a bad experience - even at 15fps.

Yea that's really good. I'm going to buy the rda tool this weekend from Rody company.

It's very helpful for live testing changes for stuff like this.

Yea, I have a vxrail install with Nvidia Cuda coming up. I'm going to need it. Never did any GPU stuff in xendesktop with Nvidia profiles so it will help I'm sure.

I need to validate these settings over time, but I was able to get 90% there:

Of course, that *Animate controls...* option is one of the biggest offenders. There are separate settings for that one, because the Settings applet can control it, too. I tried a number of different methods, and nothing was turning it off, so I'm at a loss there.

I thought VisualFXSetting"=dword:3 would disabled animate controls. Guess not anymore

Screenshot_20200529-212745.jpg

Well I was incorrect in that last comment. What about this.

Screenshot_20200529-214813.jpg

I’ll have to check, but I did go through each setting to generate ghat binary string, so it should be correct. I would love to find out that it was a typo on my part.

<@ULC9J5XPB> if I look at your RDA screenshot I think you should be able to get a higher fps. 
But in my opinion it's important to enable EDT as it can give more bandwidth during high latency scenarios. 
Why do you have the max fps capped to 15? Do you get higher FPS if you set it to 25-30?

It’s actually a leftover setting from the previous admin, and I think I’m going to change it to see how the UX is afterwards. We can’t enable EDT until after our NetScaler upgrade, because right now Mac clients have massive problems launching apps and desktops with it enabled.

Ok! You could try to disable EDT on the client side for MAC:
defaults write com.citrix.receiver.nomas HDXOverUDPAllowed -bool NO

For ADC I would pick the latest 12.1 build which supports EDT MTU Discovery for windows clients. For others you need to set the MTU size in the default.ica. 

Next version of RDA will give more EDT insight as well like bandwidth and MTU

Just so I’m clear, when you say others, you’re referring to NetScaler builds, right? Or other clients (Mac)? We’re definitely going to at least 12.1, and I was under the impression that we didn’t have to touch the MTU size in default.ica. I’m not the NetScaler guy, though, so I could be wrong.

So EDT uses UDP and by default the protocol can't detect the MTU size properly. So if you have a MTU of 1500 on your network it will send out packets of 1500 bytes. 
That works well if you are internal but it gives issues if the connection goes trough an ADC or a VPN with a lower MTU size. 

As the ADC will encrypt the packet (DTLS) it will add extra data to the packet. This means the packet will be larger than 1500 bytes which leads to fragmentation and the user will have bad UX.

So you can do 2 things:
1. Lower the MTU in the default.ica which works on almost all clients.
2. Use MTU Discovery which Citrix recently released. However this only works on Citrix Workspace app for Windows and LATEST 12.1 or 13.0 ADC build. 

Both options lead to the same results but option 2 is fully automatic and will use higher MTU when possible

Ah, we’re going to have to address Mac clients regardless then, by modifying default.ica. Yet another reason for me to hate Apple.

Actually, looks like we may have to go that route regardless for a while since we’re still on Receiver 4.9 LTSR. Waiting on CWA 2005 so that we can test and deploy.

<@ULLH5JGFR>: You're the man. Changing that 5th value from 12 to 10 does indeed cap off the changes I was looking to make. I have no idea why the guide on tenforums is different than eightforums on this, but at least for that one setting, eightforums is correct.

Nice. I found that in a windows 8 forum. Super glad that helped. I will be doing this as well. But Rody triggers something in my brain. We had EDT and it was showing ica Rtt and ICA latency of 64000. Sessions were laggy as hell only through the Ns gateway. Internally things are zippy and fast. After reading about fragmentation of the packets. I wonder if that our issue for EDT. Support can't figure it out and it's so annoying.

<@ULLH5JGFR> yes that's the behavior you get. Easy to test, use wireshark on remote client and check for fragmented packets

How and the heck doesn't support know this? Would you happen to have the article on this? If not I'll search and find it. Damn man that's awesome that you know that. Well doesn't surprise me after all you have a sweet ass tool.

<@ULC9J5XPB> here is that older link.
Good to keep them handy just in case.
<https://www.eightforums.com/threads/animate-controls-and-elements-inside-windows-on-or-off.39657/|https://www.eightforums.com/threads/animate-controls-and-elements-inside-windows-on-or-off.39657/>

<https://support.citrix.com/article/CTX231821|https://support.citrix.com/article/CTX231821>

I figured it out the hard way around a year ago at a customer. We also had massive issues when people went trough ADC

Rody think this is it

<https://support.citrix.com/article/CTX231821?_ga=2.214286097.703158003.1590614787-436762079.1577280591|https://support.citrix.com/article/CTX231821?_ga=2.214286097.703158003.1590614787-436762079.1577280591>.

And 

<https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/technical-overview/hdx/adaptive-transport.html|https://docs.citrix.com/en-us/citrix-virtual-apps-desktops/technical-overview/hdx/adaptive-transport.html>

Thanks Nick for letting me hop on this thread lol

All of this stuff affects me, too, so I’m game. And yeah, I found that link after seeing your post. I still plan on writing up what I had to do and will post it here when done.

Configuring Visual Effects in Citrix.docx

<@ULLH5JGFR>: Not my best documentation, but I really don't have it in me to duplicate the examples in Word right now. :)

<@UNBRPVD4G>: Tidying this up, I confirmed that we're going to NetScaler 12.1.56.22, the minimum recommended version for EDT MTU Discovery. Still going to have to modify Default.ica, though, along with a number of registry settings thanks to older Receiver clients.

Yes. I noticed this from my testing yesterday.  I think this is going to annoy some of my click happy users and I’ll get shouts of “....Citrix is running slow on YouTube, etc” :disappointed:

Yeah, I've used it in other versions with decent results. if there was issues for a configured site it would fail back to the VDA. This is the first I've tried in CVAD 1912, cheers

Yeah what ^^^^ said - I had a challenge with a bank and it was proxy nastiness 

Also, if a thin client check the resource utilisation on the client side. Seen it max out some older iGels. If its a laptop/desktop of course, shouldn't really cause an issue. I'd be looking at proxy on that front.

Cheers, lads. They push their proxy URL out via GPO, I have tested adding that to the BCR config (forcing server fetch, client render only) and it works as Id expect. There's a few seconds (3-5) while youtube enumerates the home page/thumbnails etc but definitely acceptable for the benefits in performance...

Currently testing 1912CU1 and yes random disconnects for a few users. No obvious trend but when it does happen VDA event viewer shows an ‘rpm’ error regarding session reliability. 

Did you get any clues?

I’m now scared to upgrade to 1912,  in case we are inundated but the noise for Teams is unbearable :disappointed:

Yes there is a problem with Windows CWA &lt; 2002 and Windows 10 1903/1909 i believe - when we upgrade the clients to CWA 2002 the problem seems to go (or at least, reduces enough that they stop complaining :)). Will try and get a link.

Check out issues 7/8:

<https://support.citrix.com/article/CTX238758>

<@ULTGG1KDJ> great stuff.thanks. :+1::skin-tone-2:

Yep, Citrix changes defaults, which is why I started tracking the changes. Some of the default changes are head scratchers and sometimes Citrix changes the name of a policy setting. 

Not just that the defaults change, but that it's controlled/set by the VDA NOT the Site.

To me, that is clearly stated because the settings apply to vda versions if you look at the policy help text. It never says applies to a site or controller version or database version

For instance, if you have a 7.15 site and are upgrading to 1912, and upgrade the VDA fist, your ICA files will now have HDXoverUDP=Preffered, even tho the default of the 7.15 site is Off. This can cause connectivity issues if you are using 4.9 Receiver thru a netscaler and the netscaler had DTLS enabled but session reliability disabled.

I assume you're just going the HKLM key route and killing it for everyone?

What about fallback mode? Leaving the default at 0?

I don't suppose it's possible to disable this warning?

That I haven't found. And I know that we are controlling those settings, and yet I searched my GPOs and WEM yesterday and can't find where...odd.

Damn, if you come across it, I'd appreciate it. I haven't been able to find any setting to disable it. I thought maybe disabling the balloon notifications (via HKLM) would do it, but nope.

It's weird, the balloon settings listed on <https://docs.citrix.com/en-us/hdx-optimization/current-release/configuration.html> don't seem to do anything on my W10 VDA. Even without those values set, if I kill MediaEngineService.exe on my machine, I can see the icon change in my VDA, but no balloon message ever appears. Does HDX RTOP 2.9 actually move those messages to Action Center? That would explain it for me, as I have notifications completely disabled in my image.

<@ULHA2PS2H>: Any chance you were able to find what controls that "Skype for Business is running locally..." warning? I'm striking out.

Hopefully they see this, but somebody here wrote a PS script to handle it. I'll see if I can find it.

<https://worldofeuc.slack.com/archives/CLSHNTWLW/p1588089084055500?thread_ts=1587718110.048800>

maybe this if (test-path C:\Users\*\AppData\Local\Microsoft\Teams\current\Teams.exe)
{kill -name teams -force;
(Get-ItemProperty C:\Users\*\AppData\Local\Microsoft\Teams\Current).PSParentPath | foreach-object {Start-Process $_\Update.exe -ArgumentList "--uninstall /s" -PassThru -Wait}
}

I have a more "controlled" way of switching between per-user and per-machine Teams:
<https://virtualwarlock.net/microsoft-teams-in-citrix/>

Ahh I meant to ask, have you tried teams as a published app yet?

When you promote it from studio, The MCS machine will ask you to appliy now or schedule reboot to apply. Think this is what you're asking.

The VMs must reboot at some point to get the updated image

That makes sense for VMs that are powered on already - or does it mean even a VM that's been in a powered off state must power on, get updated, and reboot?

Yes. It has to be powered on from studio (only) to get the new image. Powering on at the host won’t trigger the vm reconfig. 

So let's say I have 1000 vms but 500 are off. I update the image, and then go to push out the update. Will the 500 that are powered off need to do a reboot to get updated on do they get the new image on next boot?

The powered off VMs will received the updated image when they are powered on either by manually powering them on in Studio or they are powered on as part of the catalog's power management settings.

They will get it on the next boot. MCS snapshots your master (or master snapshot) and copies it to storage creating a new base disk for your catalog. The catalog doesn’t really care about your master image/snapshot anymore after that. Not that I would recommend deleting it.

If you do power management from the host, the VMs will not "know" to get anything from Studio and Studio will not "know" how to tell them there are changes. Restarting a VM from the host also does not clear the differential disk

My understanding is if you want to revert an image, you need the prior snapshot

Got it. Just wanted to make sure a single specific reboot was required and that all of the image update stuff happens purely on the storage side

sorry, <@UL8ACTGSF> is right. You need it for the rollback. I was just saying in theory you could delete and it wouldn’t break a non persistent catalog.

You can use get-provvm cmdlet to check the “assignedimage” for the catalog and the “bootedimage” that vm is running off. If these have the same guid then you know the vm is on the most recent basedisk.

Get-ProvScheme -adminaddress DDC-machine-name | select ProvisioningSchemeName, masterimagevm

ypu can also look for the pending update flag

not needed anymore. you can run the cwa installer with /forceinstall

<https://www.citrix.com/blogs/2019/10/02/still-using-receiver-cleanup-utility-its-time-to-move-on/>

It was a last ditch effort, as /forceinstall is also failing. Something must have gotten corrupted in my Platform layer (App Layering), because I was able to create a new one and install CWA 2006. I just didn't want to go that route.

I went as far as to uninstall VC++, .NET, run RCU, and then reinstall all... nada. Same error at install every time.

I'd recommend to wait with CWA 2006, there might come a fix very soon...

I saw the same thing with CWA 2002, and am about to try 1912 just to rule out one of the newer CWAs.

just saying: CWA 2006 has an issue and they are working on it

detection of the CWA doesn't work if going through gateway to a Storefront Webstore

1912 LTSR failed, too, so it's definitely my layer. But that issue you posted is a hard stop for me, so I appreciate that.

I swear to god we're never going to be able to get off this god damn receiver client at this rate.

Out of curiosity, how consistent is that issue? We have that same setup, and I haven't had issues with client detection on CWA 2006 yet.

Not sure what exactly are the parameters to trigger the issue. But I worked with the engineers this morning and they just informed me, they identified the bug and therefore stopped auto update. So I suspect it might be an issue for a couple of folks.

They literally _just_ pulled the download and docs for 2006.

Progress. Tried building this layer based on a different OS layer, and was able to get it to install. Looks like my latest Windows Update revision OS layer is causing the issue.

<@UL5TD2QM9>: Heard that the fixed CWA will be released tomorrow.

1906, 1909, 1912, 2002 &amp; HTML5 Receiver. All affected.

Yes, I can create a new Master Image with CVAD 1912.1, but this would take a moment..

Was completely solve by turning of HDX Adaptive transport (EDT aka UDP) and back to TCP.

Okay..but I am doing the POC this same version, I didn't face the issue....

Hey <@ULHA2J7RT>,

do you have any News on that issue?
Have also troubles with 1912 CU1 and Grey Screen on LogOn with Server 2016 and 2019. Only a reboot will fix it.

<@UR9T47CRM> you also disable EDT?

I solved it by disabling EDT about 4 weeks ago. Error is gone. 
Completely forgot about this thread here. 

Yes i disable edt and it works then :innocent::sweat_smile:

The error returned 5 days ago. Same customer, after the error was gone for 5 Weeks!
I haven't analyzed it, yet.

So I double checked today. It's 100% TCP and exactly the same error. Have no idea what's going on here.

Hey <@ULHA2J7RT> , we‘ve now created a new citrix case. We saw errors in CDF Traces. 

Which exact VDA version are you using on which os?
I use VDA 1912 on 2016. 
currently building a test for 2006 on 2016

Server 2016 w 1912 cu 1 
Server 2019 w 1912 cu 1
Server 2019 w 1912 
Issue Happens on all setups. 

So if we really have the same issue, that means I could skip CU1 tests and do 2006 tests instead. 

I think its the same. I will let you know any Updates on the CTX Case. Webex on Thursday 

I also talked to the customer, that after the 2006 tests, we should open a case.

If it will be possible take the affected VDA in maintenance when the issue occurs. The you could collect the cdf traces and simulate logons on the affected VDA  

I got the same issue on a customer site too. It seams that something is causing the CtxUvi to crash. We could narrow down that the “winlogon.exe” from the particular user with the “grey screen” issue has none Citrix dll’s loaded. CTX220418 guided me to on the right track i guess… Now we’re investigating what possibly could cause the driver to crash.

According to <@U015G4SG5LY> Sophos chrashed their CtxUvi. Do you also use Sophos?

TrendMicro DeepSecurity - I will have a closer look at this…

Which ones are you referring to, the one to log a user in and then back out?

So, I have the local user login.. It opens the applications.. but when it logs off it seems the vda is acting in the non-persistent way and deleting the vm.

OK, you would need to check what the behaviour is set to for when user logs off, is it set to shutdown on logoff?

I would turn that off then if possible, not sure if it is possible to filter it in some way or maybe exclude some users?

we never implemented it, but our thought was to use a script that would do the auto login, then log out, but the citrix desktop service set to manual and it would start it after the initial login, then it would register. That was how we were trying to get around to reboot

<@U01258VQW56> i've tried that and it still triggers a reboot as soon as the service starts..

Maybe the hydration feature from BISF can help!? <https://eucweb.com/kba/281219084515-2|https://eucweb.com/kba/281219084515-2>

You might need to set the service to disabled on the master  and start it with the auto login script.

Ive done that. The issue is its a non-persistent image and the catalog is set to reboot on user log off.

So yes, I had to set the services to disable on the master. Broker, ICA, and WEM or I would get the phantom " - " user, which would trigger a reboot. In the end I created a scheduled task that the login script calls, which has a delay of about 1 minute. Desktop logs off, scheduled task executes a script to enable and start the services. My login times went from 120 seconds down to 70 seconds... this is all hosted in AWS.

our logins take around 260 seconds that includes streaming down additional apps via APP-v, though 4vcpu of which only 2 out of 7000 users have access too it's super quick

are you doing ssl inspection with the proxy?

we ran into something where webex uses the cloudflare javascript cdn but doesn't tell you anywhere you need to whitelist it

run wireshark and look for ssl handshakes failing

just did and noticed "different webex urls" not added  to bcr ... maybe that is the reason (ex. we add <http://company.webex.com|company.webex.com> but the trace shows <http://akamicdn.webex.com|akamicdn.webex.com>, cdn.digitaladopt.webex.coim etc)

yeah, you basically need to whitelist all of <http://webex.com|webex.com>. <https://help.webex.com/en-us/WBX000028782/Network-Requirements-for-Webex-Teams-Services#URLs>

you mean whitelist on proxy to avoid SSL inspection or add to BCR list (that seems to be limited to 989 chars)

don't do SSL inspection, it really doesn't play nice

I would recommend at least Citrix Workspace App, have seen many performance issues when using Citrix Receiver..

it looks like this issue has to do with audio redirection. looking into ways of upgrading the client on their thin clients now

i've had a similar issue on HP thin clients and had to disable hardware acceleration for Receiver locally on the client to get around it

sorry with older Receiver .... 4.12 i believe

i can get details what we did if you need

you could do it w/ local gpedit too but this should do it i think as well

so in our case we had to disable it. i cant say what downside there is to having that off but for us not crashing was better

it was happening several times a day for those affected i remember

<@URVNXB96C> The downside of disabling Hardware Acceleration is that if you use a codec which uses H264 (Like For Actively Changing Regions) it will not offload it to the GPU on the endpoint. So if you have a old or weak endpoints which has a tiny CPU it will cause performance issues. Most endpoints do have built-in capabilities for H264 offloading

<@UNBRPVD4G>  that makes sense.... That only applies to the endpoint right? What about things like disabling gpu acceleration in teams on the vda which is a recommended thing I guess

Yes it only disables it on the endpoint. Teams does rely quite heavily on the GPU. Haven't tested it without a vGPU yet to be honest.
But there is also the option to offload it to the endpoint (Full support on Windows Endpoint, Audio Only as Experimental on Linux)

Yup that webrtc optimization we're starting to test now

about BCR (part 2 :slightly_smiling_face:): according to <https://support.citrix.com/article/CTX230052|this kb> section "feature limitation" :  there is a very annoying item about DPI scaling (_...client endpoint GPU needs to be disabled if DPI scaling factor is set to a number other than 100% in order for BCR feature to work.._). For homeworkers on private PCs it  is "impossible" to ask them to edit the registry. Anyone knows if this is solved/going to be solved in near feature ?

Sure sounds like it. I’m more concerned with the fact that users seem to be able to launch their desktop twice (one in each site). Their FSLogix profile is locked on the first, so they get a temp profile on the second).

The only thing I am finding around this is back in the IMA days of Xenapp 6x

its not showing the desktop twice is it in mult site aggregation?

With desktops not so much, but we can reproduce it easily with published apps.

Ray: Nope - SF shows just a single icon for every app and desktop.

Check all your published apps for zoning and recheck aggregation. Check Storefront logs too to make sure its not having any issues.

We’re a few versions behind in SF. We have so much customization done to make it look like our old WI (decision made by morons), that I wonder if we’ll need to upgrade.

I'll have to dive into StoreFront next week. We just have a single zone containing DCs, hosts, and MCs; no apps.

We do have some application groups, but the apps we were testing were not part of any groups.

And I honestly have no idea how the original web.config was configured for aggregation. Scrolling through it, nothing looks out of place.

&lt;add param="controller" value="DazzleResources" /&gt; Dazzle? lol, man, that's never going to die.

I'm sure u know, you can look in the SF console as well and check the GUI as well.

We've never been able to view it in the GUI:

Man, StoreFront's event logs are impossible to go through right now because one of our datacenters is down.

Poke around in this link from Carl.  Guys talk about diabling session sharing but he send him to multiple session settings on his page. Looks like you can ha e multiple sessions now with powershell. Just in case it got flipped on somehow. <http://www.carlstalhood.com/catalogs-delivery-groups/#sessions|http://www.carlstalhood.com/catalogs-delivery-groups/#sessions>

Do you have DDC spread across different datacneters and they are in 1 Zone?

Nope, all DDCs are in the same DC as their respective site, all in the same zone. We basically have two side-by-side Citrix sites, identical in every way.

I know in app groups you can disable session sharing
 But I don't know if it's possible in delivery groups. My thinking is around maybe it got turned off during the upgrade. But I'm stretching here

Session roaming is set to always for all of our apps. I don't really think that's it.

And dumb question, the apps they are launching is all in the same DG. Meaning user session is on this DG for this app A but app B isn't there so new session is created on DG for app B.

Yep - apps are a part of the same DG. The apps we tested are actually only available on one server in each site, too. First launch goes to server A (site 1); next app goes to server A (site 2).

Am I just misremembering things? Do apps actually have to be in application groups in order to enable session sharing?