https://worldofeuc.com logo
Join Slack
Powered by
This message was deleted.
# citrix-vad
s
This message was deleted.
r
This doesn't help you, more of brainstorming. I wonder if they perhaps didn't include the hotfix from the previous versions, or another undiscovered issues? This maybe extremely silly, what about trying the hotfix( not for 2203 though) just to rule that out? Not sure how much work that would be on you or if it's even an valuable option/that makes sense.
j
So the hotfix ...the .exe is numbered xx.2000.xx whereas the CU1 .exe is numbered xx.1000.xx, but they swear its included.
And I have raised this thought to our team. We may try both hotfix on one image and 2209 on another to see if its better on either.
n
We use ControlUp to keep restarting the service when it crashes
j
Well 1.) How do you know the service crashes. We don't see any events 2.) Isn't that impactful for folks there in and working on the same host?
n
No one has complained. The process kept stopping randomly or when some users log off, I opened a ticket with Citrix and got nowhere. We use a script action to detect if the WebSockets exe is running or not and just quickly restart if it’s not.
It was really bad in 1912 VDAs
j
Okay thank you
So we tried rolling back to 2203 base. No luck. We tried 2203 with the hotfix for the websocketservice.exe (We think thats a different issue), no luck. We still consistently have users unable to join meetings and it appears websocketagent.exe is missing for them. We are trying to see if we can connect a particular version of workspace paired with 2203 maybe. Our Win 10 20h2, single user desktop environment with 1912 CU5 is rock solid. Another odd thing is we have rolled back to the VDA and Teams version before we started having issues and it doesn't seem to help. That makes me think either something else in our environment, or a recently enabled or changed feature flag in Azure.
We got a private hotfix for websocketservice.exe and websocketagent.exe that another customer had, with the EXACT same issues detailed, but it seems to have not helped. I really feel like this is a feature flag enabled on MSFTs side.
r
Nice.
I wonder if that is going to be another known issue
j
I think so. Annoying as crap. The support level has been really bad too.
Can't get a response. We bumped it to Sev 2 weeks ago
Our TAM seems to indicate it's a larger lag in support turn around times
r
I have hit and miss with Support. But it's like that for most companies now. Expect for Nutanix and ControlUp. They are on point in my experiences.
j
So now we got a 3rd hotfix from Citrix, for WebsocketAgent and WebSocketService.exe lets hope this helps.
b
We opened a case for 2203 CU1 as well. OS is WServer 2016, upgrade to WServer 2022 is planned.
👍 1
n
Do you see it actually crashing in the system app log?
j
Nope. We also did procdump to monitor it and no crash generated.
n
We might be see something similar, have to restart Teams several times a day on our RDS 2203 CU1 server
j
We do see HDXRTCEngine log a broken pipe error on the client side but that seems expected.
n
What about the server with the WebSocket, is there an crash log generated?
b
Nope. Process is just disappearing
j
No not at all. So we know a user is busted if WebSocketAgent.exe is missing. We CAN restart WebSocketService.exe and it will respawn the WebSocketAgent for everyone, IF we can't get support we may roll back to 2112 since it still supports multi-window and blurring stuffs. But from what I see on reddit and this thread it isn't just us--which makes me feel better. We got a 2nd hotfix (assistant to the hotfix I guess), on the box now doing calls okay, but its so random.
b
The strange thing is, we've a lot of customers on WServer 2019, VDA 2203 CU1 and almost none of them reported this issue. This is the first one
j
So we have Trend Deep Security, and that's about it for security. We use McAfee proxy and we do have some random DNS issue with it right now, but don't think its related.
b
This customer is using Sophos AV. No Proxy.
j
I personally think that Microsoft has rolled some feature flag to a ring or something. But we've been having issues for about a month
OR maybe we have a very varied workspace version set out there. So we wonder if maybe a particular combo of workspace version and whatnot happens. Do you have a case open with Citrix @Balint Oberrauch? If you don't mind DM me the number and we'll ask our TAM to maybe compare? Up to you no pressure
r
I personally appreciate @Josh Jordan and @Balint Oberrauch updates on this.
j
No problem Ray! Also, we have a non persistent 1:1 desktop environment with 1912 CU5 without the issue happening. We have tried 2203 with and older version of teams with no luck either.
b
@Ray Davis I feel like a beta tester in the last months... @Josh Jordan, sure DM is out
👍 1
j
No change in behavior with our 2nd private hotfix
Guess we are gonna try 2112
So 2112 VDA not working either. So maybe just all multi-session VDAs affected by whatever this is
So update, we think this is an issue with BCR in Chrome.exe. So if a user opens Chrome.exe with BCR plugin enabled, it kills websocketagent.exe for ALL Users on that box. Edge doesn't do it, with same extension. We removed the extension via GPO for Chrome and seems to have resolved it, more testing needed.
d
@Nick Patel Can you share with me a guide or some quick steps to utilize controlup to restart that service? Our Organization has it but its barely used...i know tragic. One of my goals is to make it critical
🤯 1
n
d
ty will look into it
n
Their support is good so they can probably point you in the right direction as well
@Josh Jordan are you still stable after disabling Chrome BCR? We’re having all sorts of Teams issues too with crashing and calls not connecting.
j
Yep.
n
Ok, will test your findings in our environment and see if the behavior is the same.
j
So both BCR extensions (Edge and Chrome) use websocketagent.exe to handle it, but Some Chrome update 105.x and higher added a new SSL flag, and BCR extension does some .js injection and doesn't like the handshake or the flag..reproducable every time. Key is user has to have hit a BCR enabled site at some point I THINK. But after that EVERY time we open chrome.exe it kills websocketagent.exe for EVERYONE on the box.
👍🏽 1
So yeah we push that BCR extension via GPO so we just disabled it (Chrome only) no need to disable BCR policy
and it fixed it. We haven't had any complaints since.
n
We do the same
👍 1
j
They are supposedly adding it to the teams redirection page with all the current notes of issues / gotchas
This page, its not on there yet
n
Cool, thanks for the update.
I’ll let you know if we have the same issue
j
Yeah it was incredibly frustrating we tried 2112, 2203 base, 2 hotfixes, and finally it wound up being BCR.
Users were beyond angry
n
Wouldn’t be the first time BCR blew up something
You remember that crash they had a few months ago
j
No I don't. I do know that our team mostly uses Edge, so we never had the issue at least unless we were on a box and someone else happened to open Chrome
But our task workers use Chrome non stop
n
Is WebSocketAgent supposed to be running under each user account? On our server I only see it running under 1 user, wonder if it’s because it was killed for everyone on the box
j
Well that means those other users are busted
yeah if they close and re-open teams itll work
n
Ahh yeah
j
But as soon as someone on the box opens chrome its game over again
it breaks BCR too, but we only have BCR on a small handful of non work related sites. YouTube, FoxNews, CNN, Amazon, Reddit, etc
n
Hmm I just tried YouTube on Chrome and it didn’t crash it
BCR is enabled
We’re going to disable it anyway and see if it’s more stable
j
If you have Chrome 105.x or higher then I can bet it will be
Try closing and re-opening teams
n
We’re on 106
j
not teams, sorry chrome
close chrome, and then when you reopen it it will break you
like if you are in a call, it will kill it immediately
or if you go to make a call afterwards you will get "connecting..."
n
Yeah we definitely have that behavior
Do you have to be on the call for it to trigger?
j
Nope, but its a good way to see it happen live
You can watch websocketagent.exe go away for everyone typically
n
I can’t trigger it, are you using workspace app 2210?
j
Yes. It happens regardless of workspace version. Serve 2019, MCS, 2203 CU1
I have 2210 on my personal PC and it does it too
n
Ok good to know
That’s our same specs as well
Ok we disabled BCR and will find out tomorrow. Every server that has 2203 CU1 has the process missing for the majority of the users on that server.
j
How long do y'all keep sessions up?
We do week long sessions (if they log in daily)
n
3 hours idle, 3 hours disconnected , we had to decrease when the pandemic started
Probably could change it higher now
j
n
So looks like we can leave BCR enabled with that policy in place
j
Yeah. We may switch to this method later
n
We’ll test it out :)
d
Can’t wait to implement this fix. Thank you guys
no user complaints today
n
Yep, same here
Open in Slack
PreviousNext
Powered by