https://worldofeuc.com logo
Join Slack
Powered by
This message was deleted.
# citrix-vad
s
This message was deleted.
s
you have a case open and they won't give you the private hotfix?
a
i've been trying but haven't gotten a response since the first initial email
s
if you call in and get a warm body they should be able to. Email doesn't always work well.
a
our error codes are different everytime it crashes (and not the specific one called our in that article) but definitely started w/ that 2203 cu1 agent so hoping its the same thing
good idea....i'll try that!
it appears i dont have much luck w/ the warm body approach either 🙂 they wont provide me the fix until i send them a full memory dump to analyze first
yup they won't even discuss the possibility of giving us that hotfix until i gather that full memory dump, upload and give them some time to analyze
r
I was thinking the same thing @Steve Noel said. Tell them to give it to you to test atleast. Geez
Call and tell them you are about to have a stroke and you need it now haha- not really
😂 1
a
i tried everything i could on the call .....including telling them i have a very easy way to test it but they wont help....keep saying it would need to be escalated so i said please do that and they said nope, they dont want to risk our environment w/out analyzing first. i totally get that but like you said i can do a quick and simple isolated test and also wait for their analysis before pushing out to production
r
I love Citrix, but support is like a maze at times. you learn how to navigate it. But it's not just Citrix now a days. From my experience the best support is CU and Nutanix. hands down
WOW, dang man.
a
yup i agree there......CU and Nutanix are usually rock solid. Citrix has been nothing but a hassle the past couple of years for us. its a running joke if i have to open a ticket, its gonna be a good 2-3 weeks of jumping through hoops before we even get someone that is useful
r
They are probably protecting themselves in case of a CYA.
I can understand that.
a
yup thats what it is
r
From what I hear, they are going to get them in line lol.
a
the funny thing is they said if i upgrade from LTSR to CU the fix is in there and i wont have the issue......soooo why the hesitancy giving me the private fix to test then since we cant go off ltsr
r
hmmm interesting.
a
it was pretty clear the tech was going off the script and whomever he was checking with wasn't budging
r
Yea, probably them doing CYA. Hopefully they get it sent to Escalation and they are usually so much better.
I try to put myself in their shoes and think of what they have to follow.
a
they would not send to escalation when i asked. they said they won't do anything w/ the case until i provide a full 8gb memory dump
r
OH
a
oh ya i'm with ya there......i know the tech was just doing what he could. did not appreciate it not being able to escalate though. nutanix they even encourage it and give you the option on the user end
r
There is a button in support case side (in the portal) on the Citrix side and say escalation or grab managers attention
a
oh is there? damn i missed that! i'll definitely try that. thanks man
r
But the next problem is.... I bet alot of folks do that for non issues. Just to get faster support.
a
and most likely all its going to do is get them to bombard me w/ phone calls more often and still not do anything different i'm afraid
j
the button has worked for me
👌 1
i've used it when i have to pivot to 'dude, i'm telling the first guy that picked up the ticket how it works...'
⤴️ 1
f
We have it, was pretty random, deployed on production this week, no BSOD so far
a
So far today my tester has crashed 3 times and each time it's a different crash code but started immediately after the vda upgrade and never before. I'm afraid they are going to look at the crash dump, not see that specific sys file called out in the kb and still not help
s
Rat farts. is it worth trying to downgrade or upgrade to the suggested VDA? if not, maybe DM @François-Xavier Rigaud and see if he can get the hotfix?
💯 1
⤴️ 1
💪 1
a
I was going to try the downgrade actually to test so good call there. Thanks for the reference...I may reach out to him!
r
Lmao “rat farts”
b
go for 2209. you have to turn on testsigning mode which leaves a watermark for the users. and I'm not sure how stable the fix is as we got 2 different versions
f
@Balint Oberrauch Ask for the safe version of this fix, support send me one without this requirement
a
I'd imagine the fix is just a replacement dll or sys file maybe? I'm hoping my issue isn't something different side I'm getting different blue screen codes each time
b
We preferred to upgrade to 2209 and revert to 2203 CU2 when it's released. no ETA for CU2 when I asked the last time.
a
Are you using the 2209 VDA with 2203 CU1 controllers?
b
yes. not supported but sometimes you have to make things work.
💯 1
a
So it's supported to have an older vda than your controllers but not newer?
b
one version behind is allowed if i recall correctly but you cannot mix CR and LTSR.
h
Same problem here. We also updated the VDA to 2209 while the environment is still running on 2203 CU1. We hope that this will make our environment more stable, because we have many BSOD every week.
b
j
I'm not teasing you but I got the private fix, they also wanted a full dump memory, I replied something back about the fast it impacts production and it's tricky to get a full one with PVS.
a
I said more or less the same thing and still just waiting on them analyzing the memory dump for the past couple of days . I assume it was after they did that for you they then sent it?
j
No they sent it without me providing the memory dump.
a
you must have said some magic words haha
j
Not sure if it's still a thing... but there used to be a "politically hot" escalation form. I wouldn't be surprised if it has been retitled/reworded.
a
meaning if you say the magic words ? lol
j
You have to state that the bsods are impacting your users day to day ops so much (and you're getting pressure from leadership) that you need the Hotfix before you can repro on demand/create a dump.
a
good to keep that magic trick in your back pocket when needed!
j
That's kind on how I phrased it.
r
@jvitech could share it with @Andrew Gresbach 🙂
But could be a risk 😞
j
Indeed and maybe that's what I already did ;)
r
🙂
a
🙂
Support is claiming my issue is an issue with Windows defender and I should disable part of it or contact Microsoft... And also can they not archive the case.....umm nope! The issue only presented itself after the 2203 cu1 agent install (fresh install after using the vda cleanup utility first) and on multiple unique images. As soon as we uninstall that and put 1912 back on we're stable again
r
Now you can try a vda with CU2. Wack defender for testing and if it happens, then get on the horn with support again unless cu2 fixes it
a
nice! did not know cu2 was coming that quick! good timing!
thanks for the heads up @Ray Davis
r
No problem man
a
well no luck w/ CU2 unfortunately.....i've crashed 2 times this morning already so need to upload another crash file to them . but at least it gets me ahead of the curve for when they will probably ask me to update and test. funny enough the blue screens have a different crash reason every single time......but on the 2 desktops that we've been testing they've been rock solid for 2 years until this update and if i remove it and go back to 1912 its solid again
interesting.....i found this as a known issue in the original 2203 release notes . i'm pretty sure everytime we run into it , its when using Edge in some way (not entirely reproducable). but i cant find reference if it was fixed.
👀 1
actually that is listed as a FIXED issue .....it would appear thats not the case 🙂
well Citrix is digging their heels in saying its not their issue and I have to contact Microsoft....even though every crash has a different offending file w/ the blue screen, most of the time something Microsoft and one time it was even our Nutanix network driver (all of which have been in place and rock solid until the 2203 VDA update (and again revert from 2203 to 1912 and everything is stable again).
p
@Andrew Gresbach Did you have any luck with your case? We have also deployed 2203 CU2, BSOD are happening less frequently but are still there.
a
So how it was left was the Microsoft ticket/analysis SEEMED to find the picadd.sys from Citrix as the offending file for at least one of my crash files that i submitted . I sent that info over to Citrix and still waiting to hear back. On the Citrix end they told me that they won't even try to involve engineering until Microsoft and Microsoft alone can prove it was caused by a Citrix component unfortunately so its taken quite a bit of extra work. Hoping this gets the ball rolling! I believe @Nick Panaccio is also dealing w/ a similar issue and has similar tickets open so hopefully between the 3 of us we can get this one figured out (assuming all 3 of us are dealing w/ the same bug of course)
p
Thanks Andrew. Also not sure if all of us are dealing with the same bug. These are the latest BSOD:
a
hard to say w/out a detailed analysis i guess but I know I've gotten similar ones w/ kernel crashes (each one seems to call out a different reason at least with my more basic tools)
n
Man, I have seen the same exact BSODs here with 2203 CU2.
What app is that in the screenshot?
p
n
Doh, I have that already, lol
a
i've used that one as well as WinDbg
and WhoCrashed i think too
n
Same.
I use WinDbg for most initial analysis.
a
yup i've moved to that as well....pretty powerful and simple
p
I also tried those tools but none of them gave any helpful information.
a
unless you ask Citrix and in that case it wont help 🙂
n
I have tickets open with MS and Citrix, and have provided my crash dumps.
My crashes all seem to be triggered when I join a Webex, or in the middle of one. But I also think there's something going on with the virtual channel. It usually fails and closes at some point, where my Webex is unoptimized.
a
for us we've seen it when working w/ Chrome or Edge i believe (working w/ a console such as Vmware or Nutanix) but haven't pinned it down as much as Nick. Pretty random for us (but frequent enough)
n
I have seen EdgeWebView2 cause it, too.
But that could be any app, not just Edge at this point.
a
true so i guess that goes along w/ what we've seen. At any rate , Citrix claim that "we are the only customer that has reported this" doesn't sound like it holds up 🙂
n
Tell them I said they're full of shit
😅 1
a
"Nick says hi " 🙂
n
No, I said they're full of shit...
a
haha oh i'll add that in too
"you know what i've been telling you for weeks, Nick would like to add the same sentiment"
n

https://www.youtube.com/watch?v=1lRIQGU2RRk▾

🤣 2
p
We archived our case with Citrix after CU2 came out. Citrix was asking to provide a full memory dump but we have physical servers (HP Moonshot) with 128 GB RAM and we don't have the neccessary free disk space for that.
n
We have automatic memory dumps in our VDAs. Pretty sure that's enforced via policy, too, so it'd be a pain to change.
a
all good! sounds like we've submitted plenty for that!
💯 1
Citrix has been telling us for weeks that they saw nothing Citrix related in the crash files we submitted (the same ones we submitted to Microsoft) but its probably because only Microsoft has the full suite of symbols to analyze the crash files fully. I had no idea that anyone NOT Microsoft was unable to do that
n
Hopefully I can get MS to analyze our dump files. The guy assigned to my case is still trying to figure out what 'schedule a meeting' means, so who knows.
a
well that's not promising 🙂 i think i got lucky and had our MS rep escalate ours right away so maybe we'll get things moving quicker
n
Citrix got back to me saying that it could be a common problem like a memory leak, hardware failure, or corrupt software. Thanks Mr. Helper.
a
Yup.... corrupt software.... YOUR software
n
Shit only started with 2203, so I have a hard time believing that they play no role in this.
a
Exactly... I showed them numerous times that if I install 2203 we get the crashes... And then if I go back to 1912 zero crashes and so on. So yeah I don't buy them saying they don't think it's them
n
I will say the analysis I was given pales in comparison to what you were given.
I don't think they did anything more than what I did - load WinDbg and run !analyze -v
a
oh ya thats what i got from Citrix too.....the one i sent you was from Microsoft
n
Ahhh, okay. Hopefully I get something similar.
a
When i first started battling w/ the Citrix initial ticket they saw Defender was involved in some way and told me to disable that and contact Microsoft lol
n
Meeting with them tomorrow, but I assume it's just to go over what I'm seeing. He did run WinDbg quickly to tell me that it could be Tanium, but I had already uninstalled that.
a
until i sent them another one and the 2nd one showed a Nutanix driver
n
So your faulting component changes every time?'
a
ya i was pretty disappointed at the involvement from Citrix. In the past they'd at least pretend to try by asking you to do a Scout log or something
n
And is it a similar IRQL_LESS_THAN_EQUAL crashes like in my screenshot from today?
a
yup!
i get ones like that too yup pretty sure. i have a bunch saved, i can load them up and see
n
I asked them how they wanted to troubleshoot the picadd stuff. I still think that may be related.
The virtual channels suddenly being blocked mid-session make no sense.
a
ya i agree. i emailed Citrix again this morning since its been 6 days since i sent that info and they've gone silent on me so no idea if they even started to bring engineering in. i may have to escalate the case again
friggin Citrix ! finally hear back after a week when i sent Microsofts analysis and they asked if i tried starting w/ a vanilla image yet because the Microsoft analysis looks inconclusive?? it specially calls out their picadd.sys in a few spots. thats far from inconclusive....at the very least i dont understand why they cant engage w/ an escalation team or engineering. I did mention i've talked with a couple other Citrix administrators dealing w/ very similar issues only on 2203 so we definitely are not the only one to report this like they are claiming. ok vent over 🙂
👍 1
p
I noticed that sometimes the server is unresponsive for quite some times (even hours) before it actually crashes with a BSOD. Did you notice the same behavior?
n
Not in my environment, no. It locks up and crashes immediately.
👍 1
a
yup same here.....when we crash its pretty immediate.....i'll keep a 2nd vm session going on a different desktop so i can watch my console window to catch the crash code. just a matter of waiting for the full crash dump to complete and finish the reboot before i can get back in
p
ok thanks. Maybe todays server crash was something completely different.
a
i maybe should have given it the night before i replied to my Citrix ticket.....didn't pull any punches on my reply 🙂 BUT after 5-6 weeks and 2 techs now of the ticket being opened patience was wearing thin
💯 1
to me even if the initial findings don't specifically call out a Citrix component, if a customer can easily prove that one version gives you a solid environment and JUST upgrading that one component to the new version you have frequent crashes (and then at the same time rolling back restores the stability that seems pretty clear its at least SOMEWHAT due to your new version and you should work w/ them to figure it out . Really don't like this approach of basically making you w/out a shadow of a doubt prove it w/ other vendors first and point to exactly what of your own software is the issue before they'll even involve engineering
So Microsoft has me trying the driver verifier for all NON Microsoft drivers . Unfortuntely that seems to crash the VM immediately after i launch a ICA session (for both 1912 and 2203 agent versions) so not sure that is going to give much better information. It also seems to max out the CPU in both scenarios
r
Non driver verifier? What is that?
n
Interesting. I ran their TSS utility to send them logs, so they should be analyzing our crash dumps soon.
Funny enough, I haven't had a crash in a few days now.
Newly upgraded 2203 VDA I'm testing just crashed in a Webex meeting, lol.
That crash has CrowdStrike listed as the offending module. We're supposedly in audit only mode, though nobody believes that means it's incapable of crashing stuff.
s
I really hope Citrix fills their QA positions with folks like you guys...
😂 1
a
no i meant they want me to use the Verifier tool and enable it for all NON Microsoft drivers (so 3rd party only)
👍 1
j
They already use all their customers as a QA team! Anyhow private fix with CU1 worked for me.
n
Private fix for BSODs? Was it incorporated into CU2? If not, can you provide details/the private fix #?
a
ya any details you have on what you were dealing with would be great! i know there was a known BSOD in CU1 that had the fix rolled into CU2 so curious if you had to do CU2 and the private fix
j
Ouch Nick, if you have to engage CrowdStrike support... I hope you have a TAM or even Falcon Complete 😞 Working with "standard" CS support was like pulling teeth
💯 1
n
Oh, I'm not looking forward to it.
Starting to notice a pattern with my crashes. Group Policy refreshed (domain and Citrix), then I start seeing picadd entries in the System log because the virtual channels are now closed, try and join a Webex meeting and the VDA blue screens within seconds.
Also, @jvitech, don't leave us hanging
Andrew will buy you a case of beer for the info
🤣 2
a
if it fixes our issue, then yes yes i will 🙂
j
lol sorry lads I didn't check Slack. My BSOD (related to tdica.sys pretty consistent) was on Windows server 2016, it started with 2203 CU1. I've been told this private fix was incorporated into CU2 but I didn't check. Private fix is ODU-164-CVHELP-19862. I hope it helps. If you need more info let me know! Beers will have to be shipped to Ireland hehe
n
Ah damn, that's the hotfix listed in the release notes for CU2, so not our issue then.
a
damn ya i was thinking that might be it. ahh well
j
Sorry lads! I run on XenServer 8.2 if that helps. Run Teams, WebEx, Zoom and various stuff. Hardware is Dell with AMD processor, no GPU.
n
We've seen this in all environments - Nutanix, ESX - running multiple configs - Webex, no Webex. I was able to get crash dumps with Driver Verifier enabled, so hopefully MS can help us pinpoint the issue.
j
On Windows Server? What version?
n
Nah, this is all on W10, both MCS persistent and PVS non-persistent.
j
Oh alright well good luck! I know the pain ...
a
thanks anwyays!
n
No beer for you...
😂 2
Has anyone ever actually had success using Driver Verifier for a crash dump? The 16GB dump I sent to MS was apparently useless because the dump wasn't triggered by verifier. Makes sense, given that with verifier enabled, it consumes 100% of the CPU the entire time it's enabled.
I've said this before, and I'll say it again: anyone who complains about Citrix support has never had to go through Microsoft support. I'm going on weeks now of trying to get them to analyze crash dumps, and their developers don't seem to understand that the Driver Verifier settings they want me to use are causing CPU usage to skyrocket and peg so much that the tool itself is causing BSODs, meaning I can't even generate a crash that I submitted the ticket for.
👍 1
b
@Nick Panaccio Complain to your TAM, maybe you're getting more help
n
That's my next step, as yelling at my monitor hasn't yielded favorable results thus far.
b
Sad but true haha
n
To top it off, I think I probably owe my monitor an apology...
Welp, we've officially decided to move forward with 1912 CU6 due to these issues, and how long it is taking to get anything from MS on the dumps. Still going to continue troubleshooting this on the side, but nobody was comfortable with moving forward with 2203 at this point.
a
that sucks! so did you do the back end upgrade to 2203 already though i think at least?
n
Nope, going to keep everything on 1912 for the on-prem stuff.
a
ahhh damn i'm on my own then 🙂 BUT on that same note i did get a note back from citrix w/ a private fix. i'm not sure if its the same one thats in cu2 and they ignored the fact that i'm on that version or if its different...or if its the actual crash that i opened the ticket for originally but gonna try it out
if you're curious i could send it to you
n
Was TDICA the component that MS kept saying was the issue?
MS wants me to disable our security products and do more verifier logging. I don't see any component being called out yet, but I guess they're focusing on those ones.
a
it was on the LAST crash that i had only w/ the verifier enabled. before enabler it called out other ones i think but i'd have to go back and look now
the one prior to the verifier being enabled i believe it was picadd.sys actually
this new private fix they sent has modified dates of sept and october.....i bet this is the same one thats in CU2 that we already have and they are ignoring those details
n
tdica.sys in CU2 is dated 12/5/22
a
what version is the file? this is what this one shows
n
7.33.2000.92
a
wth?? did they send me an old one? am i reading it wrong or is the one they sent me older
n
sure as hell sounds like it
a
jesus
n
Let me see if I have a CU1 device out there, to see if the file version is the same as that one
a
good call!
n
From CU1:
So I'm wondering if yours is the private fix, and they just updated the version # when it was included in CU2.
a
could be....this is getting weirder now...they just wrote back and said in my crash file it shows we are on version 7.24.6000.5. i just sent a screenshot saying nope thats definitely not the case we have 7.33.2000.92
n
lol
a
so i'm betting they didnt even send this to freaking engineering yet and all he did was look at that version and say ok fine i'll just send him this
ya so Citrix wont do anything else now unless i put the original private fix file in place and recreate everything all over again even though i showed them that the file i have in place is the newer version that deployed with cu2....AND they even confirmed yes, that file is the newer version of the private fix. is common sense just flat out ignored when they are hiring people now??
n
Yes, ask to have your case escalated, because that doesn't make sense.
a
i'm already on an escalation engineer and i actually asked this dude 3 times to escalate it . but gonna have to try again
and this is all stemming from the analysis from Microsoft lists the offending citrix file as an older version that is on the machine. i even did a scan for that file name and everyone that shows up is the cu2 version so no clue where that came from. Microsoft is ignoring me on it now too
n
If your have a TAM, get them involved and have them escalate it internally. That's what I'd do.
p
Hi, I just wanted to ask if you had any success with Citrix support? Are you still getting BSOD with 2203 CU2?
n
I can't speak to Andrew's case, but we received a private hotfix that did resolve our BSOD issues in 2203 CU2. It consisted of new PicaSvc2.exe and picadd.sys files, and should be included in CU3.
👍 1
p
a
interesting! different .sys file than what Microsoft called out for us but i'll be curious with cu3 what that brings. my only hope right now (other than trying a private fix someone shared with me that i'll be testing this week) since Citrix is being pretty much useless and told me i have to talk to microsoft again beecause they dont think it has anything to do with them.....AFTER i already did that and Microsoft sent detailed analysis why it is 🙂 that article comes back 404 for me btw.
r
Yea the link shows a 404 for me as well.
s
weird, that article worked for me about 3 hours ago too, now it doesn't
🥸 1
a
thanks for the heads up anyways though!
b
There you go
a
awesome thanks! there seems to be a couple of private fixes going around that sound very similar to our long open case. for whatever reason support doesnt seem to even consider that ours is related and instead is trying their best to push the blame on a different vendor when its pretty clear it open occurs w/ their update
b
guys do you have a casenumber? need to open a new case and would like to reference to an existing one.
a
here ya go: 81564637 cant say it will help much.....really just 5 months of them telling me to start over or talk to microsoft and me yelling that i did that already 🙂
b
sounds very encouraging 🙂
🐛 1
a
i dont want to jinx anything but so far CVADHELP-21885 hotfix for picadd.sys that another admin shared with me (Citrix would not) SEEMS to be helping in my testing. i'm a day and 1/2 in and no crashing yet.....it was a bit inconsistent but usually by now i would have gotten a couple so need more time. I did share this hotfix ID w/ my ticket months ago but was told this would not help our issue but since they are not being very helpful thought i'd give it a shot. i'll report back after a few days
n
CU3 is in pilot testing right now, so that's also right around the corner.
r
I wonder what ever happen to this?
a
@Nick Panaccio i think you mentioned that this private fix was going to be in cu3 right?
n
That's what i was told, and that's what the private fix file version says.
r
Cool
a
i could very well just be getting lucky w/ the fix so far and i'll be back to cursing tomorrow but so far so good 🙂 i'm bribing another engineer to test with me now
n
Yeah, I'd keep testing, and then repeat the testing with CU3 when it arrives.
Or if you want to test the early access release now: https://worldofeuc.slack.com/archives/CK6TCV3JN/p1682362686912029
You could compare the files in that installer.
r
Yea @Andrew Gresbach get on that for us. Report back by noon ha
a
aye aye capt'! 🙂
🤣 1
i'll see what i can do
pretty sure you can have a new VDA hotfix version than is your controllers right?
alright i beat the noon deadline 🙂 so the PicaSvc2.exe looks to be the same file version comparing the private fix to cu3 test build but the picadd.sys shows newer in the test build
n
Nice, this gives me hope that it'll work in our environment.
And to answer your other question, yes, VDA can be newer (support is a different question)
a
maybe some other tweaks in the sys file since the private fix to bump up the build # but promising. man i hope this fixes our crash issue too so i can be done w/ it
just an update i'm about 6 ish days in w/out any issue (about 3 for my other tester). trying to setup a call w/ the tech on my ticket to get some clarification why they told me this private fix definitely wasn't related to our issue. maybe i'm just on a lucky stretch but looking like a promising fix so far. if this holds for another week or so i may try the cu3 early build
n
Or maybe it is related to your issue because whoever looked at your case before didn't actually look at your case.
a
That's me theory as well 😂 I have a call tomorrow with the tech so I'll see what they say this time
n
"What does Microsoft say about this?"
a
$10 says they say that again tomorrow
so the tech on this case now actually seems pretty good and said he totally agree's citrix HAS to be the root cause here. he found it interesting that the private fix seems to be helping so he's going to try and push engineering more on it and get back to me in a couple of days
n
i knew it
😂 3
a
one last follow up on this, we've had the cu3 agent deployed now for over a month and zero issues that i've heard. we still have cu1 for our DDCs and planning on that upgrade soon
👍 1
n
Same, been running CU3 on my own VDA since release and not a single issue. Currently deploying it in our lower environment.
👍 1
8 Views
Open in Slack
PreviousNext
Powered by