https://worldofeuc.com logo
Join Slack
Powered by
This message was deleted.
# microsoft-fslogix
s
This message was deleted.
t
We also have servers that stop responding / accepting connections, and when they do that, all users that log off can't log on to another server, their profile is still in use (SMB connection) from the original server.
l
Hey Tim, bad times. P1 level right? What did you upgrade?
t
I'm thinking our next step is to disable CC and see if this works any better without it. Any other thoughts for troubleshooting / next steps?
Yup yup, as of right now =(. It seems to be affecting more users as we go. We upgraded FSL last Thursday.
l
You are getting PVS session reboots right? Do you monitor disk space on thos boxes?
t
Yes, through ControlUp. Everything looks good. WAY more than enough space.
On both C: and on the write-cache drive (D:)
l
Okay, what about your file servers? Do you performance metrics through controlup for these?
t
They look great. Here's a snapshot from just now.
l
Erm.... a disk queue of 19.2 is definitely not great
t
Disk queue does occasionally increase on Profile2
haha. True that. It does fluctuate quite a bit. Let me investigate that...although that is the secondary storage for the Cloud Cache.
l
That should spike but come straight back down. Anything above 2 for a sustained period of time will impact storage operations.
My advice. Roll back to previous vdisk version, keep one or two servers in play and attempt to reproduce the issue.
Create a persistent disk for those machines and redirect all the logs to that disk. It'll really help gather information.
Where is your cloud cache , cache stored on the XenApp session server?
t
We have logs redirected to persistent disk, so can do that. Yes, the cache is stored in c:\programdata (the default, I believe)
l
Are you using ram caching with overflow to disk?
t
Yes
l
Personally I'd get that cloud cache put on the persistent disk side. That should help the performance of your session servers. I'm not a huge fan of cloud cache as I've not found it to be hugely reliable, that and you cannot really control how much data a user writes when gonna persistent machine. Only by the VHD size anyway.
My gut would say its cloud cache related. But... rollback. Reproduce, information gather and analyse the data. Happy to help Tim.
t
Yup. That's where I'm at. Thanks Leee, I appreciate it!
Looking through CC logs, and found out of order flushing. Interesting.
l
No problems at all. I'm in the UK so theres a bit of a time difference. Off the top of my head I'd check the fsl logs first
You're there already.
t
yeah, now the question is - is this FSL, or network?
l
Do some iperf tests between your file server and XenApp hosts, check your pvs session server retires. Should give you a direction to networks or not.
t
Keep proxy on C: and move cache to write cache drive, correct? We may have to try that first tonight. Rolling back image would be...problematic.
PVS retries under 10 for all VMs. Will check iperf next...
l
Exactly that what you said about the cc cache.
Why is rolling back problematic? Change control or there were other updates too?
t
One other update - Office was updated to latest C2R version. We could do it, for sure, would just have to communicate well.
l
Sure. If it's this bad though.... I hear you though. Downside of centralised management ;-)
t
Welp, Disk Queue on 2 is back down. And stable.
The underlying Flash is not seeing any performance problems. Ugghhh
d
I think it’s related to CC. Put the cache on the persistent drive. Do the FSL policies apply to your master VDA? If so, do only apply them to your workers! Check if the registry settings are identical to your GPO. Check this out: https://www.sul.de/2019/04/05/fslogix-in-a-citrix-provisioning-environment/
t
Hi Dennis! The FSL policies do not apply to the master image, only the worker VDAs. We are double checking the registry settings now! Thank you!
d
l
Hey Tim, why is the disk queue high on the second node if the storage is having no problems??? I'm not saying start there but... on a log off the differencing disk will merge to both locations. If that merge takes a while on the second locations the logoff will stick.
Agree with Dennis though, as I said. Stick the cache on the non-persistent drive. Do you have a way of generating load in your environment?
t
We looked through the logs. Found some interesting things, not sure what to do with it, though. For instance: • Local cache was found to be invalid. • [WARN: 00000000] Out of order flushing, potential data loss. Flushing change with sequence
Leee: I'm thinking it has to do with the ESX host. The VM is having problems, and the storage is fine. <sidebar> this is something I've long felt has been a problem, and haven't been able to get anywhere yet.
l
Okay, check any storage policies that apply to that host. Run iometer on both nodes and compare your output. Should give you something to go on. Watch control up go nuts when you do it.
t
Ok. We'll put the cache to the drive. We do have LoginVSI, although I don't think any of the test accounts are configured for FSL.
l
Depends how serious the situation is but I'd still be looking at rolling back and then simulating the load on a couple of standby boxes.
t
One other thing we're thinking - we're using DFS-N for the share names for the CC SMB locations. Wondering if changing that to direct server names would be beneficial. It's easy enough to change via GPO that I'm not really worried about it
It's pretty serious, for sure. Other thing we're seeing is at least 1 box a day straight up rebooting under load.
We have a new server which just arrived, I'll green light the install and config so we can do some load testing.
l
Let me know if you need anything with the lvsi setup etc... if your ovs box is flat out rebooting you'll want to redirect crash dumps to the persistent disk so you can analyse the reason. This is the assumption I have made when using FSLogix on PVS. the write cache can fill up and the box can die. There are perf counters for the ram cache too. Check those out.
t
Thanks Leee - we are keeping any eye on PVS RAM cache. I just ran the CU script, and am seeing ~68% utilization, with 0 overflow to write cache disk.
l
That's good then. Looks like you have all the right tools in play. Let us know how you get on Tim.
t
We'll change the cache location tonight during our maintenance window. Change from DFS-N to direct server names for CC config.
l
If using DFS-N just make sure it all tallies up in AD and users are always directed to the right location. Your change will take that complexity out though.
t
Well, some good news! Think we found the "smoking gun." It appears that another GPO was linked to our Production OU that had non-CC FSLogix setup (i.e. vhdlocations was populated). As those are mutually exclusive, and populate different areas (so don't overwrite), that was MUCH BAD. We removed the offending settings, and everything appears to be back to normal today!
l
Perfecto! Well done Tim.
t
Was my Citrix engineer who found it. After we'd been banging our heads against this for a while. All seems stable, thankfully. I SUPER appreciate everyone's help!
l
How are the disk queues? Lower? Seems like you had fslogix fighting cloud cache. pretty cool, draw a picture for that one.
t
Funny (not really) thing is this is the SECOND TIME this has happened to us. Happened when we initially looked at CC and started testing it. Thought we could have both configured, and it would "just work." Turns out, that's not true!
Disk queue is still consistently higher than I would like (many peaks, not continuously). I'll investigate that. The primary has much lower disk queue. Overall, though, MUCH lower disk queues on both, today.
l
Thats good, spikes aren't really an issue, that should be fine.
r
Damn man
d
Glad to hear this Tim, so enjoy the weekend now 😁
t
Thank you!
2 Views
Open in Slack
PreviousNext
Powered by