This message was deleted.
# microsoft-fslogixs
Slackbot
10/23/2019, 6:06 PMThis message was deleted.
l
Leee Jeffries
10/23/2019, 6:23 PMHey Tim, can you get stats from the different layers? Disk queues, reads and write per second, read and write latency?
t
Tim Riegler
10/23/2019, 6:24 PMYeah, one moment
l
Leee Jeffries
10/23/2019, 6:24 PMIf you could also list out what disks you are using in which configuration and the SAN model that would be grand.
Leee Jeffries
10/23/2019, 6:24 PMAlso , are dedupe and compression enabled? How do you know it's the storage?
t
Tim Riegler
10/23/2019, 6:32 PMEvent logs are definitely indicating storage. Event ID 129, pvSCSI - Reset to device, \Device\RaidPort2, was issued. <- that's on the ODFC box. Same thing, except for LSI SAS on the Profile server. Different timings, but both occur on the same day.
Tim Riegler
10/23/2019, 6:33 PMTegile T4700 AFA. Yes, dedup and compression are enabled on the array side. Here's the fun part - the ODFC has been implemented in our environment since FSLogix released it, with 0 problems, on the same underlying hardware (hypervisor and SAN, with dedup and compression enabled). We moved from Windows Server 2012 R2 to 2016 Core for this build, and now are having problems. I'm about ready to move back to 2012 R2.
Tim Riegler
10/23/2019, 6:34 PMI gotta run to a meeting...I'll throw up more info when I'm out. Thanks Leee.
b
Brandon Mitchell
10/23/2019, 6:43 PMI had an advisory case open with Microsoft asking about official support for compression and de-dupe, they told me, neither were supported, I asked if they could provide official documentation that I could give my storage team... Their response was to test, my mileage may vary, and they wouldn't provide any official documentation that compression or de-dupe wasn't supported. They said odd issues are usually related to compression and de-dupe.
l
Leee Jeffries
10/23/2019, 6:51 PMCompression can cause issues indeed. Much slower but it's all about looking at the metrics. I'd want disk stats at all levels to see if there really is an issue first.
👍 1
t
Tim Riegler
10/23/2019, 8:12 PMSAN Stats. Occurred at 11:01 am EDT on the Profile server, and 12:37pm EDT on the FSLogix server. This covers last 24 hours, and includes overall array stats as well.
Tim Riegler
10/23/2019, 8:13 PMWe're investing the compression aspect - although as I said, that was enabled previously when this worked flawlessly.
l
Leee Jeffries
10/23/2019, 8:19 PMYeah, it is a bit strange. You haven't got anything different A/V wise on the file servers?
t
Tim Riegler
10/23/2019, 9:18 PMNo A/V at all. In fact, completely disabled Microsoft Defender, just to make sure!
l
Leee Jeffries
10/24/2019, 11:50 AMTim, your Read_Latency values are quite high during the periods if the sheet is showing ms
Leee Jeffries
10/24/2019, 11:50 AMColumn AL between 11:00am and 13:30pm on 22/10/19
Leee Jeffries
10/24/2019, 11:54 AMYour write latency is good but read is high. I would more thank likely clarify this at a VMWare level and then check disk queue on the VM to see if the issue is all the way up the stack.
Leee Jeffries
10/24/2019, 11:55 AMThe man we need though as my experience in troubleshooting something like this is limited - Jim Moyle. I'm not sure he's on slack. You mind If I send this to him?
t
Tim Riegler
10/24/2019, 12:42 PMThe read latencies were for a period that we were not experiencing the disk connects. All disconnects were at the 11:01 and 12:37 on 23/10/19. You may absolutely send to Jim, Thank you 😃
Tim Riegler
10/24/2019, 12:42 PMI'll find out our VMware info today, and post it.
Tim Riegler
10/24/2019, 2:59 PMDoing some additional digging. We're seeing these disconnects across multiple VMs, VM hosts, and Arrays. Commonality is VMware version. Seeing it on LSI SAS and pvSCSI drives. Super odd. Escalating with support, and I'll post back what we find. In the meantime, we're investigating Cloud Cache to soften the disruption. Should be interesting 😃
Tim Riegler
10/24/2019, 8:07 PMOk, did some more digging, checked our assumptions, and found that we were making some major mistakes in our thinking. This is larger than just the profile / ODFC servers, and is affecting the WHOLE environment, including XenServer hosts. Looking at the moment like a storage array problem, and we are planning an upgrade tonight. I'll let you all know how that goes!
l
Leee Jeffries
10/24/2019, 8:09 PMPlease do feedback Tim. If the latency is before the the issue that's more worrying ;-)
t
Tim Riegler
10/24/2019, 9:11 PMIt's not a latency problem from what we can tell - looks like the underlying storage is acutally disconnecting. 🤯
Tim Riegler
10/25/2019, 3:22 PMSo far Array upgrade has resolved the disk disconnects. HOWEVER, it's still a little too early for me to call it fixed.
l
Leee Jeffries
10/25/2019, 7:27 PMNice one Tim. Happy days, it's nice when that happens.