HELLO ALL. If you're one of the "lucky" organizati...
# adobe
d
HELLO ALL. If you're one of the "lucky" organizations having a problem with ACF 2021 Update 12 where all MSSQL queries fail, PLEASE reach out to @Mark Takata (Adobe) here ASAP. Symptom is that queries say they timed out, but that happens immediately (not after a timeout), and logs show a low level java error. Adobe is trying really hard to nail this down, it's a VERY high priority for them internally. The problem is that only a few customers are seeing it, and the engineers don't have a reproducible case they can analyze. I had a long remote session with them, where we generated a bunch of extra logs, which I then sent to them, but it's not enough. They need more cases to examine, but so far the only person I know of besides us is @Sid Wing. On that remote call Adobe told me there were others, at least four in total. More folks with this problem getting in touch with Adobe will really help get it solved. If you're seeing these symptoms, Adobe really needs you to connect with them. Please stand up and have your aberration counted!
😮 2
s
Yessir - and it seems to be "hit or miss" as to when the problem will occur. Example - 3 identical servers (OS/CF/Codebase) We rebooted them last Thursday night (for maintenance). WHen they came back online - 2 of the 3 servers were having "Update 12-itis" - the third was fine. Mind you - these are PRODUCTION machines. Luckily they are load-balanced so the ones that had U12-itis were not visible to the "real world". I have found that uninstalling and then reinstalling U12 temporarily "fixes" the problem. But it doesn't STAY fixed.
d
We've been over this, but just for clarity, reverting the MSSQL package to the prior version, 05 I think, then restarting CF, definitely doesn't help? It fixes the problem for us, so far. No idea what other effects that has, or if there are other issues beyond queries we haven't hit yet.
s
The last time we did the SQL swap - it did not fix it - but I will get another chance to "try it" on Thursday night as we have Windows Updates to apply to those servers - so we will need to reboot them
OK - Tested a couple more servers with the SQL downgrade - that seems to have worked on them.
d
So you have some where the downgrade works, and at least one where it definitively doesn't, when tested with care? One of mine that previously would only work with MSSQL downgraded to U5 works now at U11, the current version. I've restarted cf several times, and it's stay working, can't break it. Super weird.
✅ 1
My other non-production 2021 server I rolled the whole thing back to U2, which I think is the original install, and it's working fine of course. I'm leaving it there until Adobe comes up with a real fix, then I'll try whatever that is.
✅ 1
s
We can't roll back to U11 in Prod (due to security gurus saying we MUST be at U12) - but I AM going to try the SQL downgrade on the remaining 5 prod server during our maintenance window tonight
d
We still have one production server on u11, because I can't guarantee that U12 won't break SQL, or that if it does, the MSSQL package downgrade will fix it. It's a 100% mission critical server that I just can't risk taking down. My security people are chomping at the bit somewhat, but they don't want to take that server down either, so. This is in Adobe's court, and I actually know they're working on it, hard. I can't say more about that.
✅ 1
e
Is your server NIC a boardcom, I ask for a friend...
s
Not sure what that has to do with anything
d
Agree, what's that about @Evil Ware?
e
I ask as I have a massive amount of servers, all with various NICS. The Broadcom NICS have in the past had weird software driver issues that affect everything they are attached to, including containers and virtual machines running CF. Not saying it is this, but it has been an issue with some code we have running in the past.
s
Yeah - this is not a networking issue - it is most definitely a software (Update 12) issue.
âž• 1