TIA for patience, I hope I explain this right. Our production servers are behind a load balancer, and our staging server is behind the same load balancer. The load balancer is "cloaking" a 500 response from the server by showing a 404. I am unable to find the location of the 500 error being logged in either Fusion Reactor or the lucee logs for application, exception and out. The load balancer is unable to show me what it received in the 500 and will only show the 404 of the cloak. Where would you look to get the actual response from the server? Please, if you feel you need to speak very simply, I understand! This feels very basic and I'm surprised I can't find the 500 anywhere!

maybe an error in the webserver (IIS,apache,etc)? what OS and webserver are you using?

The 404 is presented to the browser by the load balancer. In the load balancer interface, it shows as 'cloaking' and that the status given to it to was 500. But the content is not available. This is on a windows server. I've checked IIS logs for the 500 as well. The really odd thing about this is that calling the page with one client id returns immediate results, but calling the page with another specific client id gives the 500. Fusion Reactor shows the request as completed with a 200. This may be because of Lucee being told to do that. In fact, I'm going to go check that.

can you just load the page with the 500 clientID directly from the server bypassing the load balancer?

@David Johnson I have a similar environment to you. Windows shop, using IIS to front Lucee on a particular node, the nodes behind a loadbalancer. So historically how I approach these types of issues is similar to what @Erin Brewer stated. I will hit the pool/virtual ip through the loadbalancer. Then I will hit the individual servers behind the loadbalancer. Here I may hit IIS behind on the node behind the loadbalancer as well as the raw Lucee server which may run on a different port. What I'm trying to figure out is any obvious points this may be failing. My calls to the exposed endpoint on the loadbalancer are typically HTTPS on port 443, but my calls to IIS on the backend server are HTTP on port 80, and my calls to the Lucee server directly are HTTP on port 8888. I know this is a more generic response but it may help guide your analysis of the situation.

Which load balancer? Yes, it kind of matters. How are you "load balancing" your requests? Are you Using Java sessions? Are these just lucee boxes, or is it apache or nginx and lucee?

Barracuda Load Balancer, Windows IIS, BonCode, ApacheTomcat, Lucee

Have you managed to run the query directly on the server, serving the app? If you follow @Peter Amiri’s directions above - you can hit your application through IIS and Lucee directly, too. Giving you (and everyone else, here trying to help you) further information. (unless I have misread....) You also don't mention if the error is your prod or staging instance. or both? It is curious that you mention that the Load Balancer sees a 500 - despite being sent a 200. But it certainly isn't strange either that you might produce a status code of 200, despite an error occurring. Eg. Taffy/ColdBox/FW1/Mach-ii etc... all allow you to return any status you like, should you choose to. (I don't personally agree with it - but that's another story) We have code that will return a 200 - but the message is appropriate to the error instead of "OK".

With so many moving parts, I am hung up on one issue in the middle of all the issues. I am using Boncode, IIS and Tomcat. Some requests stall during server connection. They will sit for exactly one minute and then receive the response from IIS. While troubleshooting the issue, there was concern over the jquery load function adding a ./ in from of an unpathed load ( clients/index.cfm instead of /clients/index.cfm) but no amount of changes has stopped the stall. Having searched here for the word boncode, and because we are unfortunately married to IIS, I'm wondering what I can troubleshoot now that I have boncode logging to see why the stalled connect is happening. The stalled connect shows Timing of Queued - 1.68 ms Stalled - 1 minute Request Sent - 21 ms Waiting for server response - 314 ms Content Download - 2 ms It's that stalled for 1 minute that I can't figure out! Yes, there were reports of 500 server errors showing as 404s on the load balancer, but the 500 errors have been fixed and the only thing left is this Stalled thing. I'm suspecting something between IIS and Boncode, and have logs to share if necessary. What do you think I should do?

@David Johnson You still haven't told us if it is Prod or Staging ONLY - or BOTH? Further, if it is staging only - is there anything different between in your code? What differences are in you load balancer config - when it comes to dealing with Staging versus Prod? Do you get the same error in a local dev environment? Can you cause an exception hitting the same URL for the same client - in your DEV machine? Have you reviewed the queries and the data? Does the problematic client have an empty field that the other clients do not have? Missing data in tables used in a join? Also you need to give us some more information about the basics... What version of Lucee? What version of Tomcat? What version of Java? (Is Staging and Prod the same?) Perhaps there is a known interaction between version xx of tomcat and java YYY - that someone could comment on straight away... Perhaps something to do with IPv6 and IPv4? - If you're not using IPv6 - then turning v6 off - stops issues with stalling on the first request. (Not the stalling/slowness is necessarily the reason that you have an issue) Though it might be - if the "stalling" request is the only one that is causing your exception... Or do you in fact have two separate issues that are completely unrelated? (and if so - let's just concentrate on one at a tme) But all of this is just guessing.... until you can give us more information.

I'll try to do better, Gavin. Thanks for the kind words. I'll take it from here.

If he is using Windows, turning off IPv6 is not recommended. Instead, in tomcat server.xml, look for your connection setting, its either localhost or ::1, if its localhost, change it to ::1 and restart lucee/tomcat.

Yesterday morning, because we use the JSessionID and not Client Cookies, we disabled client cookies on all our production servers. I was able to show before the update that deleting those two cookies allowed the page to load. Don't ask me why, and maybe a greater mind can offer understanding, but NOT sending CFID/CFToken when not being used allowed for the request to process properly.