Hi, we are encountering an issue with Lucee to Luc...
# box-products
d
Hi, we are encountering an issue with Lucee to Lucee SOAP webservice requests when destination server is running commandbox 5.5.1, for services which worked fine under pre 5.5 I did originally post about this under #lucee, as I thought it may be JDK version related, however under further testing, this issue is still encountered when using jdk-11.0.14.1+1 with commandbox 5.5.1 Original post here: https://cfml.slack.com/archives/C06TA0A9W/p1651769879023609 Updated details as follows: ---- We are using Lucee 5.3.7.47 We have 2 CFML applications running on separate instances. One of those instances makes a SOAP call to a CFC on the other server:
Copy code
local.wsObject = createobject("webservice", "https://[host]/[cfcname].cfc?wsdl");
local.response = local.wsObject.[methodname]([parameters]);
This has worked without issue across multiple Lucee versions and Java versions, for several years. We are just in the process of testing the latest commandbox 5.5.1 release - current environment is running on CommandBox 5.4.2 - and we are testing on commandbox 5.5.1 When testing on commandbox 5.5.1, all our calls to this web service fail on line 2 in the above sample code We see a GET request logged on our web server to https://[host]/[cfcname].cfc?wsdl We do not see any subsequent POST requests (which would normally be the case. Within the exception log we see the error “java.net.ConnectException: Connection timed out (Connection timed out)”
Copy code
"FaultCode": {
         "Prefix": "",
         "LocalPart": "Server.userException",
         "NamespaceURI": "<http://schemas.xmlsoap.org/soap/envelope/>"
      }
With the 1st line in the stack trace being the 2nd line in the previous code sample where we are attempting to call a method on the object. If we switch back to CommandBox 5.4.2, then the request completes successfully. I did compare the WSDL for the CFC being referenced from instances running both versions. The only difference (other than hostname between 2 instances) was that for the working (CommandBox 5.4.2) instance, the self referential namespace references were pointing to https://[host]/[cfcname].cfc and for the non working (commandbox 5.5.1) version, it is pointing to http://[host]/[cfcname].cfc i.e. switched from https to http. It should be noted that SSL is implemented on the level 7 load balancer (AWS Application Load Balancer) with traffic between load balancer and nginx proxy, and between nginx proxy and Lucee server (WildFly / Undertow / Commandbox) being via http. Our nginx proxy is configured to redirect any incoming http:// requests to the same URL via https:// I have review load balancer logs, and determined that no incoming requests to [host]/[cfcname].cfc have been made via http - all are via https No idea if the above is the cause, or if there is some other issue… SOAP web service requests complete successfully from a commandbox 5.5.1 instance to a CommandBox 5.4.2 instance, but do not complete successfully if the destination instance is running commandbox 5.5.1 We have subsequent to initial tests, deployed a commandbox 5.5.1 instance with jdk-11.0.14.1+1 instead od the default jdk-11.0.15+10 JVM - as we had initially thought that his issue could be due to java version rather than Commandbox - but we experience the same issue, which suggests that this is a commandbox rather than a java issue,
Hi @bdw429s Are you able to give any feedback regarding the above described issue, with respect to potential cause / workarounds… Any idea if this is an issue with commandbox 5.4.2 - or could there be an issue with our configuration which causes this behaviour to occur?
Let me know if there is anything unlcear in the above description, or any further information needed for identifying / duplicating the issue…
b
@danlance Sorry, I saw "Lucee" and "SOAP" in the first sentence and stopped reading, lol. I'm unclear if you're testing with the same version of Lucee in both cases
Note, if you don't have a specific CF Engine set, CommandBox 5.4.2 will give you Lucee 5.3.8 by default and CommandBox 5.5.0 will give you Lucee 5.3.9 by default
There's a heck of a lot of details above. Do you have something like a very simple repro case. Like run
start
in this test project and make this Postman call?
d
Long and short… 2 working applications using Lucee 5.3.7.47 and base docker image ortussolutions/commandbox:3.4.4 (CB 5.4.2) together with jdk-11.0.14.1+1 were updated with only the following changes: • base image changed to ortussolutions/commandbox:3.5.0 (cb 5.5.1) • using default java included in this image (jdk-11.0.15+10) CFC SOAP web service calls which were working from App 1 to App 2, then failed with the error
java.net.ConnectException: Connection timed out (Connection timed out)
Rolling back App2 to to previous build, (so new image on App1 but previously working on App2) resolved the issue Manually overriding the java version used on App2 to use previous java version (jdk-11.0.14.1+1) but new version of CB ( 5.5.1) did not resolve the issue. Therefore - that would suggest that • The issue is caused by CB 5.5.1 and not by jdk-11.0.15+10 • The issue is on the destination processing of incoming SOAP requests, not with how the outgoing request is made ----- As I was attempting to drill down deeper into the issue, I identified a difference in the WSDL response: • for the working (CommandBox 5.4.2) instance, the self referential namespace references were pointing to https://[host]/[cfcname].cfc • for the non working (commandbox 5.5.1) version, it is pointing to http://[host]/[cfcname].cfc i.e. https had been changed to http As I say, no idea if that is the cause of the issue - but was the only notable difference I could see with respect to the WSDL
I have not gone away and reproduced a MVP test case
I also included a load of additional information with respect to the configuration of the hosting environment SSL configuration / proxying etc, in case that was relevant… as the requests coming in to Wildfly are not themselves https, as SSL is terminated at the load balancer, and not at app server / nginx
b
Yeah, no clue to be honest, I avoid SOAP when I can because it's terrible 🙂
d
yup
b
I would would recommend divide and conquer-- try to reduce the number of moving parts until you can get the smallest possible repro case.
d
legacy app never updated the SOAP part… needs to be done
b
CommandBox 5.5 does have a newer Undertow version, but really the SOAP stuff is all done inside Lucee using the Axis jars
So unless there's some completely unrelated issue that just happens to be getting in the way, I'm not sure what it could be
Are there any errors in the logs of the server receiving the request
A connection refused seems to indicate a general networking issue that didn't even reach the target server
And I really have no clue how the WSDL is generated-- that's all inside Lucee
d
yeah - lucee version should be the same in both - dockerfile has following in both:
ENV BOX_SERVER_APP_CFENGINE lucee@5.3.7+47
The destination server responded with a 200 response to the WSDL request… triggered by the line:
Copy code
local.wsObject = createobject("webservice", "https://[host]/[cfcname].cfc?wsdl");
Within the source server, the error was logged:
java.net.ConnectException: Connection timed out (Connection timed out)
with detail:
Copy code
"FaultCode": {
         "Prefix": "",
         "LocalPart": "Server.userException",
         "NamespaceURI": "<http://schemas.xmlsoap.org/soap/envelope/>"
      }
against the following line of code:
local.response = local.wsObject.[methodname]([parameters]);
For this request: • Nothing was logged on the destination server (Lucee logs)Nothing was logged on the destination server (Nginx logs)Nothing was logger on the destination Load Balancer (AWS Load Blancer Logs) i.e. this request was not made to the destination server… This led me to the conclusion: • Cause of failure most likely due to an inconsistency in the WSDL returned With respect to the network configuration: • Incoming requests to the LB on port 443 are directed to port 80 on the nginx instance, which are forward proxied to underlying Lucee instance • Incoming requests to the LB on port 80 are directed to a different port on the nginx instance, which responds with a 301 redirecting to the same domain, path and query string In the case that the SOAP request associated to the 2nd line of code came in via http rather than https: • request would be logged on LB • request would be logged on Nginx • Request would not be logged on Lucee So… with that… and with confirmation from the logs when hitting the old instance (CB 5.4.2)… and with no other changes made to the environment settings other than switching to a new docker image with the changes to the CB base image (using default java version)… it does not seem feasible that this is an environment caused network issue
But as you say… many moving parts
b
Here's a thought-- the newest CommandBox doesn't use the forwarded for IP HTTP header any longer by default for security reasons
You are safe to enable it however if you trust the proxy in front of your server
It's possible the same handler that looks at the proxy IP also looks at http headers related to whether the request was HTTPS or not
based on your setup, it sounds like you can safely re-enable the proxy IP support
I have a feeling this is a matter of an HTTP request being sent to a server that is simply rejecting it because it only accepts HTTPS
d
…but if that is coming to the same endpoint, it will hit the LB and be logged irrespective of whether it comes in via http or https
b
No clue, man
I can't really help you debug your networking stack 🙂
d
I mean… I think the evidence above suggests that it’s not the networking stack…
b
I have a feeling you're not seeing all your logs or not logging all requests
Just try the setting
if what you said was true (that the request didn't even hit the LB), then it also couldn't be an issue with CommandBox
So I mean, something has to be wrong here, lol
Those proxy headers and the HTTP/HTTP difference in your wsdl seems like a solid lead
When I said I can't debug your networking stack, I didn't necessarily mean the issue was IN the networking stack, I just means I can't explain why things seem to be the way you are seeing them.
I can't link to headers, but scroll down to the
X-Forwarded-For support disabled by default
section in the release notes https://www.ortussolutions.com/blog/commandbox-551-released
d
lol yeah - deffo - something is wrong - as I say different WSDL is the only clue - but happy to try changing proxy setting - it seems like what lucee is seeing for the initial wsdl request is different than previously, so as that has changed… then worth checking… Can this setting be changed within the server.json?
b
Read the blog post above ☝️ It answers that question
Also, I have confirmed that SOAP works fine on the latest CommandBox version
Copy code
component {

    remote function hello() {
        return "hello world";
    }

}
and I can call it like so
Copy code
local.wsObject = createobject("webservice", "<http://127.0.0.1:18219/myService.cfc?wsdl>");
dump( local.wsObject.hello() )
So while SOAP is involved in your problem domain, I don't think it's the root of any issue
d
Yeah - not tried dumping the returned object… as I say, the initial createobject call does not error and does hit the WSDL endpoint It’s when a function is called on the object which then does not result in a subsequent request being made
b
So you can enable it in your
server.json
with
Copy code
server set web.useProxyForwardedIP=true
or just set the following env var on your container
Copy code
box_server_web_useProxyForwardedIP=true
d
so within my server.json:
Copy code
{
  "web": {
    "useProxyForwardedIP": true
  }
}
Would that also work?
b
It's not an also, it's the same thing
The
server set
command simply modifies your
server.json
for you
d
👍
b
I can confirm the same Undertow handler that deals with forwarded IP also deals with the
X-Forwarded-Proto
header as well
So it makes perfect sense that having that setting off can also affect whether the server "thinks" it's HTTPS or not
Fun fact, my Github picture shows up as a contributor to that Undertow source file 🙂
d
that does make a lot of sense
updating / building / deploying fingers crossed
b
I really liked having that stuff on by default, but a security researcher in the community pointed out how it can be manipulated by a malicious proxy if your CommandBox server is just sitting out on the internet so it's not really a "secure default"
If you just blindly trust the proxy headers coming from upstream, a hacker could basically tell you they are coming from
127.0.0.1
and bypass all your IP based security!
So I had to make it opt-in to still be "secure by default"
d
Yeah - that makes sense… In our case we are double proxied before this… (AWS LB always generates, which then passed to Nginx, which is configured to forward to lucee)
b
Yep, it's totally safe in that scenario. There's just no way CommandBox itself can know unless you tell it.
d
I would so love with this app to redo the architecture… and replace the SOAP call with a simple REST workflow… and I can probably get it in under the guise of another update which has already been approved… just not in the current deployment I wouldn’t be surprised if there was no one else using CB and SOAP for real now… if ever!
@bdw429s 🎉 🥳 That sorted it 😄
b
Sweet-- I'm guessing Lucee/SOAP didn't know it was HTTPS and so the WSDL had HTTP and that was getting rejected somewhere in the stack (and not logged for some reason)
d
Yup - no idea why there was no evidednce of a subsequent http request… it’s like there’s some code within the web servicey bit of Lucee whic goes: If the service endpoints in the WSDL don’t match the protocol of the request… then sit and spin…
destination was doing the best with the information it had available
but to get a timeout from the source, rather than an error… not sure
We have so much logging in place at multiple levels… that I just don’t see how any http or https request made to the correct endpoint (and the endpoint in the WSDL was correct other than the protocol) could take place without being logged… But it’s always possible I cocked up somewhere…
(it has happened before!!! believe it or not!!!)
Thanks again @bdw429s for your assistance and advice… your knowledge and intuition there was absolutely invaluable
👍 1