We're seeing a weird issue where occassionally we'...
# cfml-general
d
We're seeing a weird issue where occassionally we're seeing requests come in that have been re-encoded by something outside of our control. For example, the application will generate a link like: /some/endpoint_is_here?a=1&b=2 And what the server receives is: /some/endpoint%5is%5here?a=1%26b=2 I've verified the issue isn't with our application stack and we only see if from two of our clients locations (but mostly from one location). This doesn't happen frequently, like I would say like 0.000001% of all requests (maybe even lower). We see it a few times a week out of millions of requests. What I suspect is happening is that either some software running in the client or site-wide (like a web firewall) is running that is altering the request and for some reason believes the URL needs to be re-encoded (or it's just a bug in the software). I know for a fact that the company we see the issue with most has some kind of outgoing web firewall/proxy inspection tool that they use. Has anyone else seen this behavior? Anyone know what might be causing this? Perhaps it's a bug with CloudFlare (which are production environment uses), but since I've only ever seen the issue from two client locations, I suspect it's more likely related to something in their environments. I've inspected all the HTTP headers of the requests and there's nothing there that help pinpoint to a root cause. I was hoping that if there was a some kind of web firewall re-writing the request that there might be some kind of signature that indicate what it was, but there's nothing there that seems to indicate anything unusual. I'm trying to figure out the best way to resolve the issue.
Oh, and when it's a POST request it tends to just drop the parameters completely.
m
What does the client browser string say in the web access log?
c
email clients are also great at messing up links if they could come from there
d
@chris_hopkins The links aren't coming from emails @mjhagen My example of what the server receives is what the logs show.
m
so, no data about request size, date, time, ip, etc? I asked because that might give you a clue of where those requests come from. What webserver are you using?
d
@mjhagen of course we have some logging on time stamps, etc, but that information has not proved useful. From the best of what I can tell, the users are clicking on the link and then somewhere between when they click on the link and the HTTP request gets into our infrastructure, the link gets re-encoded. I was hoping there would be some kind of HTTP header signature that helped us identify what might be altering the HTTP request, but that does not appear to be the case.
j
I am seeing the same issue and have not found any pattern to the requests. Sorry I don't have and ideas what is causing it.
s
Is the POST being handled by any JS? If so, maybe something has changed with that.
If your sure it's happening in the infrastructure look for WAF updates, rewrite rules, LB updates etc ... I would setup a simple test case outside of the infrastructure just to be sure that is where the cause is. It is not unheard of to force encoding on a server.
d
@JohnT Are you using Cloudflare? I really don't think it is the problem, but it could be.
j
No Cloudflare.
d
@spills It's happening outside of our environment (unless it's a CloudFlare issue, but I don't believe it is since we only see the issue with two specific clients). I really think it's either a WAF our clients are using to inspect all outbund HTTP traffic or perhaps the security software they're running on the clients. What I suspect is happening is that it's detecting something in the string that thinks it needs to encode the URL, so it's applying a second level of encoding to the URL. This doesn't break the filename, but it does break the query string.
👍 1
@JohnT Thanks. That at least helps to strengthen my opinion that it's not CF specific.