So - we applied Update 20 to our CF2021 environment - and of course (because legacy code) - we have an issue with the "remote method parameters" change. SO... Per the instructions - we tried to implement the "-Dcoldfusion.runtime.remotemethod.matchArguments" JVM flag and set it to false. THAT apparently breaks the isapiredirect connection with IIS. WHen I look at the isapiredirect logs - it says it can't connect to the backend and that Tomcat is likely down. So - my question is - is there a known issue with that JVM flag?

Sid, I haven't heard of this before. @priyank_adobe have you seen or heard of this issue?

and is the flag spelled right (typos are the bane of every devs existence)

FWIW, we've done update 20 with that flag in dev for our 2 big apps, cf2021 + IIS, and haven't seen anything like that.

Great - so I'm a one off

@Sid Wing let me test this.

Thanks Priyank

Yes, added -Dcoldfusion.runtime.remotemethod.matchArguments=false, with a space before it. No problems I'm aware of, and definitely not that one. However, I misspoke, only one of our two apps has remote methods with undeclared args, so I only added the jvm flag on that one dev server.

Yeah we appended that to the end of the JVM flags (with a space before it) - restarted the service - and then the isapi stuff started happening

@Sid Wing Does your application works even after this error?

No

ok, I did that and I see the error but it is working for me

Because isapi can't talk to tomcat

ok and your CF is running?

Yes and I can access the CFAdmin fine

Can you open the workers.properties and change the heartbeat value to 0 and then restart IIS

But that doesn't integrate wiis

\ColdFusion2021\config\wsconfig\{number}

I have heartbeat_interval and heartbeat_limit

heatbeat interval

and heartbeat_servlet_path

ok

Ok - it will have to be next week

sure

I am out of hours for this week

Take rest, will connect with you next week.

Thanks for the help

@Sid Wing: 'Now I have a "warn" in the isapi_redirect log "Reuse is set to false"'

Then I would suggest you test as follows, and see if it helps: 1. In

jvm.config

, append the flag

-Dcoldfusion.runtime.remotemethod.matchArguments=false

to the

java.args

property; 2. Add the attribute

allowedRequestAttributesPattern=".*"

to the

AJP connector

in `server.xml`; 3. Restart ColdFusion.

Right @BK BK I feel with connector recreation, it will solve the problem.

It was at 19

Then we will resume troubleshooting next week as you asked

When I upgrade to 20 - we have the issue with "arguments" -

My pleasure, @Sid Wing. By "isapi_redirect error" do you mean the application works, with the exception of

"[warn] ... AJP13 protocol: Reuse is set to false"

? If so, then add the attribute

allowedRequestAttributesPattern=".*"

to the

AJP connector

in

server.xml

as in this example

<Connector packetSize="65535" protocol="AJP/1.3" port="8024" redirectPort="8457" secret="9ce425e1-774c-4460-8507-40de1642d799" maxThreads="500" connectionTimeout="60000" tomcatAuthentication="false" allowedRequestAttributesPattern=".*" />

and see if that helps. Reference: https://stackoverflow.com/questions/60625395/111connection-refused-apache-reverse-proxy-and-tomcat-8-5-51-docker-compos

No - I mean that IIS and the isapi_redirect do not seem to be talking

And when you removed the matchArguments jvm flag it works?

after i uninstall and reinstall update 20 - apart from the arguments issue - it works

@Sid Wing, Out of curiosity: does the isapi_redirect issue occur when you use the JVM flag

-Dcoldfusion.runtime.remotemethod.matchArguments=true

? (Remember to restart ColdFusion) Another suggestion. Please share the following: • the contents of the file `workers.properties`; • the

AJP-connector (XML) element

from

/runtime/conf/server.xml

. I am thinking of the "match" between the two.

@Sid Wing Wait, I thought you said it was the matchArguments jvm flag that caused the issue, not update 20 itself. No?

Update 20 causes issues with a few functions that don't always have all possible arguments define (the change that is implemented in update 20) - when we use the JVM flag to circumvent that change - it breaks the IIS/isapi_redirect

Really wonder what's different here, where the one dev server I've used the flag on seems to be fine. Makes me worried that production may have this issue. I suppose as another test point I could add it to our other dev server, which doesn't need it since it doesn't have remote requests with undeclared args, but that doesn't really prove anything.

When I reapply update 20 - The apps work (up until the point where there is a remote request with undeclared args - then it fails with an error (as expected)

@Sid Wing pls send me an email, I will connect with you. Pls check your DM

We have undeclared remote args we can't immediately remediate too, so we need that flag, in one of our two big apps.

@Dave Merrill @BK BK - after working with Priyank - I believe the issue is resolved

Great, glad it's setting down. Can you explain a bit what the issue was, why you had it and we appear not to?

Looks like something to do with our felix cache. once it was removed and allowed to recreate - the flag works as expected

Hah! TBH, that's a known land mine, safest is to always delete the contents of that directory after applying an update. In more recent updates I've seen everything in there all have a modified date of just now, in which case you (probably?) don't need to do that, but it still might be safest.

@Sid Wing, thanks for the update. I am glad to hear the good news.

Passed regression testing with the JVM flag

Great, thanks Sid for confirming!

@priyank_adobe Should we always clear the felix cache manually after every update? As I said, I've been seeing the files in there all have a modified date of just now, without me doing anything. It's confusing when we do and don't need to clear it ourselves.

I can see your point, @Dave Merrill. I have witnessed many diverse, seemingly unrelated issues that arose because the Felix-cache folder was not emptied. Felix-cache houses many disparate bundles. As such, their interactions can be as complex as that of a quantum many-body. 😀 @priyank_adobe, given the frequency of the issues caused by not emptying the cache, would Adobe engineers consider including a process to automatically delete the Felix-cache directory after every update? For the time being at least?

@priyank_adobe It would be very helpful to know when we need to clear it manually. Seems like the update does it itself sometimes, but apparently not always.

@Dave Merrill The update itself clear the felix-cache however, I can see if we can add something in doc which will allow users to do that in case they encounter any problem.

@priyank_adobe Thanks for jumping in. That's what I've been seeing, that update does clear it. So what do you think happened to @Sid Wing? • That clear somehow didn't happen? • It didn't completely work? • It got corrupted after it was cleared? I'm asking because we haven't seen this issue in dev, and I don't want it to happen in production. If it got corrupted after it was cleared, what's to say it won't happen at some later point after the update?

It could be a possibility that the Felix was corrupted during the restart and removing and re-installing the updates couples of times.

Hmmm. So it seems like there's nothing we should do to make this less likely, just clear it if we see issues.

Apparently, Felix cache is not always automatically cleared after an update. Hence the following notice in the ColdFusion 2021 Update 17 release notes and the ColdFusion 2023 Update 11 release notes:

"Important 
After applying the update, we recommend clearing the Felix cache as a workaround for installing the HTMLToPDF package and verifying the Oracle data source."

Yes, but in practice, for a while now I've been seeing that everything in that directory was modified right then, so it seems it's been doing if "more often" than updates used to. But as we've all seen and you've said, "not always". Which doesn't seem constructive, but it is what it is.

@saghosh Can we check this from next release onwards. Let's add this point for our discussion with team.

Late to the conversation here, but I'm applying CF21 U20 now manually and last pop-up with the install button states "*‪Note:* ‪The hotfix installer will clear the cfclasses (//wwwroot/WEB-INF/cfclasses) and Felix cache (//bin/felix-cache) by default, for the selected instances only. This ensures proper reloading of updated classes and bundles." I still have gotten into the practice of stopping all instances and cleaning them manually post install, but maybe don't need to now?

I'm pretty sure I didn't get that popup. @priyank_adobe Is that new? And the behavior to do it always?

In case it matters, I downloaded the .jar and zip files yesterday and they match the hash values on the updates page.

@Sid Wing I’ve applied the update to several different servers now and implemented the jvm arg on a couple of them (all are using IIS)

"Felix cache has entered the chat" needs to be a t-shirt

That's an S class niche reference