I've talked before about apparent cache corruption in a big cf 2018 app, where it starts throwing nonsense errors until we clear the template cache, which fixes them immediately. Of course I'd rather that didn't happen, but we've been unable to nail down a cause. Adobe has pretty much stopped responding, which is unfortunate, but I understand, somewhat anyway, because I don't have steps to reproduce. So my next stop is to try to automate recovering from it. Using <cfcache...> to clear the template cache is one piece, but the other thing that often happens in this case is that some datasources stop working. The fix we do for that is to refresh all the datasource, by verifying them all in cf admin. Here's my question: Is there some procedural equivalent to that action in the admin UI?

not sure what may be in the admin api for that, but couldn't you just simply do basic INFORMATION_SCHEMA queries to each datasource inside a try/catch without having to leverage the admin api at all?

Have you taken your code base and spun up another server and let it run for X amount of time to see if you still get the same issue. Would help begin to narrow down if its code vs server. Or just spin up Lucee and dont look back 🙂

You can probably tickle this i wrote it for lucee but I am sure it can be translated

component rest="true" restPath="/datasource"{

	/**
	 * Verify one or all datasource connections defined in the administrator or datasources.yml
	 *
	 * @dsn string default "all" the datasource name for the test
	 * @output false
	 * @return struct a json structure containing each datasource with the connection status any error message returned during the test and the timing of the test
	 */
	remote struct function verify(required string dsn restargsource="Path") httpmethod="GET" produces="application/json" restpath="verify/{dsn}"{
		var result = {"success"=true};
        local.pc = getPageContext();
		local.dsm = local.pc.getDataSourceManager();
		local.amd = getApplicationMetadata();
		result["result"] = {};
		local.dataSourceArray = StructKeyArray(local.pc.getConfig().getDatasourcesAsMap());
		if (structKeyExists(local.amd, "datasources")){
			local.dataSourceArray.addAll(StructKeyArray(local.amd.datasources));
		}
		if( arguments.dsn == "all"){
			for( var key in local.dataSourceArray){
				structInsert(result.result, key, verifydsn(local.pc, local.dsm, key, local.amd));
			}
		} else if (local.dataSourceArray.containsNoCase(dsn)){
			structInsert(result.result, dsn, verifydsn(local.pc, local.dsm, arguments.dsn, local.amd));
		} else {
			result.result[dsn] = {"con" = false,"message" = "no datasource definition"};
			result.success=false;
		}
		return result;
    }

	/**
	 * helper function verifies the connection to a datasource
	 *
	 * @pageContext the coldfusion page context
	 * @manager the datasource manager
	 * @dsn the datasource to verify
	 * @metadata the application metadata
	 * @output false
	 * @return struct containing the connection status elapsed time and any message generated during the test
	 */
	private struct function verifydsn(required any pageContext, required any manager, required string dsn, required struct metadata){
		var db = {}
		if (structKeyExists(metadata.datasources, dsn)){
			db = metadata.datasources[dsn]
		} else {
			var ds = pageContext.getConfig().getDatasource(dsn);
			db.username = ds.getUsername();
			db.password = ds.getPassword();
		}
		local.start = gettickcount();
		try{
			manager.releaseConnection(pageContext, manager.getConnection(pageContext, dsn, db.username, db.password));
			return {"con" = true,"message" = "connection OK", "time" = getTickCount() - local.start}
		} catch (any e){
			return {"con" = false,"message" = e.message, "time" = getTickCount() - local.start};
		}
	}
}

Thanks folks.

Rebuild a new Production and toss the old one. Prob save from the headache pretty straightforward.

The symptom is recurring patches of errors that are fixed by a cache clear, until the next time it happens. Rebuilding the server seems a weird approach to a fix, more or less saying a Hail Mary and jiggling the handle.

With commandbox you can redeploy an app in minutes in any flavor of CF. And this has been an ongoing issue for sometime for you, so a hail mary seems appropriate.

Its not what you think it is. Restart networking services, flush windows temp/ user_temp / and service_user temp / then restart iis / coldfusion If you have real disks (sata sas not SSD) then defrag them as well. Windows defragger isnt the greatest on busy sites. Additionally bump up your cached templates and GC cycle