I am trying to utilize the parallel feature in a queryeach I cfml #adobe

I am trying to utilize the parallel feature in a q...

Mark Berning

12/14/2022, 4:49 PM

I am trying to utilize the parallel feature in a queryeach. I am new to the whole parallel capability. What I am trying to do is loop through a query that invokes a CFC function in which makes an API call to AWS and returns a structure. This usually takes a couple of seconds. Then I want to take a value from the structure and update SQL server DB. When I don't use the parallel option all of the values returned correctly. When I do use the parallel option all of the values are exactly the same.

Mark Berning

12/14/2022, 5:00 PM

Here is my non parallel code

Copy code

<cfquery name="getVlabs" datasource="#application.config.DSNr#" maxrows="10">
	SELECT *
	FROM vlabResources
	WHERE BuildTemplateID is NULL
</cfquery>

<!--- TODO: loop through and retrieve BT value --->
<!--- TODO: ADD the BT to DB --->

<cfset tick = GetTickCount()>
<cfscript>

AWS = createObject("component","cfcomponents.amazon_api" );

_LT = structNew();

getVlabs.each (function (r,cr){
	// get BT
	_LT = invoke(AWS,"GetLaunchTemplate",{LTID="#r.vlabID#"}).data.templateReference?:'';
	WriteOutput(_LT);
	WriteOutput("<BR>");
}, false);

</cfscript>
<cfset tock = GetTickCount()>
<cfset time = tock-tick>
<cfoutput>it took #time#ms</cfoutput>

Output:

Copy code

bt-26f392d5bdb54c09b92164bab45c5816








it took 27101ms

Mark Berning

12/14/2022, 5:04 PM

When I turn on the parallel option:

Copy code

<cfset maxthreads = 10>
<!--- TODO: Query for LT without BT --->

<cfquery name="getVlabs" datasource="#application.config.DSNr#" maxrows="10">
	SELECT *
	FROM vlabResources
	WHERE BuildTemplateID is NULL
</cfquery>

<!--- TODO: loop through and retrieve BT value --->
<!--- TODO: ADD the BT to DB --->

<cfset tick = GetTickCount()>
<cfscript>

AWS = createObject("component","cfcomponents.amazon_api" );

_LT = structNew();

getVlabs.each (function (r,cr){
	// get BT
	_LT = invoke(AWS,"GetLaunchTemplate",{LTID="#r.vlabID#"}).data.templateReference?:'';
	WriteOutput(_LT);
	WriteOutput("<BR>");
}, true,"10");

</cfscript>
<cfset tock = GetTickCount()>
<cfset time = tock-tick>
<cfoutput>it took #time#ms</cfoutput>

Output:

Copy code

bt-26f392d5bdb54c09b92164bab45c5816
bt-26f392d5bdb54c09b92164bab45c5816
bt-26f392d5bdb54c09b92164bab45c5816
bt-26f392d5bdb54c09b92164bab45c5816
bt-26f392d5bdb54c09b92164bab45c5816
bt-26f392d5bdb54c09b92164bab45c5816
bt-26f392d5bdb54c09b92164bab45c5816

bt-26f392d5bdb54c09b92164bab45c5816
bt-26f392d5bdb54c09b92164bab45c5816
it took 4316ms

Mark Berning

12/14/2022, 8:14 PM

I think it is related to the AWS CFC object. Since the native

invoke

command reuses the object instance it is stepping on it (thread safe?). replacing the invoke command with this

Copy code

_LT = invoke("cfcomponents.amazon_api","GetLaunchTemplate",{LTID="#r.vlabID#"}).data.templateReference?:'';

Output

Copy code

bt-26f392d5bdb54c09b92164bab45c5816








it took 3696ms

mtbrown

12/14/2022, 10:20 PM

It's hard to say for sure without seeing the implementation of the function you're invoking. But just based on the code shared, it's not clear why you're declaring

_LT

outside of the closure. It makes more sense to

var

scope it based on the usage here.

👍🏻 1

zackster

12/15/2022, 2:34 PM

keep in mind that parallel has some overhead creating a pageContext for each thread

👍🏻 1

zackster

12/15/2022, 2:35 PM

everything should always be var scoped with threads

👍🏻 1

Mark Berning

12/15/2022, 3:28 PM

Thanks for the tips guys. Another thing I noticed is that if you don't specify the maxThreadCount, it will not run in parallel. Even though there is a default in CF Administration of 10 and parallel is set to true, it will not run in parallel without a value.

zackster

12/15/2022, 3:39 PM

you batch and use parallel to avoid the overhead, so do like two threads, each doing a batch of 5 jobs

Open in Slack

Previous Next