Is there a way to make sure, Parallelizable / Coll...
# hamilton-helpv
Viktor
09/20/2024, 2:00 AMIs there a way to make sure, Parallelizable / Collect is NOT run in parallel, but consecutively, in yielded order?
e
Elijah Ben Izzy
09/20/2024, 2:05 AMYep! You can use the SynchronousLocalTaskExecutor to run it
❤️ 1
👍 1
Elijah Ben Izzy
09/20/2024, 2:06 AMImage from iOS.jpg
Elijah Ben Izzy
09/20/2024, 2:06 AMSorry for the screen cap on my phone
Elijah Ben Izzy
09/20/2024, 2:06 AMBut basically you set the “remote” executor (which means anything between a parallel and collect) to be the local one
Elijah Ben Izzy
09/20/2024, 2:07 AMThis is assuming you want to run locally — if not executors have concurrency maximums that you can set
v
Viktor
09/20/2024, 2:10 AMBeautiful. I'll be using it for partitioning a download by date.
I'm seeing it in use here, too — but the local executor bit was missing there. Thank you for pointing this out.
https://hamilton.dagworks.io/en/latest/concepts/builder/#enable-dynamic-execution
e
Elijah Ben Izzy
09/20/2024, 2:10 AMMakes sense! Easy way to do a map-style operation. And yeah, we can probably make the docs a bit clearer as to what “remote” and “local” mean for executors, it’s a bit confusing
👍 1
v
Viktor
09/20/2024, 2:13 AMtrue, it's not clear, what would be exectured remotely, and that makes it sound like the remote executor would be used only where a "remote execution" mode is declared for a part of the dag.
e
Elijah Ben Izzy
09/20/2024, 2:13 AMYep, we can fix up the docs