Dear Hamilton Community, while trying the dynamic execution feature I noticed that I run into race conditions. That is expected in parallel executions. However since I used the SynchronousLocalTaskExecutor rather than the MultiProcessingExecutor, I was under the assumption that all tasks are run after one another. The word syncronus implies multiple agents, but in Java the syncronize keyword means only one threat at a time is allowed. Besides communicating my misunderstanding I wanted to ask if there are build in options for the management of race conditions. Or is this meant for pure parallel executable code? That would be fine for me I am just interested for the concept and plans. Thank you for all you help and awesome work so far!

Hi Richard! Can you share the code used to create the

Driver

? Also, how did you identify the race condition? Was there a raised exception? For synchronous execution, you should have something like

from hamilton import driver
from hamilton.execution.executors import SynchronousLocalTaskExecutor

dr = (
    driver.Builder()
    .with_modules(parallel_linear_basic)
    .enable_dynamic_execution(allow_experimental_mode=True)
    .with_remote_executor(SynchronousLocalTaskExecutor())
    .build()
)

To add — your assumption (with java) is correct — although there is confusion there 😆 The trick is you want to set the remote executor to be the synchronous local one — remote executor gets called for any of the “Parallel”-block tasks — E.G. between

Parallizable[…]

and `Collect`… Curious if that solves your issue!

@Thierry Jean That is exactly what i used. The race condition I observed was triggerd by

<http://pd.to|pd.to>__sql("my_table", data, 'append')

. The method failed because the Table was already created. That can't happen in a non parallel execution, beacuse it checks if the table exists before creation.

Thanks for the details. I'll try reproducing the error!

@Richard Schulz nope, I think you want to set the remote executor (

.with_remote_executor

) to the synchronous executor, not set the local one to the multiprocessing one. Can get you a snippet later! As to that error, it’s an odd case but worth looking into? Don’t think that code is doing what you want…