This message was deleted.

some other stuff: • we'd like to avoid just combining all the SQLs into 1 big scary SQL since our data scientists may not always need all of those columns, and we're also switching to Hamilton to move away from our old featurestore that is just 1 giant scary SQL • I saw there was a

async

decorator but the README said

async

doesnt play well with other decorators, and we're using

extract_columns

to break the dataframes from queries into individual columns

Good morning! So yes, this is a pretty common use-case. Just to be clear — you run a bunch of SQL operations that each load a dataframe of sorts, then join/manipulate them in some way, correct?

async

could work (although its still a little undeveloped). Our approach for parallelization has generally been to delegate to other frameworks. So, the

ray

and

dask

graph adapters both naturally do horizontal parallelism. The idea is its a quick swap for the driver, and you get the power of distributed systems. Some resources: • Quick post about scaling with ray • More information about horizontal scaling with ray/dask • dask hello_world • ray hello_world

@Culver McWhirter another idea (as a stop gap) would be to split things into two drivers: 1. One that uses Ray/Dask to parallelize and load the data. 2. Then one that does the downstream computation — passing in the output of the first driver; it is not always ideal to use Ray/Dask because the serialization cost between processes can outweigh any parallelization benefits. We currently don’t support arbitrary parallelization of a DAG, but your use case is definitely a motivating one to provide functionality for.

sorry for the very late response, i wasnt sure exactly what i wanted, so i wanted to put in more effort and see if i could actually get something working this is what i ended up doing (im sure its not ideal/perfect, but it does work):

Nice! Glad it works. OK, a bit confused — are you using the async decorator? If so, then why are you getting the event loop and adding it into another thread? Shouldn’t you get the benefits of async on a single thread?

yeah @Culver McWhirter we also have an AsyncDriver for Hamilton - which we can jump on a call to explain too. But a little more context on where you want this to run would help 🙂

this is my first dive into async with Python, so i might be doing stuff a little wrong the reason i ended up having to put

run_query()

and

get_results()

in new threads is because theyre not async functions themselves, so i couldnt

await

them

@Culver McWhirter do you have time now to jump on a quick call?