This message was deleted.
# hamilton-helps
Slackbot
11/18/2022, 9:44 PMThis message was deleted.
e
Elijah Ben Izzy
11/18/2022, 9:45 PMMeanng running it in separate threads in parallel? I'd say yep -- I'm pretty sure we avoid global state and with the GIL we're good to go.
✅ 1
s
Stefan Krawczyk
11/18/2022, 9:45 PM@Filip Piasevoli to clarify, what part do you want in parallel?
f
Filip Piasevoli
11/18/2022, 9:45 PMone sec while I sketch out my use case
👍 1
Filip Piasevoli
11/18/2022, 9:49 PMI have a flask app deployed with nginx and gunicorn (read: copies of webapp on multiple workers). I'm planning on using hamilton within the guts of my webapp because it makes the transformations from payload -> response so much more organized.
I want to make sure that running hamilton in such an application deployed this way won't run into issues
e
Elijah Ben Izzy
11/18/2022, 9:51 PMOh awesome! FYI not sure if you've seen https://github.com/stitchfix/hamilton/blob/main/examples/async/README.md.
So I'm scouring for global state to double-check but I can't find anything -- if it does I'd call that a bug and we're happy to fix.
s
Stefan Krawczyk
11/18/2022, 9:52 PM@Filip Piasevoli I think if you create a new driver for each request, you’re definitely fine.
If you are trying to use the same driver, across multiple concurrent requests, we’ll have to double check — and yeah if that’s a bug we should figure out how to enable that.
f
Filip Piasevoli
11/18/2022, 9:53 PMGot it, I think I can get by creating a separate driver for each request. Thanks for the reference material! I'll check it out
s
Stefan Krawczyk
11/18/2022, 9:54 PM@Filip Piasevoli on second thought. I think you should be able to reuse the driver.
Gunicorn spins up separate processes of the flask app. Flask IIRC, doesn’t use threads. So each request, would be in its own process. So would not interfere with any other requests.
✅ 1
f
Filip Piasevoli
11/18/2022, 9:55 PMDoes each Driver create a lot of overhead or is it pretty lightweight since it's using most of Python's built in introspection?
s
Stefan Krawczyk
11/18/2022, 9:57 PMDoes each Driver create a lot of overhead or is it pretty lightweight since it’s using most of Python’s built in introspection?Depends on how big your DAG is? Worth timing it to convince yourself 🙂
f
Filip Piasevoli
11/18/2022, 9:58 PMMakes sense!
Thanks again for all the guidance. Have had some good momentum so far and just trying to learn more as I go
🙌 1
👍 1
e
Elijah Ben Izzy
11/18/2022, 9:58 PMIf I were you I'd try it in parallel with the same driver then yell at us for doing something stupid if it doesn't work 😆
FYI created https://github.com/stitchfix/hamilton/issues/234
2 Views