https://github.com/stitchfix/hamilton logo
Join Slack
Powered by
Hello guys, I was trying out Hamilton with asyncio...
# hamilton-help
l
Hello guys, I was trying out Hamilton with asyncio, and hoped I could 
visualize_execution
of my 
async_module
. But I hit an issue, what am I doing wrong here?
Copy code
async def driver_visualize_execution(foo:str="foo"):
    input_data = {"foo": foo, "bar": "bar", "some_data": dict(lol=42)}
    dr = await builder.build()
    await dr.visualize_execution(["pipeline"], output_file_path=f"dag_execute_{foo}.png", inputs=input_data)
... gives me this error:
Copy code
Traceback (most recent call last):
  File "dataflow_async_run.py", line 59, in <module>
    asyncio.run(main())
  File "C:\Users\User\.pyenv\pyenv-win\versions\3.8.10\lib\asyncio\runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "C:\Users\User\.pyenv\pyenv-win\versions\3.8.10\lib\asyncio\base_events.py", line 616, in run_until_complete
    return future.result()
  File "dataflow_async_run.py", line 39, in main
    await driver_visualize_execution("foo")
  File "dataflow_async_run.py", line 21, in driver_visualize_execution
    await dr.visualize_execution(["pipeline"], output_file_path=output_file_path, inputs=input_data)
TypeError: object Digraph can't be used in 'await' expression
e
I think you can avoid the 
await
in 
visualize_execution
— it’s not a coroutine!
E.G.
Copy code
async def driver_visualize_execution(foo:str="foo"):
    input_data = {"foo": foo, "bar": "bar", "some_data": dict(lol=42)}
    dr = await builder.build()
    dr.visualize_execution(["pipeline"], output_file_path=f"dag_execute_{foo}.png", inputs=input_data)
l
ahh, yes, thanks a bunch !
btw, how can I instantiate 
dr
outside the scope of this (or another) function, like a global object? I guess it has to be `await`ed, from within an 
async
function...
s
> btw, how can I instantiate 
dr
outside the scope of this (or another) function, like a global object? I guess it has to be `await`ed, from within an 
async
function... @Louwrens yep correct. (though I think 
asyncio.run(…)
might work — as well as what Elijah mentions below). Are you using it within a web-service? or?e
l
yeah, I'm using 
asyncio.run()
now, I'm not really looking to integrate it in a web service. I want to try to use Hamilton in doing a classification of photos produced by a robot, there is a lot of home made existing code and I want to disentangle it and begin with a small clean project
👍 1
BTW2, do you have pointers to using 
shared_memory
objects correctly with Hamilton? The photos are currently stored there, and I was thinking I might need to write a dataloader decorator or something...
s
what do you mean?
l
well, we need to do calculations on those photos right? and I would like to pass a reference to the shared memory objects that existing code produces... can I have nodes in the dag accept shared memory references? or should I get the `numpy.ndarray`out of the memory first and pass that as input to the nodes?
okay the historical reason for shared memory objects was because the old classification routines is using Celery in an attempt to do the classification functions in parallel, that would `json`serialize the photo data and that's very inefficient... we think we can do the lot with 
asyncio
instead. The robot has one PC, so celery seems overkill. In Hamilton, the data passed between nodes remains in-memory right? Maybe using 
shared_memory
might also be redundant?
s
ah yes — passing pointers should be fine.
👍 1
data passed between nodes is in memory yes
👍 1
so whether it’s numpy or some object — passing around the reference should be fine. There be dragons if you mutate it (depends on context if there are any).
👌 1
e
@Louwrens FYI, you can use the async builder outside of an async context. Use `build_without_init`:
Copy code
def driver_visualize_execution(foo:str="foo"):
    input_data = {"foo": foo, "bar": "bar", "some_data": dict(lol=42)}
    dr = builder.build_without_init()
    await dr.visualize_execution(["pipeline"], output_file_path=f"dag_execute_{foo}.png", inputs=input_data)
Note that, if you have any async initialization (E.G. the Hamilton UI), you’ll want to call 
await dr.ainit()
later on
l
Thanks! (that example works -- without the 
await
😉 )
e
Ahh, yep, sorry copy/pasted 😆
Open in Slack
PreviousNext
Powered by