https://github.com/stitchfix/hamilton logo
Join Slack
Powered by
Hello :wave: I was wondering if <we can make cachi...
# hamilton-help
l
... is it OK to simply 
try: ... except
around 
dr.execute()
?
s
@Louwrens interesting thought. There would need to be some logic to know what not to cache in the case of an error. Do you have a use case where this would be helpful?
l
Thanks for your attention!
Our situation is that we have dags that perform analyses of photo data resulting in outcomes containing some score of anomaly which is later collected and used to classify the object in the photos. These photos are shot by a robot that manipulates those objects and delivers the photo data to shared memory blocks. The dags are executed in different processes but they have explicit data loading nodes that sit and wait for particular shared memory blocks until those blocks are populated (btw we use Redis to listen for this event). This goes all very well and, unless there is an exception and the robot runs into an error or user abort. Then those blocks will never be populated and I wish for the dags to abort gracefully. Nodes upstream of the shmem loading node are not invalidated by an abort, anything cached up to this point is harmless. In fact, I believe in fact that most of the upstream nodes are loaded from cache already and should remain for the next run. I've created a custom exception that is raised when this "hang up" event is issued by the robot so that analysis workers (long lived processes) can safely abort the current Hamilton execution and clean up their states as to get ready for the next run.
What do you think of this scenario, any advice?
s
Thanks. So how would the graceful error adapter help you here? I assume you want to use the caching adapter because it would speed things up the next time things are run since it could just resume from where it left off?
l
haha! I jumped at thinking I needed the graceful error adapter, but you're absolutely right, I don't need the feature it brings in this case, that saves some worries. Thanks. I thought about using it here but it is a memory from the past and for another use case: where we were discussing running a multitude of analyses (e.g. subdags?) as (within?) a large dag. and that one analysis might already conclude other analyses are no longer needed: it should trigger early aborting of the other analyses....
👍 2
Thanks again. Seems work. I just have some clutter in my logs. How can I suppress the traceback such as:
Copy code
ERROR:hamilton.execution.graph_functions:
********************************************************************************
> get_photo_data_list [modules.module_data.get_photo_data_list()] encountered an error<
> Node inputs:
{'get_photo_metadata_list':  ....
 'photobook': ....
********************************************************************************
Traceback (most recent call last):
  File "/home/louwrens/.local/share/virtualenvs/backend-pjVm9lev/lib/python3.10/site-packages/hamilton/execution/graph_functions.py", line 294, in execute_lifecycle_for_node
    result = __adapter.call_lifecycle_method_sync(
  File "/home/louwrens/.local/share/virtualenvs/backend-pjVm9lev/lib/python3.10/site-packages/hamilton/lifecycle/base.py", line 955, in call_lifecycle_method_sync
    return getattr(adapter, method_name)(**kwargs)
  File "/home/louwrens/.local/share/virtualenvs/backend-pjVm9lev/lib/python3.10/site-packages/hamilton/caching/adapter.py", line 1258, in do_node_execute
    result = self._execute_node(
  File "/home/louwrens/.local/share/virtualenvs/backend-pjVm9lev/lib/python3.10/site-packages/hamilton/caching/adapter.py", line 829, in _execute_node
    result = node_callable(**node_kwargs)
s
set the logger for that module to CRITICAL, but it would suppress logs from the module entirely.
👍 1
l
Thanks, but if possible I would like to receive other (unexpected) errors... can I?
s
yep -- I think it's just a matter of catching the right exceptions. -- but logging them is at the module level or not; so if there was another error you wouldn't get that logging if you turned it off. You'd have to log it yourself since that exception is raised.
Open in Slack
PreviousNext
Powered by