Hello! I have a Python (

3.8.18

) process which starts several Hamilton (

1.84.0

) DAGs, each in one subprocess (

multiprocessing.Process

). I want to be able to terminate all of them when an event (a stop request) happens in the master process. I have added a

SIGTERM

handler in the subprocesses, and I am using

Process.terminate()

from the master process to actually send the signal. The handler does some cleanup, and then issues a

sys.exit(signum)

. The handler seems to be triggered correctly, and the exit is triggered as well, but something (apparently related to caching, which I do use) breaks, and the subprocesses end up in a sleeping state (

S

in

htop

) and somehow do not terminate. The error is this:

sys.exit(signum)
SystemExit: 15

During handling of the above exception, another exception occurred:
...
File "/my/venv/lib/python3.8/site-packages/hamilton/caching/adapter.py", line 1424, in post_node_execute
    assert data_version is not SENTINEL

Process.kill()

works just fine, but I would rather not go for the nuclear option if I can help it 😅 How can I abort a Hamilton DAG execution more-or-less cleanly?

Hmm, hanging appears to be a bug — a few different state stuff that isn’t cascading properly.. Curious — could you create a simple example that we could look at/rerun to debug? Maybe open an issue? Would need a bit more information to debug. On first glance this looks like it has to do with try/finally/except blocks and it getting stuck somewhere. Also 3.8 is EOL now (we’re going to be removing support for it soon), but I highly doubt that’s the cause of this — worth trying in 3.9 as well just to see if this is a python-specific thing.

Hey Elijah, thanks for your reply! Unfortunately I must devote my time to another thing today, but I will get back to you re: the minimal example sometime next week! As for the Python version: yes, I'm painfully aware we are behind, we already bumped into a 3.8 limitation last week 😓 There is another dependency which allegedly requires 3.8, and I don't have the time at the moment to test that assumption. But that dependency doesn't directly touch Hamilton, so I can test another Python version without it.

Makes sense! To be clear hamilton still works on 3.8, and I doubt this is the issue, but when you have the repro we can look in more!