https://github.com/stitchfix/hamilton logo
Join Slack
Powered by
Hi! I have several Hamilton DAGs running in parall...
# hamilton-help
l
Hi! I have several Hamilton DAGs running in parallel; each DAG runs a distinct type of analysis on some incoming data. Each DAG uses caching for most (but not all) nodes, and each DAG shares the logic for ingesting the data. This has worked perfectly fine for months until today, when all of a sudden I got hit by this error in several DAGs (full traceback in thread, to reduce spam):
Copy code
mrt 04 16:24:00 myhost sudo[841]:     cur.execute("INSERT INTO history (cache_key, run_id) VALUES (?, ?)", (cache_key, run_id))
mrt 04 16:24:00 myhost sudo[841]: sqlite3.OperationalError: attempt to write a readonly database
Unfortunately, I don't have access to this machine right now. This happened at a client, and a colleague lifted the logs for later inspection. I will have access to the machine later this week, but I would like some hypotheses on what went wrong. I would first look at simply the disk being clogged up, but we've put Hamilton's cache in 
/dev/shm/hamilton_cache
, which does not reside on disk, AFAIK. And the RAM equivalent of this problem would likely result in the OOM killer killing... well, something noticeable. But since this worked perfectly fine for months, I can't help but suspect a system-wide issue. But I might be wrong. Any suggestions on where to look?
Full traceback:
Copy code
mrt 04 16:24:00 myhost sudo[841]: ********************************************************************************
mrt 04 16:24:00 myhost sudo[841]: Traceback (most recent call last):
mrt 04 16:24:00 myhost sudo[841]:   File "/home/myuser/.local/share/virtualenvs/backend-84hNa8eQ/lib/python3.8/site-packages/hamilton/execution/graph_functions.py", line 294, in execute_lifecycle_for_node
mrt 04 16:24:00 myhost sudo[841]:     result = __adapter.call_lifecycle_method_sync(
mrt 04 16:24:00 myhost sudo[841]:   File "/home/myuser/.local/share/virtualenvs/backend-84hNa8eQ/lib/python3.8/site-packages/hamilton/lifecycle/base.py", line 955, in call_lifecycle_method_sync
mrt 04 16:24:00 myhost sudo[841]:     return getattr(adapter, method_name)(**kwargs)
mrt 04 16:24:00 myhost sudo[841]:   File "/home/myuser/.local/share/virtualenvs/backend-84hNa8eQ/lib/python3.8/site-packages/hamilton/caching/adapter.py", line 1377, in do_node_execute
mrt 04 16:24:00 myhost sudo[841]:     self._set_stored_metadata(
mrt 04 16:24:00 myhost sudo[841]:   File "/home/myuser/.local/share/virtualenvs/backend-84hNa8eQ/lib/python3.8/site-packages/hamilton/caching/adapter.py", line 763, in _set_stored_metadata
mrt 04 16:24:00 myhost sudo[841]:     self.metadata_store.set(
mrt 04 16:24:00 myhost sudo[841]:   File "/home/myuser/.local/share/virtualenvs/backend-84hNa8eQ/lib/python3.8/site-packages/hamilton/caching/stores/sqlite.py", line 147, in set
mrt 04 16:24:00 myhost sudo[841]:     cur.execute("INSERT INTO history (cache_key, run_id) VALUES (?, ?)", (cache_key, run_id))
mrt 04 16:24:00 myhost sudo[841]: sqlite3.OperationalError: attempt to write a readonly database
mrt 04 16:24:00 myhost sudo[841]: ERROR:hamilton.driver:-------------------------------------------------------------------
mrt 04 16:24:00 myhost sudo[841]: Oh no an error! Need help with Hamilton?
t
The errors point to 
sqlite3
, the default backend for the 
metadata_store
, which stores metadata about what node was executed, what code version, what data version, etc. The 
readonly
error has to do with how sqlite handles multiple connections.
After it failed, were you able to retry the workflow? did it fail again? If your code used to work, and this failed only once, my guess would be that a process or someone is locking the database by keeping write access
When you run a single DAG against a cache, you don't get write conflicts because 
node A
executes, read/write what it needs, then 
node B
does its thing, then 
node C
. These operations are very quick If you're running multiple DAG in parallel against the same cache, they could collide. If your cache grows really big, the metadata read operation required by the cache logic could become slower/more expensive and increase the likelihood of this error
Any idea how big the 
metadata_store
is ?
l
Unfortunately, I was not present at the moment, and the colleague had to wrap up, so I don't have any information about reproducibility etc 😕
I would have to check the size of 
metadata_store
on that machine - which I cannot do right now
Since it's in-memory, I wouldn't expect it to be huge (it's shut off at the end of the work day), but we were running an extensive validation run today with a lot of incoming image data, so it possibly finally grew enough to become a problem
During development we frequently clear the cache, so we didn't even think to check the size here. But there are indeed 4 processes using the same cache directory in parallel, so that seems like a plausible explanation
t
if you're using the 
metadata_store
with SQLite, it should be a single file and be easy to inspect. For something more robust, you could help build a Postgres backend (it shouldn't be too difficult if you're following the open source sqlite) or you could use the in memory cache for each of your parallel process and use some handwritten logic to 
load_from
/ `persist_to`; Your parallel processes will benefit from other's work only when they sync though
l
you could help build a Postgres backend
You mean, build a Postgres backend for Hamilton's caching store?
or you could use the in memory cache for each of your parallel process
Indeed, having separate caches for each process seems like the cleanest approach.
I'm curious: like I said, we're storing the cache in 
/dev/shm/
, so it already is in-memory, it's just not using your built-in 
InMemoryMetadataStore
etc. Is there a fundamental difference between instantiating that class vs manually specifying a 
tmpfs
path? Is there some significant logic there that we're leaving on the table?
t
Ok, backtracking a bit: • the default 
metadata_store
writes metadata to an SQLite database; on the next execution, you can retrieve metadata from disk • the main limitation is having multiple DAG execution accessing the same cache concurrently. • if you must have concurrent access, Postgres could be a good option because it supports concurrent connections. • the 
InMemoryMetadataStore
doesn't write data to SQLite / to disk; this is useful in notebook or interactive python or if you loop over 
Driver.execute()
in a single script. After the process terminates, the metadata disappears if you're not persisting it So far I was assuming that you were running the "same" dataflow in parallel. If you're running different dataflows, you can set the cache path such that each process writes to a different SQLite database. This should entirely prevent the error your encountered
Copy code
dr = Builder().with_modules(...).with_cache(path="/dev/shm/process1").build()
l
Thank you very much for your immediate feedback! Much appreciated.
l
Ah, yes indeed, it's different dataflows! Some nodes are shared, but the drivers are instantiated separately, so your last suggestion is also what I had in mind. Sorry if I was unclear. I'm just very surprised that we didn't hit this issue sooner. Today was not the heaviest usage of the entire system.
l
Yep, agreed with the last solution 😉
👍 1
🙌 2
l
In any case, like @Louwrens said, thank you so very much for the immediate feedback! 🙏
❤️ 1
🙌 2
2 Views
Open in Slack
PreviousNext
Powered by