https://github.com/stitchfix/hamilton logo
Join Slack
Powered by
Hi guys, I have a question regarding Hamilton UI....
# hamilton-help
v
Hi guys, I have a question regarding Hamilton UI. As describe here (https://hamilton.dagworks.io/en/latest/hamilton-ui/ui/#changing-behavior-of-what-is-captured), I am able to disable capturing any data statistics of each run. However, I don´t get, what MAX_LIST_LENGTH_CAPTURE and MAX_DICT_LENGTH_CAPTURE are used for? Furthermore, I´d like to setup a hamitlon dataflow (using flowerpower), that cleans all stats older than a threshold time from the hamilton ui postgres db, but I need some more input. Which are the relevant database tables? Do I just need to delete all entries from the database tables older than the threshold time?
👀 1
e
Ok so this impar the max amount of data we capture from lists and dicts, I think it’s recursive but I’m afk and will have to look. Basically num results versus storage
And yes, there should be a table but I have to look at the schema. Will get back to you!
Might be a few tables but shouldn’t be too complicated, can probably write a quick Django script
🙏 1
s
Might be a few tables but shouldn’t be too complicated, can probably write a quick Django script
or add an endpoint for it.
👍 2
🙏 1
However, I don’t get, what MAX_LIST_LENGTH_CAPTURE and MAX_DICT_LENGTH_CAPTURE are used for?
Copy code
def some_func(...) -> list[str]:
   return [ some really long list]
This will limit the amount of ^ that is logged. i.e. first N elements < MAX_LIST_LENGTH_CAPTURE. Similarly for a dictionary.
👍 1
v
Adding an configurable clean up endpoint would be very useful!
e
Yeah good idea! @Volker Lorrmann will slot scoping this out tomorrow — sorry I got kind of slammed today
The bulk of the values would be stored here — https://github.com/DAGWorks-Inc/hamilton/blob/1057a40db370d235fce647d65872221365da6bfa/ui/backend/server/trackingserver_run_tracking/models.py#L60. That said, curious what you’re seeing/why you’re wanting to do this in the first place. Are you seeing the db grow too much in size? Or are you anticipating it? There are also blob stores it uses, which might take up disk space…
v
Hi @Elijah Ben Izzy Thanks. Should be this table in the db (see attached screenshot), right? Is it safe to simply delete records in this table, or should I just override the entry in the 'value' (jsonb) column with a "placeholder" or even delete it? And yes, we running dataflows frequently, that process rather large dataframes (billions of rows), the size of the db is increasing quickly and even results in hamilton ui being unresponsive (Note: This was experienced before https://github.com/DAGWorks-Inc/hamilton/pull/1232) As a follow up question to my first post: Does MAX_LIST_LENGTH_CAPTURE or MAX_DICT_LENGTH_CAPTURE have any effect on how many rows of the dataframe is tracked/captured?
e
I think it’s safe but I have not tested — should be effectively nullable and not impact any others. Replacing with a placeholder will break the UI if you look at it (given the schema/value), but I’d recommend just deleting one of them and trying to access it/seeing what happens. Otherwise I’ll be able to look on my own instance later today, will clear it up!
🙏 1
v
Ok. Will give it a try.
If it works. I´ll setup a FlowerPower pipeline for this job 😝
s
> Does MAX_LIST_LENGTH_CAPTURE or MAX_DICT_LENGTH_CAPTURE have any effect on how many rows of the dataframe is tracked/captured? No. For dataframes the default is to compute summary statistics over the entire dataframe.
🙏 1
e
OK, so clearing out the node run attributes table does appear to work — I’ve been experimenting locally
I’ve been using django but a sql statement can work
Otherwise an endpoint would be nice — scoped out a bit here: https://github.com/DAGWorks-Inc/hamilton/issues/1228
👏 1
2 Views
Open in Slack
PreviousNext
Powered by