Hi guys,
I have a pipeline, that processes a lot ...
# hamilton-helpv
Volker Lorrmann
11/04/2024, 11:40 AMHi guys,
I have a pipeline, that processes a lot of data on each run, and runs about 10x an hour. When I am using hamilton ui (deployed using docker-compose) to track this pipeline, the data stored by hamilton ui becomes several GB within a short period of time and the "Runs" overview page becomes unresponsive.
I wonder, if it is possible to clean up the hamilton ui database periodically and/or to only track the success info of each pipeline node without the node results.
s
Stefan Krawczyk
11/04/2024, 3:09 PMFirst of all, congrats on all the usage!
Stefan Krawczyk
11/04/2024, 3:09 PMYeah you should be able to delete what you need by going to the DB
Stefan Krawczyk
11/04/2024, 3:10 PMdo you want to create an issue for this? we can then:
1. document a workaround
2. open up an endpoint for doing so
Stefan Krawczyk
11/04/2024, 3:11 PMotherwise configuring what to capture is something we’ve thought about - it’s more how would you like to specify that? via some configuration? or?
v
Volker Lorrmann
11/04/2024, 3:12 PMYes. I can open an issue.
One more question. Saw you blogpost on OpenLinage.
I wonder, how OpenLineage + Marqueze compares to Hamilton UI.
s
Stefan Krawczyk
11/04/2024, 3:50 PMYeah Hamilton UI doesn't ingest open lineage events yet.
But if you're just using Hamilton you have the same information and more.
🙏 1
Stefan Krawczyk
11/04/2024, 4:04 PMI also know the team driving Marquez and so we've shared a few ideas
👍 1
v
Volker Lorrmann
11/04/2024, 10:18 PMThanks.
s
Stefan Krawczyk
11/04/2024, 10:44 PM@Volker Lorrmann do let us know priority on this too.