A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hello everyone,
I implemented a DataHub integration with Airflow that works for version 2.5+ (the version I use), but I'm having a little problem with my database volume. Seem that the dataProcessInstance entity is created with a mutable urn, <https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/api/entities/dataprocess/dataprocess_instance.py#L79|as you can see here>, and the problem is that every data pipeline run generates lots of new dataProcessInstances with a different URN. I have pipelines thar runs frequently and have a lot of taks. Because of that, I have about 1.5 milion new entities a month.  I'm doing something wrong?

I'm using the default retention policy. I think I will need to create a custom one based on <https://github.com/datahub-project/datahub/blob/master/docs/advanced/db-retention.md|this documentation>, right?

<@U02G4B6ADL6> Any ideia here?

<@U01GZEETMEZ>?

HI <@U0398JJG49W>, thanks for the patience! I’ll float this one up and get you an answer :slightly_smiling_face:

Thanks for the ping- we’ve been a bit swamped with release stuff, I’ll get back to you here!

I believe you’ll need to create a custom policy though