A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hello Team, I am new to Datahub - our team is working on integrating it with Airflow - we have found the documentation mighty helpful :slightly_smiling_face:
I am stuck on a weird issue - we have 2 different Airflow instances which we are integrating with Datahub. We see only one Airflow instance in Datahub which has DAGs from both the instances, however for DAGs that have the same name - we see only one pipeline, also the task list is of just of that DAG - is there a way to separate the 2 Airflows ? Request your guidance - Thank you!

Hi <@U04HD266EMB>, you may be able to use `datahub.cluster`  to solve this issue. See step 4 in this doc: <https://datahubproject.io/docs/lineage/airflow/#using-datahubs-airflow-lineage-plugin>

hi <@U042KCFV9GX>,
I just wanted to follow up on this topic since this post is the most recent one that refers to this issue.

I've found the following tickets that are related to this:
<https://github.com/datahub-project/datahub/issues/3001>
<https://feature-requests.datahubproject.io/p/support-multiple-platfrom-instances>

Multiple platform instances is a cool feature, but in case of push-based ingestion of Airflow it doesn't help since allowed fabric names are pre-defined. Would there be any possibility to make it more flexible? Is that something that you potentially have on the radar?
Thanks

Hi <@U04KLUC9MFX>,  what type of flexibility are you imagining here? I think this is definitely something we’re interested in pushing, whether in Open source or Managed DataHub. It’s just a matter of getting it scoped and delivered

Acryl Data delivers an easy to consume DataHub platform for the enterprise - sign up here: <https://www.acryldata.io/sign-up>

I'd imagine that environment/fabric type should be allowed to be set to any string. That would be very useful to separate metadata from different environments emitting dataflows and datajobs to one centralized datahub. For example in a setup with multiple dev Airflow instances.

In such setup you could also imagine multiple airflow instances that share the same dags, thus the same IDs. They would override each other in datahub unless they use different fabric names. Currently you would have to name a dev1 airflow as `DEV` fabric, while dev2 would have to be named something else, which would be very confusing.

just wanted to chime in here that being able to set the platform_instance for airflow ingestion is something we are interested in as well