General Question- How to we connect an existing ai...
# ingestionh
handsome-football-66174
08/20/2021, 5:28 PMGeneral Question- How to we connect an existing airflow environment to the datahub docker ? Also in the documentation https://datahubproject.io/docs/metadata-ingestion/#lineage-with-airflow , where does the airflow.cfg be present (new to airflow as well) ?
e
early-lamp-41924
08/20/2021, 6:59 PMIs airflow running in the same cluster?
early-lamp-41924
08/20/2021, 7:00 PMAlso, will you be using the lineage backend? or operators?
h
handsome-football-66174
08/20/2021, 7:04 PM@early-lamp-41924 - Yes will be using Airflow lineage backend, though airflow might not be running on the same cluster
e
early-lamp-41924
08/21/2021, 1:02 AMIn that case, you need to expose the gms through ingress
early-lamp-41924
08/21/2021, 1:03 AMand then go through steps 2 and 3
early-lamp-41924
08/21/2021, 1:04 AMseems like its created on airflow deploy https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html
h
handsome-football-66174
08/27/2021, 1:10 PM@early-lamp-41924 - Can you share the steps for exposing gms through ingress ?
e
early-lamp-41924
08/27/2021, 3:33 PMShould be very similar to setting up ingress for frontend! https://datahubproject.io/docs/deploy/aws#expose-endpoints-using-a-load-balancer
early-lamp-41924
08/27/2021, 3:33 PMIf you already set up alb controller, you can skip to the helm values.yaml change at the end of this section
early-lamp-41924
08/27/2021, 3:34 PMYou will have to add the ingress section to datahub-gms instead of frontend
h
handsome-football-66174
08/30/2021, 2:34 PM@early-lamp-41924 - I followed the steps and was able to connect Datahub and airflow both running on different servers. I had overlooked pip install acryl-datahub[airflow] step and so the lineage was not visible.
e
early-lamp-41924
08/30/2021, 3:46 PMGreat news!!!