hi everone, I have a question regarding the ingest...
# getting-startedq
quick-raincoat-6968
11/10/2021, 7:54 PMhi everone, I have a question regarding the ingestion scheduling. How will I best orchestrate the ingestion runs periodically to datahub? I would like to use airflow for that. Anybody has some experiences? Thanks a lot for your help
m
mammoth-bear-12532
11/10/2021, 7:55 PM
Hi @quick-raincoat-6968 , yes most folks in the community use Airflow for scheduling it.
mammoth-bear-12532
11/10/2021, 7:56 PM
we have a few sample DAGs in the repo
mammoth-bear-12532
11/10/2021, 7:56 PM
mammoth-bear-12532
11/10/2021, 7:56 PM
q
quick-raincoat-6968
11/10/2021, 7:57 PMI need to be more specific. I have running airflow already out of the machine where I'm running datahub. So how will I trigger kind of the command "datahub ingest -c ..." via my airflow?
quick-raincoat-6968
11/10/2021, 7:58 PMor will I just need to specify my path to the datahub-gms in your example in the pipeline?
quick-raincoat-6968
11/10/2021, 8:01 PMthanks a lot to your insights
m
mammoth-bear-12532
11/10/2021, 8:02 PM
if you are running a local airflow already, are you following this guide? https://datahubproject.io/docs/docker/airflow/local_airflow/
mammoth-bear-12532
11/10/2021, 8:04 PM
if so, you just have to add a dag with the recipe to your airflow installation
q
quick-raincoat-6968
11/10/2021, 8:18 PMno I'm running on a GoogleComputeEngine an airflow that I use to orchistrate other stuff. and now I want to orchistrate the datahub ingestion as well and I'm looking for the best way to launch the ingestion jobs from there? does this make sense
quick-raincoat-6968
11/15/2021, 12:11 PMhi, sorry an other question. I would like to follow your proposition. Using the datahub.ingestion.run.pipeline. we need to make these request to the datahub-rest authorized in order to decline others the connection. I couldn't find a proper documentation of how I'm able to configure the sink more sophistically, can you point me to some resources? thanks a lot
3 Views