Join Slack
Powered by
Whats the best way to run ingestion? Have a server...
# ingestion
c
careful-insurance-60247
11/03/2021, 6:32 PM
Whats the best way to run ingestion? Have a server just run multiple recipes with a cronjob?
thank you
1
g
green-football-43791
11/03/2021, 6:42 PM
That works- another option is to use airflow for scheduling. Here is an example dag for how you may want to do that:
green-football-43791
11/03/2021, 6:42 PM
https://github.com/linkedin/datahub/blob/master/metadata-ingestion/src/datahub_provider/example_dags/generic_recipe_sample_dag.py
c
careful-insurance-60247
11/03/2021, 7:13 PM
Thanks Gabe. We are actually not using airflow.
3
Views
Open in Slack
Previous
Next