Two Quick question:
1. Is there a way to have inge...
# ingestions
some-crayon-90964
01/04/2022, 9:52 PMTwo Quick question:
1. Is there a way to have ingestion runs frequently (such as BigQuery -> Datahub every 12 hours);
2. How do we config the yml file such that it lists all projects without we hardcoding all projects we have
Thanks in advanced!
some-crayon-90964
01/04/2022, 9:57 PMFor the 2nd question, below file is what i am referring to: https://github.com/linkedin/datahub/blob/master/metadata-ingestion/examples/recipes/bigquery_to_datahub.yml
h
helpful-optician-78938
01/04/2022, 10:18 PMHi @some-crayon-90964, for (1), you can use a scheduler like airflow that you can invoke ingestion from.
helpful-optician-78938
01/04/2022, 10:29 PMFor (2), you don't have to hardcode the project ids. If it is not provided, they will be auto-detected based on the credentials. See the documentation for more details.
b
big-carpet-38439
01/05/2022, 1:25 AMLet us know if this clears it up!
s
some-crayon-90964
01/05/2022, 2:27 PMThanks for your reply @helpful-optician-78938, will the ingestion process take all projects? Or just first one it sees?