Hello, is anyone in the community working on creat...
# ingestioni
incalculable-ocean-74010
03/05/2021, 6:02 PMHello, is anyone in the community working on creating a helm chart for the metadata-ingestion module?
g
gray-shoe-75895
03/05/2021, 6:48 PMThere is a docker image for metadata-ingestion https://hub.docker.com/r/linkedin/datahub-ingestion, but we haven't documented it well yet
i
incalculable-ocean-74010
03/05/2021, 6:53 PMI'm actively working on creating a helm chart for it, I was wondering if somebody else was and I could help out instead
e
early-lamp-41924
03/05/2021, 7:02 PMThat would be great! Curious about your ingestion setup. Do you use airflow to continuously push metadata changes in?
i
incalculable-ocean-74010
03/05/2021, 7:15 PMThe objective is for it to be templated in a way that generates multiple cronjobs (one per datasource if you will)
incalculable-ocean-74010
03/05/2021, 7:16 PMOn my side we don’t yet use airflow but instead an azure managed service. We intend to swap but it is not a priority. I’m actually curious, why did you opt for airflow?
e
early-lamp-41924
03/05/2021, 7:21 PM+1 to adding cron job helm template. We currently only run adhoc jobs atm, so was wondering what you’re setup was.
i
incalculable-ocean-74010
03/15/2021, 5:16 PM