Hi, I’m Jensen and I just deployed datahub on k8s ...
# all-things-deploymenta
adventurous-scooter-52064
07/09/2021, 4:58 AMHi, I’m Jensen and I just deployed datahub on k8s with the guides here.https://github.com/acryldata/datahub-helm/tree/master/charts/prerequisites
this might be a silly question, if I want to connect aws athena & redshift on the prerequisites charts how can i do it? I just wanna test how it will look like after connecting datahub with aws.
https://datahubproject.io/docs/metadata-ingestion The guide here seems to only explain how to ingest it through docker or locally. is there a fast way to deploy an ingestion with k8s?
e
early-lamp-41924
07/09/2021, 5:06 PMHi. The quickest way is to use the ingestion-cron
early-lamp-41924
07/09/2021, 5:08 PMSave the recipe yaml file you created by following the guide, and upload it as a configmap
early-lamp-41924
07/09/2021, 5:08 PMkubectl create configmap sample-recipe --from-file=sample_recipe.ymlearly-lamp-41924
07/09/2021, 5:08 PMThen add this to the values.yaml file
early-lamp-41924
07/09/2021, 5:08 PM Copy code
datahub-ingestion-cron:
enabled: true
image:
repository: linkedin/datahub-ingestion
tag: "head"
crons:
demo:
# Daily at midnight (we may want to offset this to not conflict with other processes)
schedule: "0 * * * *" # Every hour
recipe:
configmapName: sample-recipe
fileName: sample_recipe.ymlearly-lamp-41924
07/09/2021, 5:08 PMnote you can add multiple under crons
early-lamp-41924
07/09/2021, 5:08 PMso one for athena and one for redshift
early-lamp-41924
07/09/2021, 5:09 PMThat “demo” can be changed to anything as it’s just an identifier for which job it’s running so in your case you would change it to “athena” or “redshift”
early-lamp-41924
07/09/2021, 5:09 PMThe other way of doing it is to expose gms through ingress
early-lamp-41924
07/09/2021, 5:09 PMand use airflow to run the recipes!
early-lamp-41924
07/09/2021, 5:10 PMor any other external scheduling software
a
adventurous-scooter-52064
07/10/2021, 1:11 AMthanks dexter! I ended up doing the same thing 🙂 I was looking through the subcharts and found a cron….poc it and it worked…thanks
adventurous-scooter-52064
07/10/2021, 1:12 AM@early-lamp-41924 is there any chance datahub will create a recipe for metabase or redash just like superset? or we just have to develop it ourselves?
adventurous-scooter-52064
07/10/2021, 1:15 AM^^^btw i didnt use airflow nor gms….i connected it with datahub-kafka…works like a magic!
e
early-lamp-41924
07/10/2021, 1:30 AMit would be great to get contribution there but feel free to ask inthe #datahub-ingestion channel to see if theres anyone working on it!
✅ 1
2 Views