https://datahubproject.io logo
Join Slack
Powered by
I am trying to get the rendered yaml with ingestio...
# ingestion
a
I am trying to get the rendered yaml with ingestion-cron enabled, getting this error. Not super familiar with Helm, can someone help? Thanks!!
Copy code
>> helm install datahub datahub/datahub --set-string datahub-ingestion-cron.enabled=true --dry-run
>> dependencies.go:49: Warning: Condition path 'datahub-ingestion-cron.enabled' for chart datahub-ingestion-cron returned non-bool value
e
Can you do --set instead of --set-string ?
a
@early-lamp-41924 if I use --set, the command runs with no error, but the output yaml doesn’t include the datahub-ingestion-cron I was looking for
e
Ah so you need to add more details to the values like specifying the configmap name, key, and cron schedule
Let me send you an example!
Copy code
datahub-ingestion-cron:
  enabled: true
  crons:
    sample:
      schedule: "0 * * * *" # Every hour
      recipe:
        configmapName: sample-recipe
        fileName: sample_recipe.yml
You can run the following to push your local recipe into a configmap
Copy code
kubectl create configmap sample-recipe --from-file=sample_recipe.yml -n <<namespace>>
The recipe should look something like the following (note the sink portion of the recipe)
Copy code
source:
  type: "file"
  config:
    filename: "/datahub-ingestion/examples/mce_files/bootstrap_mce.json"

sink:
  type: "datahub-rest"
  config:
    server: '<http://datahub-datahub-gms:8080>'
a
Ah I see, thanks @early-lamp-41924! Appreciate the help
e
Let us know if it doesnt work!
note that under crons you can put
as many different jobs (one recipe per job) as youlike
sample is just a random name you put to identify the job
a
It worked! Just trying to understand things better: since I used 
sample_recipe.yml
to create the 
sample-recipe
configmap, why do I still need to specify in 
values.yml
both the 
configmapName
and 
fileName
? Can a configmap actually have multiple files in it?
e
Yes!
We have a recipe for bigquery and anowflake in the same configmap
they are keyed by the file name unless you explicitly set the name like --from-file=key=recipe.yaml
a
Got it! thankyou
l
@early-lamp-41924, adding my question in this thread as it's relevant. I am using ingestion-cron to schedule my ingestion job. My confogMap has multiple recipes(yml). I am able to successfully run this cron job for one recipe using following in values.yml
recipe:
configmapName: recipes
filename: hive.yml
I need help with Syntax on how to run multiple yml files. Appreciate any help!
Hi @dazzling-judge-80093 / @incalculable-ocean-74010 / @early-lamp-41924 appreciate any help on the above question
i
Hello, Something like this should suffice. 
crons
is a yaml dictionary
Copy code
datahub-ingestion-cron:
  enabled: true
  crons:
    hive:
      schedule: "0 * * * *" # Every hour
      recipe:
        configmapName: recipe-config
        fileName: hive_recipe.yml
   druid:
      schedule: "0 * * * *" # Every hour
      recipe:
        configmapName: recipe-config
        fileName: druid_recipe.yml
   snowflake:
      schedule: "0 * * * *" # Every hour
      recipe:
        configmapName: recipe-config
        fileName: snowflake_recipe.yml
l
Thanks I will try and share my results!
2 Views
Open in Slack
PreviousNext
Powered by