A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hi there, I'm going through the EKS/AWS setup steps and I'm not seeing the mce/mae-consumer pods coming up- I'm probably missing a configuration step here as this is all new territory for my team.

Hi. We recently merged GMS with the consumer jobs by default, so you don’t have to manage as many pods. If you do want the consumer jobs separate for scaling purposes, you can set this field to true <https://github.com/linkedin/datahub/blob/master/datahub-kubernetes/datahub/values.yaml#L65> in your values.yaml file

Ok thanks- just trying to work out where to point my kafka ingest configs to.

Yes! So now you can just set it in the gms. If it is ssl settings, we already have a global config for setting it! <https://github.com/linkedin/datahub/blob/master/datahub-kubernetes/datahub/values.yaml#L115>

*Description*

I'm working through upgrading from 0.8.1 to 0.8.8, using the `helm` chart at <https://github.com/acryldata/datahub-helm>.

In my current deployment, I'm using kafka-based metadata ingestion. Containers running `datahub ingest` have recipes that include configuration like this:

```sink:
  type: "datahub-kafka"
  config:```
I saw that in the chart for 0.8.8, the standalone MAE and MCE consumers are now NOT deployed by default.

<https://github.com/acryldata/datahub-helm/blob/648c3b6f44287db0efacf8ff14e044dc9bec88ae/charts/datahub/values.yaml#L63>

i.e. will I need to change my configs for `datahub ingest` to something like

```sink:
  type: "datahub-rest"
  config:```


By the way, <@U01NN9CTQ01> now in GitHub if you hit `y` on the keyboard, it will change line-specific links to be specific to the most recent commit to touch the file you're looking at.

(mentioning it since the links at the top of this thread now point to files that no longer exist)

Awesome! datahub_standalone_consumers_enabled just decides where to run the consumers in the backend! Ingestion should be oblivious to this change, so no need to change the ingestion scripts

kafka and rest should both work correctly

ooo interesting, ok so if `datahub_standalone_consumers_enabled: false`, that means that the consumers will run as which of the following?

• separate containers within the GMS pod
• separate processes within the main GMS container
• separate threads within the GMS process within the main GMS container
• something else
I've been trying to figure this out myself from source code and my best guess is "separate processes within the main GMS container", but I'm not sure.

the second. separate processes within the main gms container

We wanted to simplify the docker deploy, since before it required around 10 separate containers!

got it ok, makes sense! Thanks very much for the help

minor correction: it's separate threads within the main GMS deployable