hi there, ive deployed datahub on ecs having deployed 4 containers datahub-frontend, datahub-gms, datahub-actions and confluent-schema-registry, eveything seems to be working fine having ingested metadata from numerous sources. Im just a bit confused as to the purpose of the datahub-mae-consumer and datahub-mce-consumer components and whether i need to deploy these aswell as everything seems to be working fine as is

Hey Randal, those two containers are used for ingestion and handling certain types of writes in the product. Are you planning to use ingestion to continue adding data to your deployment or will it primarily be used in a read-only way?

currently we ingest data into datahub in two ways, firstly by the ui pointing it to snowflake and looker, secondly we run

datahub ingest -c datahub/recipe.yml

for dbt ingestion, moving forward we plan to use the same mechanisms

Both those ingestions you mentioned will use the consumers!

but we havent deployed either of those containers 🙃

@famous-waitress-64616 do you know about this one? When these would be needed?

I believe those are both embedded within GMS -- they are spring jobs and gms is a spring application. @bitter-author-2524 can you confirm?

^ has been confirmed internally by the team

ok thanks