Hello everybody, can you tell me please what is the main reason that DataHub uses 2 kafka streams in back groud one for ingestion another one for elastic and graph ? If it takes info from mysql which was generated by another information from another kafka stream, why it is not consuming directly from first kafka topics?

Which 2 streams are you referring to here? Did this question arise due to some doc being unclear? If yes, can you point me towards that doc? Is this causing some concern in your deployment?

@square-activity-64562 so from docs I can see that proposal stream (MCE) gets info from metadata ingestion and applier (mce-consumer) sends this information to mySql after with GMS commit stream (MAE) gets info from mySQL and with mae-consumer sends it to elastic and neo4j. My question is about MCE and MAE

MAE is supposed to be deprecated. MCE is supposed to replace it. The migration was started but never completed.

@square-activity-64562 so this means that MCE consumer sends info also to elastic and neo4j ? Can I somehow have updated docs?

This is a gap in docs. This doc update is on the backlog alongwith migration. Please feel free to raise a github issue https://github.com/datahub-project/datahub/issues with context on how this will help you out.

We are on a stage to understand which technology use for each process but before we need to understand how all connected to each other, for security setup, devops setups and also to understand if we can easily go with Confluent Cloud or we need to setup MSK also

You can use confluent cloud or MSK. In case of MSK you can use confluent-schema-registry as a container

Just to sum-up, current version of DataHub uses also MAE architecture but in the future it will deprecated and replaced by MCE yes?

yes that is the current plan

Thanks 🙏