Hi team, I'm a data engineer in Samsung, and we're considering to use DataHub as a engine of our company's data catalog system. We are expecting that huge number of

Metadata Change Log

events will be generated, so we need to scale-out

mae-consumer

server. Currently,

mae-consumer

is deployed as a

Deployment

in k8s, so physically scaling out is not a problem, but I'm curious about whether the event processing order will be the problem or not. For example, let's assume

MCL1

is generated first, and

MCL2

is generated after it and

mae-consumer-1

pod,

mae-consumer-2

pod is deployed.

mae-consumer-1

pod recieves

MCL1

,

mae-consumer-2

pod receives

MCL2

. For some reason,

mae-consumer-2

pod processed

MCL2

first, so some action is done to ElasticSearch, Neo4j. Next

mae-consumer-1

pod processed

MCL1

after, so another action is done to ES, Neo4j too. In this situation, I expect that there can be some problems... So, will there be some problems because of the processing order in multiple

mae-consumer

? Will our team need to develop some functionalities for them?