Hi Team, We are facing Kakfa clients re-balancing issue, At this point of time our UI is also freezed as the consumer is going in a re-balancing loop and its not consuming offsets and offset lag keeps on Increasing as Ingestion is pulling more info. Upon debugging we found that the datahub is using MetadataChangeLog_Versioned_v1 topic for all the changes made to Metadata Graph using UI and also while using kafka sink for Ingestions. So for this reason our UI is in freezed state till the consumer (

generic-mae-consumer-job-client

) reads all the partitions from the topic as the change made to UI is also some where in the queue in the kafka topic. 1. Can we use seperate topic for all the changes we made using UI so that our UI be free from freeezing issue? 2. Also how can we let ourselves come out from the re-balancing of groups issue and speed up our ingestion, As kafka is Asynchronous . MCE consumers are slow in reading the offsets. we are yet to create standalone MCE and MAE Consumers. Hope it increase the speed of the Ingestion but yet to find solution for re-balancing issue.

I’m not sure about this one. @incalculable-ocean-74010 do you have any suggestions here?

Move to standalone mode, 1 mae, 1 mce replica with async ingestion enabled