Team… `Issue`: We have been seeing some scalability issues in MAE. MCE is able to process events without lag MAE has been much slower `Inference`: We found that bottle neck is in gms spending 50% of its time in mysql. `Actions taken`: These were some of the steps taken to mitigate the issue…. • Increase partitions in MAE topics to 16X • Increase MAE instances and GMS instances proportional to the topic partitions • Increase the

EBEAN_MAX_CONNECTIONS

in GMS to 150 to increase the mysql threads. All these steps reduced the lag on the MAE topic little bit but still not satisfactorily. Are there any other tweaks or suggestions that you guys would suggest?

If I understand the question correctly, you're saying that MAE queue is building up because the processor can't keep up? If so, it should have nothing to do with GMS or MySQL but more likely to be ES or Neo4j?

Next step would be figuring out whether ES or Neo4j is the bottleneck

Thanks guys…My suspect is on Neo4j..I will look at that first. Appreciate your advice!

For ES we have bulk processor enabled; Neo4j pipeline is not performing well, we are looking for improvement such as Neo4j Streams Kafka Integration. How much data do you have for graph?

@cool-river-24902 The issue was in Neo4j. Exposing Neo4j Driver configs was useful so pushed a PR - https://github.com/linkedin/datahub/pull/1998 One of our producers started pushing all their datasets every 5 mins though it was a mistake we saw that MAE was clogging quite a bit. Regarding the volume it was close to 2000 datasets in MAE every 5 min. As you see it was close to a million datasets per day. Though the producer quickly remediated it definitely uncovered the perf issue in MAE -> Neo4j connectivity. So, are you exploring the Kafka Connect Sink from Kafka -> Neo4j ? How will that be invoked from MAE?

@nutritious-bird-77396 Glad that you found the root cause : ) We haven't got our hands dirty on this yet, probably will try Neo4j Streams plugin or Kafka Connect plugin as mentioned in the integration guidelines.

@cool-river-24902 is there some secret sauce in those plugins in how they write to Neo4j? That would be surprising.

@mammoth-bear-12532 My understanding is it essentially run

UNWIND

cypher command to handle batch events. We could have our own batch write logic too but may not want to invent the wheel.