Hi All, im new to this slack group and DataHub, was looking for best practices in Scaling the MAC and MAE in a K8s cluster

Hello Lorenzo 👋 Could you explain a little more what you mean with scaling MAEs? Are you hitting a bottleneck? Also what do you mean with MAC?

hi Pedro, sorry i meant MCE, we arent hitting bottle necks yet but im not sure what the architecture is supposed to look like and if its meant to autoscale or not. Currently the deployment has no HPAs in our K8s cluster, not sure if its intentionally or something we are missing

So Datahub does not yet offer HPA (horizontal pod autoscaling), it is something we are likely to look into in the future but not on the immediate roadmap. I can tell you however from benchmarks that datahub’s components are not the bottleneck in the overall system. That would be the search database (elasticsearch, neo4j or a managed option) so if you can scale that well, datahub as is should work quite well.

ah thank you so much, Pedro! Very informative