Hi Everyone, I'm running Airbyte on an EKS cluster. I am currently testing size, scope, and scale of the cluster, so I'm purposefully trying to blast it and identify how many concurrent sync jobs I can start before overloading the server. However this has lead our product team to pose an interesting question. When I overload the cluster, even before failure, we start to see performance of the web app be impacted. Is there any way we can separate the blast radius of our sync jobs, and the performance of the web app? ie. Can I push our daily sync jobs to the limit, but still be confident that our data science team can interact with, and use the interface for adhoc/one-off work without being impacted from a performance standpoint? Thanks for all your help in advance! Really love the service, Airbyte is great! P.S. At a higher level, do you have any suggestions/templates/guidance for EKS Cluster Node Size, Amount, Configuration, etc.?

Hey there, thanks so much for the compliments! We don't have anything specific for EKS but we do have the Scaling Airbyte doc which is usually helpful. If you haven't seen, there is also detailed info on k8s performance in this doc including some on job parallelism: https://docs.airbyte.com/deploying-airbyte/on-kubernetes#deploy-airbyte-on-kubernetes-in-production Beyond that, you might be interested in implementing metrics collection if that hasn't already been set up: https://docs.airbyte.com/operator-guides/collecting-metrics

This was perfect, thank you so much! We have addressed all of our challenges and are running magnificently.