I am trying to reduce the resource utilization of Airbyte jobs, because they are quite excessive: a job requests more than an entire CPU, but I’ve only got 4 to work with in my environment (Amazon EKS). The resource configuration only applies to a subset of containers in the job pod, so I’m wondering if anyone has any advice on further limiting job resources.

New message text here

“Airbyte on Kubernetes in Production” doesn’t seem to exist, @kapa.ai.

New message text here

If you deployed airbyte with helm, you can set default compute resource for all jobs https://github.com/airbytehq/airbyte-platform/blob/c7fbb072c60f53f6c46228530a6649aeeff81973/charts/airbyte/values.yaml#L108 it is not recommended to set global values though. https://docs.airbyte.com/operator-guides/configuring-connector-resources/

as marie mentions, per-job values are recommended, as each source/destination combination can have significantly different resource requirements, as well as even within these combinations the requirements can be heterogenous - one person’s postgres source might need much more memory than another one’s, for example, due to the nature of the data trying to be moved being different. I actually recommend starting with an instance wide limitation first and selectively increasing on a per connection basis first, then on a per connector basis if you find a common pattern of increasing per connection. The defaults are a little aggressive in terms of what resources they request in order to create a good first experience (increases successful chance of sync).

Thanks for these answers! I’m just now seeing them but will incorporate them into our deployment. For what it’s worth, the latest Airbyte (as of a week ago) has massively improved response times and better resource utilization in general across components. It seemed to be a very good move to set up an API server apart from the server.

Do you mean instance wide vars? Per-connection settings are stored in the database itself

Ok, thanks. I was referring to the environment variables referenced in the documentation.