Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

How do I change the resources limit/requst for the `read` and `writer` pods airbyte creates in k8s?

Can I add a

```apiVersion: v1
kind: ResourceQuota
metadata:
  name: mem-cpu-demo
  namespace: airbyte
spec:
  hard:
    requests.cpu: "2"
    requests.memory: 2Gi
    limits.cpu: "2"
    limits.memory: 2Gi```
To my cluster?

Don’t think that solves my question though.

Feels like there should be some configuration via your env for this?

is it these?

```# Docker Resource Limits
JOB_MAIN_CONTAINER_CPU_REQUEST=
JOB_MAIN_CONTAINER_CPU_LIMIT=
JOB_MAIN_CONTAINER_MEMORY_REQUEST=
JOB_MAIN_CONTAINER_MEMORY_LIMIT=```

<https://docs.airbyte.com/operator-guides/configuring-airbyte/#jobs>

I was configuring:

```  JOB_MAIN_CONTAINER_CPU_REQUEST: "1"
  JOB_MAIN_CONTAINER_CPU_LIMIT: "1"
  JOB_MAIN_CONTAINER_MEMORY_REQUEST: "1Gi"
  JOB_MAIN_CONTAINER_MEMORY_LIMIT: "1Gi"```
But the `check/read/writer` pods still have:

```    Limits:
      cpu:     200m
      memory:  50Mi
    Requests:
      cpu:        100m
      memory:     25Mi```

I also tried: `CONNECTOR_SPECIFIC_RESOURCE_DEFAULTS_ENABLED`

But same thing there.

<@U03GB8G9WTH> there's a separate setting per connection. I don't think it's exposed via the UI, but can be set via the API. <https://airbyte-public-api-docs.s3.us-east-2.amazonaws.com/rapidoc-api-docs.html#post-/v1/connections/update>

see `resourceRequirements` in the connection object

image.png

Hello.

I managed to get it working now.

I also could verify that I had to delete the server and worker pod, it was not enough for me to just update the config and re-apply the deployments. It was not picking up the changes, with the API it does instantly.

I also made one mistake by looking at the init containers resources when I was investigating, to add on, to the confusion.

Thanks for the help

<@U03GB8G9WTH> were you able to set resources for check/read/writer pods? I’m struggling to figure this out now and could use some pointers…and yes I realize it’s been a year :slightly_smiling_face:

Yes but the issue is that it is the same for both checker and read write, which don’t make much sense to me.

```JOB_MAIN_CONTAINER_CPU_REQUEST
JOB_MAIN_CONTAINER_CPU_LIMIT
JOB_MAIN_CONTAINER_MEMORY_REQUEST
JOB_MAIN_CONTAINER_MEMORY_LIMIT```
It is also not well documented from Airbyte.