Hey <@U04S8PLJWMT>, I have setup the Airbyte usin...
# replication-ask-ai
h
Hey @kapa.ai, I have setup the Airbyte using abctl in my VM (with 4 CPU and 32 GB Memory) I am trying to sync from MySQL to PostgreSQL DB. But when we try to sync a large table, Airbyte caches a large number of records from the source connection. Because of that the replication-orchestrator pod kubernetes experiences excessive memory consumption during syncs. We can clearly see it on Pod status:
Copy code
airbyte-abctl replication-job-662-attempt-0 0/3 Error 0 4h24m
airbyte-abctl replication-job-690-attempt-0 2/3 OOMKilled 0 9m15s
As I installed Airbyte using abctl, but I didn’t set any resource limits during the installation. Now I’d like to adjust the memory allocation and apply new resource limits. Is there a way to update these settings after installation, or do I need to reinstall Airbyte to change them?
k
You don’t need to reinstall Airbyte. You can change resource limits after installation and have new syncs pick them up. ### 1. Why you see OOM on large tables For sync jobs, the source worker keeps up to 10,000 records in memory. With large row sizes this can easily translate into multiple GB of RAM needed for a single sync. [[What to scale](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#memory)] This is usually what drives OOMs on big tables, even if the node itself has a lot of RAM. ### 2. Instance‑wide job resource limits (Helm / abctl) When deployed on Kubernetes (which
abctl
uses), connector job pods (including orchestrator/read/write) take their default CPU/memory requests/limits from global job settings in the Helm values, e.g.: [[Scaling Airbyte Enterprise](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#connector-cpu--memory-settings); [Configuring connector resources](https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources#configuring-instance-wide-requirements)]
Copy code
yaml global:   jobs:     resources:       limits:         cpu: "2"      # example         memory: "4Gi" # example       requests:         cpu: "1"         memory: "2Gi"
or via env vars (depending on chart / version): -
JOB_MAIN_CONTAINER_CPU_REQUEST
-
JOB_MAIN_CONTAINER_CPU_LIMIT
-
JOB_MAIN_CONTAINER_MEMORY_REQUEST
-
JOB_MAIN_CONTAINER_MEMORY_LIMIT
[[Configuring connector resources](https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources#configuring-instance-wide-requirements); [Forum orchestrator resources](https://discuss.airbyte.io/t/3775)] These apply to orchestrator/read/write job containers. How to apply after install (conceptually): 1. Edit the Helm values used by
abctl
(usually a
values.yaml
in your deployment folder) to add/update the
global.jobs.resources
section or the
JOB_MAIN_CONTAINER_*
envs. 2. Re‑apply the deployment (
helm upgrade
or
abctl
equivalent) so the worker pods are recreated with the new settings. 3. New replication job pods created after that will have the updated limits; you don’t need to reinstall. The GitHub issue you shared confirms that updating
global.jobs.resources
and recreating the worker pod causes new replication-job pods to use the new values. [[Workload resources issue](https://github.com/airbytehq/airbyte/issues/68162)] ### 3. Per‑connector / per‑connection tuning (optional, more targeted) If only this MySQL→Postgres sync is problematic, you can increase resources just for that connector or connection via the database, without reinstalling: 1. Find the connector definition ID:
Copy code
sql select * from actor_definition where actor_definition.docker_repository like '%<image-name>';
2. Set higher resources for sync jobs for that definition:
Copy code
sql update actor_definition set resource_requirements = '{"jobSpecific": [{   "jobType": "sync",   "resourceRequirements": {     "cpu_limit": "2",     "cpu_request": "2",     "memory_limit": "4096Mi",     "memory_request": "4096Mi"   } }]}' where id = '<id-from-step-1>';
This overrides instance‑wide defaults for all connectors of that type. [[Configuring connector resources](https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources#configuring-connector-definition-requirements)] There is also a higher‑precedence “connection-specific” level you can use in the same way if needed. [[Configuring connector resources](https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources)] --- Summary: You can safely adjust memory/CPU limits after installation by editing the Helm/`values.yaml` (or
JOB_MAIN_CONTAINER_*
envs) and redeploying, and optionally fine‑tune resource requirements per connector/connection in the Airbyte DB. No full reinstall is required.