https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> We’re using Airbyte to run a sync ...
# replication-ask-ai
p
@kapa.ai We’re using Airbyte to run a sync from Postgres to BigQuery. However, when the replication runs for a single table with total size of 1658MB, the replication job ends up consuming around 30 GB of memory. Our setup is running on GKE, using Docker with a Kubernetes deployment. The server specs are 30 GB RAM and 8 vCPUs
k
Airbyte jobs can indeed use very high memory in some situations, especially with database sources that have large rows or very large tables. From the docs, a Sync job’s main memory consumer is the Source worker, which can buffer up to 10,000 records in memory at a time. For tables with large row sizes, this quickly turns into several GB of RAM (e.g., 0.5 MB average row size → ~5 GB RAM just for the in‑memory batch).[[Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#what-to-scale)] When combined with other overhead, this can explain why a ~1.6 GB table might drive memory usage into the tens of GBs. There are also known situations where connectors (e.g., CockroachDB source) or the platform showed suspiciously high or growing memory usage (e.g., 18 GB table causing ~50 GB usage, suspected memory leak).[[Cockroach OOM](https://discuss.airbyte.io/t/1261); [Unusual RAM usage](https://discuss.airbyte.io/t/2167)] Similar behavior has been reported for Postgres→BigQuery syncs where RAM usage keeps increasing across syncs and the worker container in particular grows in memory.[[RAM increasing](https://discuss.airbyte.io/t/1083)] Given your case (Postgres → BigQuery on GKE, 30 GB node, single ~1.6 GB table using ~30 GB RAM), what’s clear from the sources: - This level of memory usage is not expected to be strictly proportional to table size and has been treated as an issue in other threads (suspected leak or connector inefficiency).[[Cockroach OOM](https://discuss.airbyte.io/t/1261)] - Airbyte currently does not stream “table-by-table” in a way that fully releases memory between tables for a single connection; users have observed that memory only drops after all tables are processed.[[Progressive push](https://discuss.airbyte.io/t/1977)] - The recommended levers for memory are: - Increase/decrease connector job resources (e.g., 
JOB_MAIN_CONTAINER_MEMORY_REQUEST
/ 
JOB_MAIN_CONTAINER_MEMORY_LIMIT
when running via Docker/Kubernetes).[[Low memory footprint](https://discuss.airbyte.io/t/1399); [Scaling variables](https://discuss.airbyte.io/t/995); [Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#what-to-scale)] - Apply Docker/Kubernetes resource constraints on worker containers so they cannot exhaust the whole node’s RAM (accepting that overly low limits may cause Java OOM in the job rather than bringing down the node).[[RAM increasing](https://discuss.airbyte.io/t/1083)] However, the knowledge sources do not provide a concrete fix or configuration that guarantees Postgres→BigQuery syncs will cap memory safely at some given number for a specific table size, nor do they confirm a resolved bug that exactly matches your scenario. They mainly: - Acknowledge high/ever-growing RAM has occurred in practice. - Suggest that this may be a bug or memory leak in some connectors. - Recommend opening a GitHub issue with full logs for deeper investigation.[[Unusual RAM usage](https://discuss.airbyte.io/t/2167); [Cockroach OOM](https://discuss.airbyte.io/t/1261)] Given that limitation, the best supported next steps based on the sources are: 1. Constrain job memory at the connector/worker level via the documented resource settings (
JOB_MAIN_CONTAINER_MEMORY_LIMIT
, Kubernetes pod limits) so a single job cannot consume the entire node, accepting that a too-low limit may cause that job to fail with OOM instead of crashing the node.[[Low memory footprint](https://discuss.airbyte.io/t/1399); [RAM increasing](https://discuss.airbyte.io/t/1083)] 2. Check if memory usage is concentrated in the worker / connector pod, as other users have done, to confirm which container is responsible.[[RAM increasing](https://discuss.airbyte.io/t/1083)] 3. If memory is clearly disproportionate (as in your 1.6 GB → 30 GB case), **open a GitHub issue with logs and details**; similar cases have been treated as potential memory leaks and investigated by the team.[[Cockroach OOM](https://discuss.airbyte.io/t/1261); [Unusual RAM usage](https://discuss.airbyte.io/t/2167)] The knowledge sources do not contain more specific tuning advice or a confirmed patch that would directly resolve your exact behavior.
p
@kapa.ai provide me values.yaml Current one
Copy code
global:
  auth:
    enabled: true
  jobs:
    resources:
      limits:
        cpu: "3"
        memory: "6Gi"
k
A message from kapa.ai
A message from kapa.ai
Based on the docs and the related GitHub issues, job‑level resource settings in 
values.yaml
are typically set under 
global.jobs.resources
for the Helm chart, and/or via 
global.env_vars
to map to the 
JOB_MAIN_CONTAINER_*
environment variables that Airbyte uses internally.[[Configuring resources](https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources); [K8s OOM issue](https://github.com/airbytehq/airbyte/issues/48816)] The docs only show partial examples and refer you to the full chart on GitHub for the complete 
values.yaml
structure.[[Values.yaml ref](https://docs.airbyte.com/platform/deploying-airbyte/values)] They do not provide a single canonical full 
values.yaml
block for your exact case (Postgres→BigQuery, 30 GB node), so the following is a pattern derived from the examples, not a guarantee of optimal values. A minimal example extending your current snippet, wiring job resource limits and the corresponding env vars:
Copy code
yaml global:   auth:     enabled: true    # Default job pod resources (all jobs) – Helm chart v1/v2 pattern   jobs:     resources:       requests:         cpu: "1"        # example         memory: "2Gi"   # example       limits:         cpu: "3"         memory: "8Gi"   # example    # Environment variables consumed by Airbyte to set per-job container limits   env_vars:     JOB_MAIN_CONTAINER_CPU_REQUEST: "1"     JOB_MAIN_CONTAINER_CPU_LIMIT: "3"     JOB_MAIN_CONTAINER_MEMORY_REQUEST: "2Gi"     JOB_MAIN_CONTAINER_MEMORY_LIMIT: "8Gi"      # Optional: per‑job‑type overrides shown in the GitHub issue example     CHECK_JOB_MAIN_CONTAINER_CPU_REQUEST: "1"     CHECK_JOB_MAIN_CONTAINER_CPU_LIMIT: "3"     CHECK_JOB_MAIN_CONTAINER_MEMORY_REQUEST: "2Gi"     CHECK_JOB_MAIN_CONTAINER_MEMORY_LIMIT: "8Gi"      DISCOVER_JOB_MAIN_CONTAINER_CPU_REQUEST: "1"     DISCOVER_JOB_MAIN_CONTAINER_CPU_LIMIT: "3"     DISCOVER_JOB_MAIN_CONTAINER_MEMORY_REQUEST: "2Gi"     DISCOVER_JOB_MAIN_CONTAINER_MEMORY_LIMIT: "8Gi"      SPEC_JOB_MAIN_CONTAINER_CPU_REQUEST: "1"     SPEC_JOB_MAIN_CONTAINER_CPU_LIMIT: "3"     SPEC_JOB_MAIN_CONTAINER_MEMORY_REQUEST: "2Gi"     SPEC_JOB_MAIN_CONTAINER_MEMORY_LIMIT: "8Gi"      # Optional sidecar tuning, from the same issue example     SIDECAR_MAIN_CONTAINER_CPU_REQUEST: "250m"     SIDECAR_MAIN_CONTAINER_CPU_LIMIT: "500m"     SIDECAR_MAIN_CONTAINER_MEMORY_REQUEST: "256Mi"     SIDECAR_MAIN_CONTAINER_MEMORY_LIMIT: "512Mi"
This structure matches: - Instance‑wide resource env vars as documented: 
JOB_MAIN_CONTAINER_CPU_REQUEST
, 
JOB_MAIN_CONTAINER_MEMORY_LIMIT
, etc.[[Configuring connector resources](https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources)] - The 
global.env_vars
pattern from the closed Kubernetes OOM issue, where these env vars are set in 
values.yaml
.[[K8s OOM issue](https://github.com/airbytehq/airbyte/issues/48816)] Because the docs explicitly say to “see full example on GitHub” for the chart’s 
values.yaml
, you should cross‑check this structure and adjust the actual numbers (Gi/Mi, m CPUs) to match your workload and node size using the current chart version you deploy.[[Values.yaml ref](https://docs.airbyte.com/platform/deploying-airbyte/values)]
Based on your current snippet and the docs, you should set connector job resources via 
JOB_MAIN_CONTAINER_*
env vars under 
global.env_vars
(Helm chart v1), or via 
global.workloads.resources.mainContainer.*
(Helm chart v2 “Next”).[[Config connector resources](https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources#configuring-instance-wide-requirements); [Upgrade to chart v2](https://docs.airbyte.com/platform/next/deploying-airbyte/chart-v2-community#update-your-valuesyaml-file)] Since your current 
values.yaml
already uses 
global.jobs.resources
, here is a minimal example extending it with the recommended instance‑wide env vars (Helm v1 style):
Copy code
yaml global:   auth:     enabled: true    # Instance-wide job container resources (applied to all jobs, unless overridden)   env_vars:     JOB_MAIN_CONTAINER_CPU_REQUEST: "3"     JOB_MAIN_CONTAINER_CPU_LIMIT: "3"     JOB_MAIN_CONTAINER_MEMORY_REQUEST: "6Gi"     JOB_MAIN_CONTAINER_MEMORY_LIMIT: "6Gi"    # (Optional legacy block, often present in chart v1 – does NOT directly control pod limits   # unless wired to env vars as above)   jobs:     resources:       requests:         cpu: "3"         memory: "6Gi"       limits:         cpu: "3"         memory: "6Gi"
If you are on Helm chart v2 (Next), the documented mapping is instead:
Copy code
yaml global:   auth:     enabled: true   workloads:     resources:       mainContainer:         cpu:           request: "3"           limit: "3"         memory:           request: "6Gi"           limit: "6Gi"
In both cases you can then use connector‑/connection‑specific overrides via the 
resource_requirements
mechanism if needed.[[Config connector resources](https://docs.airbyte.com/platform/operator-guides/configuring-connector-resources#configuring-connector-definition-requirements)]
p
@kapa.ai it's helm chart v2
k
A message from kapa.ai
For Helm chart v2, the instance‑wide job container resources are no longer configured via 
global.jobs.resources
but via 
global.workloads.resources.mainContainer.*
, which is also the documented replacement for the 
JOB_MAIN_CONTAINER_*
env vars.[\[chart v2 mapping\]](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#update-your-valuesyaml-file) So, starting from your current snippet:
Copy code
yaml global:   auth:     enabled: true   jobs:     resources:       limits:         cpu: "3"         memory: "6Gi"
you should instead use:
Copy code
yaml global:   auth:     enabled: true    # Chart v2: instance‑wide job main container resources   workloads:     resources:       mainContainer:         cpu:           request: "3"           limit: "3"         memory:           request: "6Gi"           limit: "6Gi"
The docs show that: - 
JOB_MAIN_CONTAINER_CPU_REQUEST
→ 
global.workloads.resources.mainContainer.cpu.request
- 
JOB_MAIN_CONTAINER_CPU_LIMIT
→ 
global.workloads.resources.mainContainer.cpu.limit
- 
JOB_MAIN_CONTAINER_MEMORY_REQUEST
→ 
global.workloads.resources.mainContainer.memory.request
- 
JOB_MAIN_CONTAINER_MEMORY_LIMIT
→ 
global.workloads.resources.mainContainer.memory.limit
[\[chart v2 mapping\]](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#update-your-valuesyaml-file) 
global.jobs.resources
is still present in the chart values file but is effectively legacy; an open issue notes that 
global.jobs.resources
is not used in v2 and the configuration lives under 
global.workloads.resources.*
instead.[\[helm v2 jobs.resources issue\]](https://github.com/airbytehq/airbyte/issues/65515)
9 Views
Open in Slack
PreviousNext
Powered by