https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT>, my cluster autoscaller always star...
# replication-ask-ai
d
@kapa.ai, my cluster autoscaller always starts new nodes in the node group for each replication job. Here are my values.yaml
Copy code
jobs:
    resources:
      requests:
        cpu: 200m
        memory: 256Mi
      limits:
        cpu: 400m
        memory: 4Gi
    kube:
      nodeSelector:
        airbyte_node_type: worker
      namespace: airbyte
the instance type in my worker node group is m7i.2xlarge (8 vCPUs and 32 GiB memory, network up to 12.5 Gigabit)
k
Your 
values.yaml
looks reasonable, but there are a few known issues and nuances with how Airbyte applies resource settings on Kubernetes that can explain why the autoscaler keeps adding nodes. From the knowledge sources: 1. Global job resources sometimes aren’t respected by job pods There are multiple open issues where 
global.jobs.resources
(and even 
global.workloads.resources
) are ignored or only partially applied by the workload-launcher / orchestrator, causing pods to request much larger defaults than expected. [jobs.resources broken; resource limits ignored] In those cases, users see: ◦ Replication pods with hardcoded defaults like 
cpu=2
, 
memory=2Gi
or more, regardless of 
values.yaml
. ◦ Scheduler errors like “Insufficient cpu” and aggressive node scaling. [resource limits ignored] 1. Per-connector / per-connection overrides can trump your global settings The workload-launcher can pull resource requirements from the database (actor definitions or connection 
resource_requirements
), which override the global 
jobs
/ 
workloads
settings—even when 
useConnectorResourceDefaults: false
. [workload resources not applied; orchestrator ignores limits] If any of your connectors or connections have higher CPU/memory configured in the DB, the resulting pod requests may be much larger than the 
200m/256Mi
you expect, forcing the autoscaler to add nodes. 2. Millicore parsing bugs / integer-only behavior There are reports that some resource env vars are only parsed correctly when given as whole numbers (e.g. 
"1"
instead of 
"500m"
), and that invalid parsing causes a fallback to large defaults (e.g. 
cpu=2
). [workload resources not applied; jobs not removed / whole numbers] So even if your Helm values are correct, the effective CPU request seen by the scheduler might be higher than 200m. 3. Docs vs. behavior The docs show the pattern you’re using (
global.jobs.resources
) as the right way to set instance‑wide defaults. [Scaling Airbyte; Configuring connector resources] But the open issues above indicate that, in some versions, these values don’t reliably propagate to replication pods. Given this, the autoscaler behavior you’re seeing is likely because the actual pod resource requests are larger than your 
values.yaml
suggests. To confirm and narrow it down, you can: 1. *Inspect a replication pod’s actual requests/limits*kubectl -n airbyte get pod replication-job-pod -o yaml | grep -A5 "resources:"If you see 
cpu: "2"
or similar instead of 
200m
, you’re hitting the same issues as in the linked GitHub reports. [resource limits ignored] 2. Check for DB-level overrides ◦ Look for non‑empty 
resource_requirements
on your connections and/or actor definitions as described in the docs and issues. [Configuring connector resources; workload resources not applied] ◦ If they’re set to high values, they will override your global 
jobs.resources
and drive up pod requests. 1. (If applicable) Avoid millicores in some env‑var based settings Where you’re using env vars (e.g. 
JOB_MAIN_CONTAINER_CPU_*
, 
REPLICATION_ORCHESTRATOR_CPU_*
), prefer whole numbers (
"1"
, 
"2"
) instead of 
"500m"
until the parsing bugs are fixed. [workload resources not applied; jobs not removed / whole numbers] Because the knowledge sources don’t include a definitive fix for your exact version, I can’t point to a specific version upgrade or flag that will fully resolve this. The most reliable next step is to: • Compare what you configured vs what the pods actually request, and • Remove or lower any DB‑level 
resource_requirements
that are higher than you intend. If you can share the 
resources
section from a 
replication-job-*
pod spec, I can help interpret whether it matches these known issues. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
d
@kapa.ai $ kubectl get pod replication-job-78-attempt-0 -o yaml | grep -A5 "resources:" resources: limits: cpu: "2" memory: 2Gi requests: cpu: "2" -- resources: limits: cpu: "2" memory: 2Gi requests: cpu: "1" -- resources: limits: cpu: "2" memory: 1Gi requests: cpu: "1" -- resources: limits: cpu: "2" memory: 2Gi requests: cpu: "2" -- resources: limits: cpu: "2" memory: 1Gi requests: cpu: "1" -- resources: limits: cpu: "2" memory: 2Gi requests: cpu: "2" -- resources: limits: cpu: "2" memory: 2Gi requests: cpu: "1" -- resources: limits: cpu: "2" memory: 2Gi requests: cpu: "2" The below log I can see in the airbyte-workload-launcher pod: 2026-06-08 060035,774 [default-15] INFO i.a.w.l.p.KubePodClient(launchReplication):100 - [source] image: airbyte/source-bigquery:0.4.5 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=2}, requests={memory=1Gi, cpu=1}, additionalProperties={})
k
Your pod spec and the workload-launcher log confirm you’re hitting a known bug: replication job resources are not honoring 
values.yaml
and instead default to hard‑coded values around 
cpu=2
, 
memory=1–2Gi
. [resource limits ignored; workload resources not applied] That explains why your autoscaler keeps adding nodes: each replication pod is effectively requesting ~2 vCPU per container, not the 
200m
you configured. From the issues: • Users see exactly what you’re seeing: replication pods with 
limits={memory=2Gi, cpu=2}, requests={memory=1–2Gi, cpu=1–2}
regardless of 
global.jobs.resources
or 
global.workloads.resources
. [resource limits ignored] • Engineering has confirmed this is a bug in how the workload-launcher resolves and applies resources; Helm values and even env vars like 
JOB_MAIN_CONTAINER_*
/ 
REPLICATION_ORCHESTRATOR_*
are ignored or overridden. [resource limits ignored; orchestrator ignores limits; REPLICATION_ORCHESTRATOR_* do not work] Known workarounds from the knowledge sources: 1. Set env vars directly on 
workload-launcher
 Example from the issue (adjust values as needed): [resource limits ignored]workloadLauncher: extraEnv: - name: JOB_MAIN_CONTAINER_CPU_LIMIT value: "0.4" - name: JOB_MAIN_CONTAINER_CPU_REQUEST value: "0.2" - name: JOB_MAIN_CONTAINER_MEMORY_LIMIT value: "4Gi" - name: JOB_MAIN_CONTAINER_MEMORY_REQUEST value: "256Mi"Then restart the 
workload-launcher
and verify inside the pod:kubectl exec -n airbyte deploy/airbyte-workload-launcher -- env | grep JOB_MAIN 2. Connection‑level 
resource_requirements
in the DB This reliably controls source/destination containers (not the orchestrator) and has worked for others when global settings were ignored: [workload resources not applied; orchestrator ignores limits]UPDATE connection SET resource_requirements = jsonb_build_object( 'cpu_request', '0.2', 'cpu_limit', '0.4', 'memory_request','256Mi', 'memory_limit', '4Gi' ) WHERE id = 'your-connection-id'; 3. Kubernetes mutating webhook Some users implemented a mutating admission webhook (or Kyverno policy) that patches the orchestrator container resources on every replication pod to the desired values (e.g. 4Gi instead of 1Gi). [orchestrator ignores limits] Given the knowledge sources, there is no documented way today to make 
global.jobs.resources
alone reliably control these replication pods in the affected versions. To reduce autoscaler churn right now, you’ll need to use one of the workarounds above (env vars on 
workload-launcher
, DB 
resource_requirements
, or a mutating webhook) to bring the actual pod requests down to what your nodes can comfortably run. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
d
@kapa.ai, I get
Copy code
rror: UPGRADE FAILED: server-side apply failed for object airbyte/airbyte-workload-launcher apps/v1, Kind=Deployment: failed to create typed patch object (airbyte/airbyte-workload-launcher; apps/v1, Kind=Deployment): errors:
  .spec.template.spec.containers[name="airbyte-workload-launcher-container"].env: duplicate entries for key [name="JOB_MAIN_CONTAINER_CPU_LIMIT"]
  .spec.template.spec.containers[name="airbyte-workload-launcher-container"].env: duplicate entries for key [name="JOB_MAIN_CONTAINER_CPU_REQUEST"]
  .spec.template.spec.containers[name="airbyte-workload-launcher-container"].env: duplicate entries for key [name="JOB_MAIN_CONTAINER_MEMORY_LIMIT"]
  .spec.template.spec.containers[name="airbyte-workload-launcher-container"].env: duplicate entries for key [name="JOB_MAIN_CONTAINER_MEMORY_REQUEST"]
after I have added these values
k
You’re running into a known class of problems: duplicate env vars in Helm‑rendered manifests cause server‑side apply to fail. The error you see is exactly what happens when the same env var name is defined twice for a container. [duplicate AB_JWT var] In your case, 
JOB_MAIN_CONTAINER_*
is already being set somewhere (likely via the chart’s existing templates or a ConfigMap), and adding them again under 
workloadLauncher.extraEnv
creates duplicates:
Copy code
...containers[name="airbyte-workload-launcher-container"].env:
  duplicate entries for key [name="JOB_MAIN_CONTAINER_CPU_LIMIT"]
  ...
Because the knowledge sources don’t include the exact v2 chart snippet for 
workload-launcher
envs, I can’t point to the precise line, but the pattern is the same as the 
AB_JWT_SIGNATURE_SECRET
duplication that broke upgrades until it was fixed. [duplicate AB_JWT var] Given that, you have two options that avoid the duplicate‑key error: 1. *Do not set 
JOB_MAIN_CONTAINER_*
via 
extraEnv
if the chart already sets them*Instead, configure them through the chart’s supported values so they are rendered only once. For chart v2, the docs map these env vars to Helm values: [chart v2 env mapping]global: workloads: resources: mainContainer: cpu: request: "0.2" limit: "0.4" memory: request: 256Mi limit: 4GiThis sets the equivalents of:without adding a second copy via 
extraEnv
. ◦ 
JOB_MAIN_CONTAINER_CPU_REQUEST
◦ 
JOB_MAIN_CONTAINER_CPU_LIMIT
◦ 
JOB_MAIN_CONTAINER_MEMORY_REQUEST
◦ 
JOB_MAIN_CONTAINER_MEMORY_LIMIT
1. *If you must use 
extraEnv
, remove or override the existing definitions*Since server‑side apply rejects duplicate keys, you’d need to ensure the base chart does not emit those env vars. The knowledge sources don’t document a supported flag to disable them, so I can’t give a concrete value to turn them off. In similar issues, users either: ◦ Patched the chart templates in a parent chart to avoid generating the duplicate env entries, or ◦ Waited for a chart release that removed the duplicate. [duplicate AB_JWT var; duplicate ConfigMap keys] Because of that, the safest path based on the docs is: • Remove the 
JOB_MAIN_CONTAINER_*
entries you added under 
workloadLauncher.extraEnv
. • Set the corresponding 
global.workloads.resources.mainContainer.*
values instead (as above). • Re‑deploy and then check the 
airbyte-workload-launcher
pod’s env to confirm the values:kubectl -n airbyte exec deploy/airbyte-workload-launcher -- env | grep JOB_MAIN_CONTAINER If you share the 
workloadLauncher
section of your 
values.yaml
, I can help you rewrite it so it doesn’t conflict with the chart’s existing env definitions. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
This error matches a general class of Helm/Kubernetes issues where the rendered Deployment ends up with the same env var defined twice, which server‑side apply rejects as “duplicate entries for key …”. There are several similar reports for other env vars and components (e.g. 
STATSD_HOST
, 
AB_JWT_SIGNATURE_SECRET
) that were fixed by removing duplicate definitions in the chart or overriding templates. [duplicate env key; duplicate AB_JWT_SIGNATURE_SECRET; duplicate ConfigMap keys v2] In your case, the duplicates are for: • 
JOB_MAIN_CONTAINER_CPU_LIMIT
• 
JOB_MAIN_CONTAINER_CPU_REQUEST
• 
JOB_MAIN_CONTAINER_MEMORY_LIMIT
• 
JOB_MAIN_CONTAINER_MEMORY_REQUEST
Given the knowledge sources, the most likely cause is: • These env vars are already being set by the chart (e.g. from a ConfigMap or default 
extraEnv
), and • You added them again under 
workloadLauncher.extraEnv
, so the rendered Deployment now has two entries with the same 
name
for each. The knowledge sources don’t show the exact v2 chart template for 
workload-launcher
, so I can’t point to the specific line, but the pattern is the same as the other duplicate‑env issues: the fix is to ensure each env var is defined only once. [duplicate env key; duplicate ConfigMap keys v2] To move forward, based on what is documented: 1. Remove the direct 
extraEnv
entries for those four vars and re‑run 
helm upgrade
to confirm it succeeds. 2. If you still need to override them: ◦ Check how they are currently injected (likely via a ConfigMap like 
airbyte-env
or 
airbyte-abctl-airbyte-env
in abctl setups). [resource limits ignored; Helm v2 env mapping] ◦ Prefer using the Helm v2 equivalents instead of raw env vars where possible. For example, the docs map: ▪︎ 
JOB_MAIN_CONTAINER_CPU_REQUEST
→ 
global.workloads.resources.mainContainer.cpu.request
▪︎ 
JOB_MAIN_CONTAINER_CPU_LIMIT
→ 
global.workloads.resources.mainContainer.cpu.limit
▪︎ 
JOB_MAIN_CONTAINER_MEMORY_REQUEST
→ 
global.workloads.resources.mainContainer.memory.request
▪︎ 
JOB_MAIN_CONTAINER_MEMORY_LIMIT
→ 
global.workloads.resources.mainContainer.memory.limit
[env → v2 mapping] ◦ This avoids redefining the same env vars in 
extraEnv
and prevents duplicates. Because the knowledge sources don’t include your exact chart version or full rendered manifest, I can’t give a precise one‑line patch, but the key is: don’t set those four env vars twice. Use either the chart’s built‑in values (the 
global.workloads.resources.mainContainer.*
keys) or a single 
extraEnv
definition path, not both. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
d
@kapa.ai, once again after running 4 connectors 4 nodes were launched. I can confirm that the requests limits are in the deployment:
Copy code
ubectl exec -n airbyte deploy/airbyte-workload-launcher -- env | grep JOB_MAIN
DISCOVER_JOB_MAIN_CONTAINER_CPU_REQUEST=
CHECK_JOB_MAIN_CONTAINER_MEMORY_REQUEST=
JOB_MAIN_CONTAINER_MEMORY_REQUEST=256Mi
DISCOVER_JOB_MAIN_CONTAINER_MEMORY_LIMIT=
CHECK_JOB_MAIN_CONTAINER_MEMORY_LIMIT=
JOB_MAIN_CONTAINER_CPU_LIMIT=0.4
DISCOVER_JOB_MAIN_CONTAINER_CPU_LIMIT=
CHECK_JOB_MAIN_CONTAINER_CPU_REQUEST=
JOB_MAIN_CONTAINER_MEMORY_LIMIT=4Gi
DISCOVER_JOB_MAIN_CONTAINER_MEMORY_REQUEST=
CHECK_JOB_MAIN_CONTAINER_CPU_LIMIT=
JOB_MAIN_CONTAINER_CPU_REQUEST=0.2
k
The knowledge sources confirm that what you’re seeing (each replication job effectively “needing its own node” despite low 
JOB_MAIN_*
values) matches known bugs in how Airbyte 2.x applies resources, not a misconfiguration on your side. Key points from the sources: 1. *`JOB_MAIN_*` env vars are set correctly, but replication pods still use ~2 CPU* Your 
workload-launcher
env shows:JOB_MAIN_CONTAINER_CPU_REQUEST=0.2 JOB_MAIN_CONTAINER_CPU_LIMIT=0.4 JOB_MAIN_CONTAINER_MEMORY_REQUEST=256Mi JOB_MAIN_CONTAINER_MEMORY_LIMIT=4GiYet the replication pods are still created withlimits cpu: "2", memory: 2Gi requests: cpu: "1" or "2", memory: 1–2GiThis is exactly what’s reported in multiple issues: the workload-launcher ignores Helm/env resource settings and falls back to internal defaults (2 CPU / 1–2Gi) for replication jobs. [resource limits ignored; workload resources not applied; orchestrator ignores limits]With 4 replication jobs each requesting ~2 vCPU per container, the cluster autoscaler is behaving as expected by adding nodes. 2. *This is acknowledged as a platform bug, not just Helm config*So even though your 
JOB_MAIN_*
env vars are present, the current workload-launcher logic is not reliably using them for replication jobs. ◦ In 73346, engineering confirms a “genuine bug in how resource configurations are propagated to replication pods”; Helm values and even 
airbyte-env
ConfigMap changes don’t affect the final pod resources. [resource limits ignored] ◦ In 68162 and 72833, users see the same pattern: 
REPLICATION_ORCHESTRATOR_*
/ 
workloads.resources.*
/ 
jobs.resources
are set, but replication/orchestrator containers still run with hardcoded CPU/memory. [workload resources not applied; orchestrator ignores limits] 1. *Workarounds others are using (none are perfect, but they’re the only documented options)*Based on the issues, the only ways people have successfully reduced replication pod resources are:*a. Connection‑level 
resource_requirements
in the DB* This does affect source/destination containers and has been confirmed to override the defaults when global settings are ignored. [workload resources not applied; orchestrator ignores limits]Example (adjust to your target 0.2/0.4 CPU, 256Mi/4Gi):UPDATE connection SET resource_requirements = jsonb_build_object( 'cpu_request', '0.2', 'cpu_limit', '0.4', 'memory_request','256Mi', 'memory_limit', '4Gi' ) WHERE id = 'your-connection-id';This won’t fix the orchestrator container, but it will at least shrink the source/destination containers so each job uses less total CPU, which should reduce autoscaler pressure.b. Mutating admission webhook / Kyverno policy Several users implemented a Kubernetes mutating webhook (or Kyverno policy) that patches replication job pods on creation, forcing the orchestrator (and/or other containers) to the desired resources. [orchestrator ignores limits]This is external to Airbyte but has been reported to work reliably until the upstream bug is fixed.c. Avoid millicores in some env vars There are reports that some CPU env vars only parse correctly as integers (
"1"
, 
"2"
) and that millicores like 
"500m"
cause a fallback to defaults. [workload resources not applied] You’re already using decimals (
0.2
, 
0.4
), which might be parsed correctly, but the issues suggest parsing is fragile. If you experiment, use whole numbers (e.g. 
0.25
→ 
"0.25"
or 
"1"
) and then re‑inspect the pod spec. However, the existing reports show that even correctly set env vars are sometimes ignored, so this may not be sufficient. ◦ They match pods with 
label: airbyte=job-pod
. ◦ Find the 
orchestrator
container. ◦ Apply a JSONPatch to set 
resources.requests/limits
to e.g. 
cpu: "0.4"
, 
memory: "4Gi"
. Given the knowledge sources, there is no documented configuration change that will make your current version honor 
JOB_MAIN_*
/ 
global.jobs.resources
for replication pods in a fully reliable way. The behavior you’re seeing (4 jobs → 4 nodes) is a direct consequence of those bugs. To reduce node churn with the information available: 1. Keep your 
JOB_MAIN_*
env vars as you have them (they’re correct). 2. For each heavy connection, set 
connection.resource_requirements
in the DB to low CPU/memory so at least source/destination containers don’t request 1–2 CPU each. [workload resources not applied] 3. If orchestrator memory/CPU is still too high and drives autoscaling, consider a mutating webhook / Kyverno policy to patch replication pods on creation, as other users have done. [orchestrator ignores limits] I’m limited to what’s in these sources, so I can’t point to a specific Airbyte/Helm version where this is fully fixed. If you open or comment on the linked GitHub issues with your findings (env vars set but pods still at 2 CPU), that will also help the engineering team prioritize a proper fix. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
d
@kapa.ai, n the replication pod I see the following events:
Copy code
Warning  FailedScheduling  9m8s                   default-scheduler  0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  8m44s                  default-scheduler  0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  8m43s (x2 over 8m44s)  default-scheduler  0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  8m43s (x2 over 8m43s)  default-scheduler  0/3 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 2 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  8m38s (x2 over 8m43s)  default-scheduler  0/4 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  8m33s (x2 over 8m33s)  default-scheduler  0/5 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 4 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  8m30s                  default-scheduler  0/5 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 2 Insufficient cpu, 2 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/5 nodes are available: 2 No preemption victims found for incoming pod, 3 Preemption is not helpful for scheduling.
as seen initially 
had untolerated taint(s). no new claims to deallocate, preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling
but then I see 
2 Insufficient cpu, 2 node(s) had untolerated taint(s).
here's the full events:
Copy code
Warning  FailedScheduling  9m8s                   default-scheduler  0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  8m44s                  default-scheduler  0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  8m43s (x2 over 8m44s)  default-scheduler  0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  8m43s (x2 over 8m43s)  default-scheduler  0/3 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 2 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  8m38s (x2 over 8m43s)  default-scheduler  0/4 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  8m33s (x2 over 8m33s)  default-scheduler  0/5 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 4 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/5 nodes are available: 5 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  8m30s                  default-scheduler  0/5 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 2 Insufficient cpu, 2 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/5 nodes are available: 2 No preemption victims found for incoming pod, 3 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  8m30s                  default-scheduler  0/5 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint(s), 3 Insufficient cpu. no new claims to deallocate, preemption: 0/5 nodes are available: 2 Preemption is not helpful for scheduling, 3 No preemption victims found for incoming pod.
  Warning  FailedScheduling  8m28s                  default-scheduler  0/5 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had untolerated taint(s), 3 Insufficient cpu. no new claims to deallocate, preemption: 0/5 nodes are available: 2 Preemption is not helpful for scheduling, 3 No preemption victims found for incoming pod.
  Normal   Scheduled         8m20s                  default-scheduler  Successfully assigned airbyte/replication-job-83-attempt-0 to ip-10-188-3-128.ec2.internal
I'm a bit confused
k
You’re seeing two different scheduling constraints show up over time: 1. Node selector / taints issue (primary at first) The early events are all about affinity/selector and taints:This means: ◦ 
1 node(s) didn't match Pod's node affinity/selector
◦ 
node(s) had untolerated taint(s)
◦ Your replication pod has a 
nodeSelector
(or node affinity) that only matches a subset of nodes. ◦ Some of the nodes the autoscaler is adding have taints that the pod does not tolerate, so the scheduler can’t place the pod there. This pattern (pods not honoring or not having the right tolerations) is similar to what’s discussed in the forum thread about 
JOB_KUBE_TOLERATIONS
and taints/tolerations. [JOB_KUBE_TOLERATIONS] 1. CPU insufficiency appears once more nodes exist Later events add:At that point, among the nodes that do match the selector and taints, some don’t have enough free CPU for the pod’s (still high) requests (e.g. 
cpu: "1"
or 
"2"
per container, as you saw earlier). This “insufficient cpu” pattern is exactly what’s reported in several issues where replication pods request more CPU than expected and remain Pending. [resource limits ignored; insufficient cpu example] ◦ 
2 Insufficient cpu
◦ 
3 Insufficient cpu
So the sequence is: • Initially: only 1 node; it doesn’t match the pod’s nodeSelector → 
didn't match Pod's node affinity/selector
. • Autoscaler adds nodes, but they’re tainted in a way the pod doesn’t tolerate → 
had untolerated taint(s)
. • As more nodes come up, some match selector/taints but don’t have enough CPU for the (large) pod requests → 
Insufficient cpu
. • Eventually, one node both matches selector/taints and has enough CPU, so the pod is scheduled: 
Successfully assigned ...
. The knowledge sources don’t give enough detail to say exactly how your nodeSelector/taints are configured or how to change your autoscaler/node groups, but they do show: • Taints/tolerations must be aligned between your nodes and Airbyte job pods; otherwise you’ll see “untolerated taint(s)” and Pending pods. [JOB_KUBE_TOLERATIONS] • High default CPU requests for replication pods can easily lead to “Insufficient cpu” and aggressive node scaling. [resource limits ignored; S3 datalake insufficient resources] In other words, the logs are consistent: • First, the pod can’t run anywhere because of selector/taints. • Then, even where it can run, some nodes don’t have enough CPU for its (too large) requests. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
d
@kapa.ai, Taints: DeletionCandidateOfClusterAutoscaler=1780912348:PreferNoSchedule this is the only taint I have on my nodes
k
Thanks for the extra detail. Based on the knowledge sources, I can’t see anything specific about the 
DeletionCandidateOfClusterAutoscaler=...:PreferNoSchedule
taint itself, but I can summarize what is clear and how it relates to your situation. 1. That taint is from the cluster autoscaler, not Airbyte The sources show Airbyte interacting with autoscalers (e.g. GKE autoscaler evicting Temporal / workload‑api‑server pods), but they don’t document this specific taint or how Airbyte should treat it. [MySQL CDC + autoscaler] So I can’t say from the docs whether Airbyte jobs are expected to tolerate 
DeletionCandidateOfClusterAutoscaler
. 2. Your scheduling events still make sense with this taint present The events you shared show:From the knowledge sources:The docs don’t say whether Airbyte sets any toleration for autoscaler taints by default, so I have to assume it does not tolerate 
DeletionCandidateOfClusterAutoscaler=...:PreferNoSchedule
unless you add a matching toleration via 
jobs.kube.tolerations
/ 
JOB_KUBE_TOLERATIONS
. ◦ 
node(s) didn't match Pod's node affinity/selector
◦ 
node(s) had untolerated taint(s)
◦ later: 
Insufficient cpu
◦ Airbyte job pods use 
nodeSelector
/ tolerations derived from 
jobs.kube.nodeSelector
and 
JOB_KUBE_TOLERATIONS
/ related env vars. [Helm jobs.kube.*; JOB_KUBE_TOLERATIONS usage] ◦ If a node has a taint that the pod doesn’t tolerate, you get exactly the “had untolerated taint(s)” message. [JOB_KUBE_TOLERATIONS] 1. *What is clearly documented you can control from Airbyte’s side*From the chart snippets and forum posts: [Helm jobs.kube.*; JOB_KUBE_TOLERATIONS example; JOB_KUBE_TOLERATIONS forum]jobs: kube: nodeSelector: airbyte_node_type: workerjobs: kube: tolerations: - key: taint-key operator: Equal value: taint-value effect: PreferNoScheduleworker: extraEnv: - name: JOB_KUBE_TOLERATIONS value: "key=taint-key,operator=Equal,value=taint-value,effect=PreferNoSchedule"The docs don’t explicitly mention autoscaler taints, but this is the only documented way to make Airbyte job pods tolerate a taint. [JOB_KUBE_TOLERATIONS; Helm jobs.kube.tolerations] ◦ You can set node selectors for all jobs: ◦ You can set tolerations for job pods: ◦ Or via env var on worker / workload‑launcher: Because the knowledge sources don’t cover that specific autoscaler taint in detail, I can’t definitively say whether you should tolerate it or adjust your autoscaler instead. What they do show is: • Airbyte job pods will not schedule on tainted nodes unless you configure matching tolerations via 
jobs.kube.tolerations
/ 
JOB_KUBE_TOLERATIONS
. [JOB_KUBE_TOLERATIONS; Helm jobs.kube.tolerations] • If you see “untolerated taint(s)” in events, adding an appropriate toleration is the documented way to let those pods run there. If you want, you can share the 
jobs.kube
section of your current 
values.yaml
, and I can suggest how to add a toleration there using only the patterns shown in the docs. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
d
@kapa.ai, ok here's an example of requests: for a replication job:
Copy code
k describe pod replication-job-87-attempt-0 | grep -i requests: -A2
    Requests:
      cpu:     2
      memory:  2Gi
--
    Requests:
      cpu:     2
      memory:  2Gi
--
    Requests:
      cpu:     1
      memory:  1Gi
--
    Requests:
      cpu:     1
      memory:  1Gi
how are these determined? Where how are they configured? I guess due to this values new nodes are schedulled ?
k
You’re right: with requests like 
cpu: 1–2
per container, the scheduler/autoscaler will often add new nodes. The knowledge sources show how these numbers are supposed to be determined, and also that there are bugs in 2.x where they don’t always follow your 
values.yaml
. 1. Where these requests are intended to come from For sync (replication) jobs, Airbyte can derive container resources from several layers, with this precedence (highest wins) [Configuring connector resources]: 1. Connection-specific: 
connection.resource_requirements
in the DB 2. Connector-specific: per-connector config in the UI (if present) 3. Connector definition: 
actor_definition.resource_requirements
in the DB 4. Instance-wide defaults: ◦ Old style: 
global.jobs.resources.*
→ 
JOB_MAIN_CONTAINER_*
env vars ◦ New chart v2 style: 
global.workloads.resources.*
(e.g. 
replication
, 
mainContainer
) [jobs.resources broken; Configuring connector resources] So in a healthy setup, the replication pod’s source/destination/orchestrator containers should get their `requests`/`limits` from: • Connection 
resource_requirements
(if set), otherwise • Connector definition 
resource_requirements
, otherwise • The instance-wide defaults (
JOB_MAIN_CONTAINER_*
/ 
workloads.resources.*
). 2. What’s actually happening in 2.x (matches your numbers) Multiple users on 2.0.x report exactly what you’re seeing: • Replication pods with: ◦ 
limits={memory=2Gi, cpu=2}
◦ 
requests={memory=1–2Gi, cpu=1–2}
• This happens even when: ◦ 
global.jobs.resources
or 
global.workloads.resources
are set to much smaller values, and ◦ 
JOB_MAIN_CONTAINER_*
/ 
REPLICATION_ORCHESTRATOR_*
env vars are correctly present in the workload-launcher pod. [resource limits ignored; workload resources not applied; orchestrator ignores limits] Examples from the issues: • Workload-launcher log: 
resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=2}, requests={memory=1Gi, cpu=1}, ...)
[resource limits ignored] • Pod spec: 
limits={memory=2Gi, cpu=2}, requests={memory=2Gi, cpu=2}
despite lower values in 
values.yaml
and env vars. [workload resources not applied] Engineering has confirmed this is a bug in the workload-launcher / resource resolution, not just Helm config: the launcher is using internal defaults (around 2 CPU / 1–2Gi) instead of the configured values. [resource limits ignored; orchestrator ignores limits] So in your case: • Those 
Requests: cpu: 2 / 1, memory: 2Gi / 1Gi
are coming from Airbyte’s internal defaults in the current 2.x code path, not from your 
global.jobs.resources
or 
JOB_MAIN_*
values. • Yes, those high requests are exactly why the scheduler/autoscaler keeps adding nodes. 3. What you can actually control today (workarounds) Based on the sources, the only reliable knobs that do affect replication job resources in 2.0.x are: 1. Connection-level 
resource_requirements
in the DB This has been confirmed to override the defaults for source/destination containers when global settings are ignored [workload resources not applied; orchestrator ignores limits]:UPDATE connection SET resource_requirements = jsonb_build_object( 'cpu_request', '0.2', 'cpu_limit', '0.4', 'memory_request','256Mi', 'memory_limit', '4Gi' ) WHERE id = 'your-connection-id';That will shrink the source/destination containers for that connection, reducing total CPU per job. 2. Mutating webhook / Kyverno policy Several users implemented a Kubernetes mutating webhook that patches replication job pods on creation, forcing the orchestrator (and/or all containers) to desired resources (e.g. 0.4 CPU, 4Gi). [orchestrator ignores limits] 3. *Instance-wide env vars (
JOB_MAIN_*
)* You already set these, and the envs are visible in 
workload-launcher
, but the issues show they are not consistently honored for replication pods in 2.0.1. [resource limits ignored] So they’re intended to determine the defaults, but in your version they are effectively overridden by the bug. So to answer your questions directly: • How are these 
cpu: 2 / 1
, 
memory: 2Gi / 1Gi
requests determined? In your version, they are coming from hardcoded / internal defaults in the workload-launcher / platform, not from your 
values.yaml
, even though they should. This matches multiple open GitHub issues. • Where/how are they configured? They should be configured via: ◦ 
connection.resource_requirements
(DB), ◦ 
actor_definition.resource_requirements
(DB), ◦ and instance-wide 
global.jobs.resources
/ 
global.workloads.resources
→ 
JOB_MAIN_CONTAINER_*
env vars. [Configuring connector resources] But due to the bug, only the DB-level 
resource_requirements
and external patching (mutating webhook) are reliably effective right now. • Are these values why new nodes are scheduled? Yes. With each replication pod requesting ~2 vCPU for some containers, the scheduler often can’t fit multiple jobs on a single node, so the cluster autoscaler keeps adding nodes. This is the same “Insufficient cpu” behavior reported in the linked issues. [resource limits ignored] If you want to reduce node churn immediately, the most effective levers (per the knowledge sources) are: • Set 
resource_requirements
per connection in the DB to low CPU/memory. • Optionally add a mutating webhook / Kyverno policy to cap orchestrator resources. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
11 Views
Open in Slack
PreviousNext
Powered by