<@U04S8PLJWMT> Warning Evicted 21m kubelet ...
# replication-ask-ai
k
@kapa.ai Warning Evicted 21m kubelet The node was low on resource: ephemeral-storage. Threshold quantity: 2146223340, available: 2095756Ki. Container orchestrator was using 6688Ki, request is 0, has larger consumption of ephemeral-storage. Container destination was using 9541984Ki, request is 0, has larger consumption of ephemeral-storage. Container source was using 8Ki, request is 0, has larger consumption of ephemeral-storage.
k
Your node is evicting Airbyte job pods because they exceed the node’s ephemeral-storage threshold. The eviction message shows that especially the
destination
and
orchestrator
containers are using several GiB of ephemeral storage while having no storage requests/limits set. From the docs: • Airbyte recommends paying attention to disk usage (including logs and images) and ensuring enough disk per node, especially on Kubernetes. [Scaling Airbyte] • Connector pods (source, destination, orchestrator) support explicit resource configuration, including
ephemeral-storage
, and the effective values are logged when a job starts. [Configuring resources] Given that, the concrete actions are: 1. Increase node disk / ephemeral storage capacity Ensure your Kubernetes nodes have sufficient disk; Airbyte recommends at least 30GB per node, and often more for many connectors or heavy logging. [Scaling Airbyte] 2. Set
ephemeral-storage
requests/limits on connector pods
Configure connector resources so that
source
,
destination
, and
orchestrator
containers have explicit
ephemeral-storage
requests and limits, not
0
. This lets the scheduler place pods appropriately and can prevent surprise evictions. You can then confirm the values in the job logs as shown in the docs. [Configuring resources] 3. Reduce log / temporary data volume if possible Long-running syncs can generate a lot of logs and temporary data, which contributes to ephemeral-storage usage. The docs note this as a known disk-space concern. [Scaling Airbyte] The knowledge sources do not provide exact YAML examples for setting
ephemeral-storage
on Airbyte’s Kubernetes deployment, but they do confirm that connector resource requirements (including ephemeral storage) are configurable and visible in logs. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
Your node is evicting Airbyte job pods because it runs out of ephemeral-storage. The warning shows that the
orchestrator
and especially the
destination
container are consuming significant ephemeral storage, and since their
ephemeral-storage
requests/limits are
0
, the kubelet can’t reserve space for them and eventually evicts the pod. From the docs: • Airbyte recommends paying attention to disk usage (including logs and connector images) and provisioning enough disk per node (at least ~30GB per node, and often more) to avoid disk-related issues on Kubernetes. [Scaling Airbyte] • You can explicitly configure ephemeral-storage requests/limits per connector (source/destination) via the connector resource configuration; these settings are logged when a job pod is launched, including
ephemeral-storage
if set. [Configuring resources] Given that, the concrete levers you have (all at the Kubernetes/Airbyte config level) are: 1. Increase node disk / ephemeral storage capacity Ensure your Kubernetes nodes have enough disk so that Airbyte job pods (and their logs, temp files, and images) don’t hit the kubelet’s eviction thresholds. [Scaling Airbyte] 2. Set
ephemeral-storage
requests/limits for connectors
In your Airbyte deployment, configure connector resource requirements to include
ephemeral-storage
for
source
,
destination
, and possibly
orchestrator
so the scheduler reserves space and you can cap per-pod usage. You can then confirm in the job logs that the expected
ephemeral-storage
values are being applied. [Configuring resources] 3. Reduce concurrent jobs / pods if the node is saturated Since each sync job creates a pod and Airbyte recommends sizing the cluster to handle up to
2 x
number-of-possible-concurrent-connections pods, you may need either more nodes or fewer concurrent syncs so that total ephemeral-storage usage stays below eviction thresholds. [Scaling Airbyte] The knowledge sources do not provide a more detailed, step‑by‑step Kubernetes YAML example for setting
ephemeral-storage
, but they do confirm that ephemeral-storage can be configured as part of connector resource requirements and that disk/ephemeral storage sizing is a key scaling concern. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
Your pod was evicted because the node ran out of ephemeral-storage, and the Airbyte job containers (
orchestrator
,
destination
,
source
) have no ephemeral-storage requests/limits set (
request is 0
), so the kubelet treats them as unbounded and evicts them when node thresholds are hit. From the Airbyte docs: • Airbyte explicitly supports configuring ephemeral-storage for connector containers (source, destination, orchestrator) as part of their resource requirements. [Configuring resources] • When a job starts, Airbyte logs the effective resource requirements for each container, including
ephemeral-storage
, so you can verify what is actually applied. [Configuring resources] • Disk usage (including logs and images) is a known concern; Airbyte recommends provisioning sufficient disk per node because long-running syncs can generate a lot of data. [Scaling Airbyte] Given that, the actionable steps (as far as the docs go) are: 1. Ensure enough disk on the nodes Airbyte recommends at least 30GB of disk per node, and more if you run many connectors or long syncs. [Scaling Airbyte] 2. Configure connector resource requirements including
ephemeral-storage
Use Airbyte’s resource configuration (per connector / per connection) so that
source
,
destination
, and
orchestrator
containers have explicit
ephemeral-storage_request
and
ephemeral-storage_limit
values instead of
0
. The docs confirm these fields exist and are logged, but they do not show the exact Kubernetes YAML or UI steps in the provided snippets. [Configuring resources] 3. Verify in job logs After changing resource settings, start a sync and check the job logs for lines like:
Copy code
[source] ... limits={memory=2Gi, ephemeral-storage=5G, cpu=1}, requests={memory=1Gi, ephemeral-storage=5G, cpu=0.5}
   [destination] ...
   [orchestrator] ...
to confirm that
ephemeral-storage
is being applied as expected. [Configuring resources] The knowledge sources do not provide more detailed, step‑by‑step configuration examples for ephemeral-storage specifically, so I can’t give an exact config snippet, only the high‑level guidance above. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
Your pod was evicted because the node ran out of ephemeral-storage, and all three Airbyte job containers (
orchestrator
,
destination
,
source
) have no ephemeral‑storage requests set (
request is 0
), so the kubelet treats them as unbounded and evicts them once node thresholds are crossed. From the Airbyte docs: • Airbyte jobs (syncs, discover, etc.) run in Kubernetes pods launched by the workload system; scaling and stability depend heavily on having enough disk/ephemeral storage on the nodes. [Scaling Airbyte] • Disk space is a known concern: long‑running syncs and connector images can consume significant disk, so Airbyte recommends at least 30GB per node, and to err on the side of over‑provisioning. [Scaling Airbyte] • Connector containers (source, destination, orchestrator) have configurable resource requirements, including
ephemeral-storage
limits/requests, and the effective values are logged when a job starts (you’ll see
ephemeral-storage=...
in the log line for each container). [Configuring connector resources] Given what’s in the docs, the concrete mitigations are: 1. Increase node disk / ephemeral storage capacity Ensure your Kubernetes nodes meet or exceed the recommended disk size (≥30GB per node, more if you run many or long syncs). [Scaling Airbyte] 2. Configure connector resource requirements to include
ephemeral-storage
Use Airbyte’s connector/connection resource configuration so that
source
,
destination
, and
orchestrator
containers have explicit
ephemeral-storage
requests and limits instead of
0
. You can then confirm they’re applied by checking the job logs for lines like the example in the docs that show
ephemeral-storage=...
for each container. [Configuring connector resources] The provided knowledge sources do not show the exact UI steps or YAML/Helm fields for setting
ephemeral-storage
, only that these fields exist in the resource requirements and appear in logs, so I can’t give a precise configuration snippet. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
k
@kapa.ai Is this correct configuration in values.yaml? jobs: resources: ## Example: ## requests: ## memory: 256Mi ## cpu: 250m # -- Job resource requests requests: memory: 2Gi ephemeral-storage: 3Gi cpu: 1 ## Example: ## limits: ## cpu: 200m ## memory: 1Gi # -- Job resource limits limits: cpu: 8 memory: 48Gi ephemeral-storage: 5Gi
k
The knowledge sources confirm that the
jobs.resources
block in
values.yaml
is the right place to configure job-level resource requests and limits, but they do not explicitly document using
ephemeral-storage
there, so I can’t fully validate that specific key from the Airbyte docs alone. What is documented: • The
jobs.resources.requests
and
jobs.resources.limits
structure is correct and is how you set CPU and memory for all jobs in the Helm chart. [Values.yaml ref; 1.8 values] • Example from the docs (showing the same structure you’re using, but only with CPU/memory):
Copy code
jobs:
      resources:
        # -- Job resource requests
        requests: {}
        # -- Job resource limits
        limits: {}
[1.8 values; 2.0 values] So, based on the docs: • Structurally, your snippet matches the documented pattern for
jobs.resources
(requests/limits under
jobs.resources
). • The docs do not explicitly show
ephemeral-storage
under
jobs.resources
, so I cannot confirm from these sources alone that Airbyte’s Helm chart will propagate that specific field as you expect. If you apply this config, you can verify whether it’s honored by checking the actual pod spec (
kubectl get pod ... -o yaml
) and/or the job logs where Airbyte prints effective resource requirements. [Configuring connector resources] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.