https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> Does CDC based workflow consume mor...
# airbyte-agents-ask-ai
p
@kapa.ai Does CDC based workflow consume more memory and cpu as compared to batch in Airbyte ?
k
CDC vs. Batch: Memory and CPU Consumption in Airbyte Yes, CDC-based workflows can be more memory and CPU intensive compared to standard batch (non-CDC incremental) syncs in Airbyte. Here's a breakdown based on available information: Memory Considerations • In-memory buffering: CDC uses an internal queue buffer (default: 10,000 records) to hold change events before processing. Larger queues handle burst changes efficiently but consume more memory. [CDC Best Practices] • Large CDC syncs can cause OOM errors: There are known cases where large CDC syncs (e.g., processing 70+ million row changes) result in Java heap space / OutOfMemoryError crashes, even when significant memory is allocated. [GitHub Issue] • Orchestrator memory: The container orchestrator in CDC replication jobs has been reported to have a hardcoded 1Gi memory limit in some versions, causing OOM crashes on large CDC syncs with big binlog/WAL gaps. [GitHub Issue] • General sync memory: The Source worker reads up to 10,000 records in memory. For tables with large row sizes, this can require significant RAM (e.g., 0.5MB average row size × 10,000 records = ~5GB RAM). [Scaling Airbyte] CPU Considerations • CDC involves continuous log reading and event processing via Debezium, which adds processing overhead compared to simple periodic queries used in standard incremental syncs. • For high-activity databases or schemas with many tables, CDC processing time (and thus CPU usage) increases independently of data volume. [CDC Best Practices] Key Differences from Batch Recommendations • For large CDC workloads, allocate more memory to connector pods via 
connection.resource_requirements
or 
values.yaml
. [Scaling Airbyte] • Monitor memory and CPU usage closely, especially during the initial snapshot phase or after large bursts of changes. • For very large databases, consider setting the Invalid CDC Position Behavior to "Fail Sync" to avoid unexpected resource-intensive automatic re-snapshots. [CDC Best Practices] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
p
@kapa.ai For 
global.jobs
I have allocated these many resources:
Copy code
resources:
      requests:
        cpu: 500m
        memory: 4Gi
      limits:
        cpu: 2000m
        memory: 8Gi
For `server`:
Copy code
resources:
    requests:
      cpu: 200m
      memory: 512Mi
    limits:
      cpu: 1000m
      memory: 1Gi
For `temporal`:
Copy code
resources:
    requests:
      cpu: 200m
      memory: 256Mi
    limits:
      cpu: 500m
      memory: 512Mi
For `worker`:
Copy code
resources:
    requests:
      cpu: 500m
      memory: 512Mi
    limits:
      cpu: 2000m
      memory: 2Gi
for 
workloadLauncher
and 
bootloader
there are no specific resource allocation << this gonna follow gloabl one.. correct ? For `connector`: No specific resource allocation << so this will follow global one right ? Connectors are only gonna spin up in 
workers
node-pool where as all above services are going to be spinned up in 
core
node-pool.. Help me tune the configuration of my Airbyte instance to make things work... In worker node-pool.. these are machine configs:
Copy code
name: "workers"
number of nodes = 3 (autoscaling enable) >> we need this for running jobs
Resources: 4vCPU & 16 GiB RAM
For cpu node-pool these are machine configs:
Copy code
name: "core"
number of nodes = 2 (no autoscaling) >> we need this to be running 24X7 because this will host services that will be running all the time
Resource (machine type): General Purpose 4vCPU & 16 GiB RAM
Help me tune my airbyte config.. please note I am running CDC and Batch both workflows..
k
Tuning Your Airbyte Configuration Let me analyze your setup and provide recommendations based on your node pools and workload mix. Understanding Your Setup Core Node Pool (2 nodes × 4vCPU / 16GiB RAM = 8vCPU / 32GiB total) Hosts: 
server
, 
temporal
, 
worker
, 
workloadLauncher
, 
bootloader
, 
workload-api-server
Workers Node Pool (3 nodes × 4vCPU / 16GiB RAM = 12vCPU / 48GiB total, autoscaling) Hosts: Connector pods (source, destination, orchestrator) Issues with Your Current Configuration 1. workloadLauncher and bootloader — They do NOT follow global.jobs > ⚠️ Important correction: 
global.jobs.resources
applies to connector/job pods (source, destination containers), not to Airbyte service components like 
workloadLauncher
or 
bootloader
. These services need their own explicit resource definitions. [Configuring Connector Resources] 2. Connector pods — They DO follow global.jobs.resources Your connector pods (source/destination) will use 
global.jobs.resources
as the instance-wide default, which can be overridden per connector definition or per connection. [Configuring Connector Resources] 3. Orchestrator container memory bug There is a known open bug where the orchestrator container ignores 
CONTAINER_ORCHESTRATOR_MEMORY_LIMIT
and always uses 1Gi, which is especially problematic for CDC workloads. [GitHub Issue] Recommended values.yaml Configuration
Copy code
global:
  jobs:
    resources:
      requests:
        cpu: 500m
        memory: 4Gi
      limits:
        cpu: 2000m
        memory: 8Gi
    kube:
      nodeSelector:
        node-pool: workers   # connector pods go to workers node pool

  workloads:
    resources:
      # Attempt to set orchestrator memory (note: bug exists in v2.0.x)
      containerOrchestrator:
        memory:
          request: 2Gi
          limit: 4Gi   # increase for CDC workloads

# --- Core Services ---

server:
  nodeSelector:
    node-pool: core
  resources:
    requests:
      cpu: 200m
      memory: 512Mi
    limits:
      cpu: 1000m
      memory: 1Gi

temporal:
  nodeSelector:
    node-pool: core
  resources:
    requests:
      cpu: 200m
      memory: 256Mi
    limits:
      cpu: 500m
      memory: 512Mi

worker:
  nodeSelector:
    node-pool: core
  resources:
    requests:
      cpu: 500m
      memory: 512Mi
    limits:
      cpu: 2000m
      memory: 2Gi
  maxSyncWorkers: 5    # tune based on concurrent syncs needed
  maxCheckWorkers: 3

workloadLauncher:
  nodeSelector:
    node-pool: core
  resources:
    requests:
      cpu: 250m
      memory: 512Mi
    limits:
      cpu: 1000m
      memory: 1Gi
  # Attempt env var overrides for connector pod resources
  extraEnv:
    - name: JOB_MAIN_CONTAINER_CPU_REQUEST
      value: "500m"
    - name: JOB_MAIN_CONTAINER_CPU_LIMIT
      value: "2000m"
    - name: JOB_MAIN_CONTAINER_MEMORY_REQUEST
      value: "4Gi"
    - name: JOB_MAIN_CONTAINER_MEMORY_LIMIT
      value: "8Gi"
    - name: CONTAINER_ORCHESTRATOR_MEMORY_REQUEST
      value: "2Gi"
    - name: CONTAINER_ORCHESTRATOR_MEMORY_LIMIT
      value: "4Gi"

airbyteBootloader:
  nodeSelector:
    node-pool: core
  resources:
    requests:
      cpu: 200m
      memory: 256Mi
    limits:
      cpu: 500m
      memory: 1Gi

workloadApiServer:
  nodeSelector:
    node-pool: core
  resources:
    requests:
      cpu: 200m
      memory: 256Mi
    limits:
      cpu: 500m
      memory: 1Gi
Core Node Pool — Resource Budget Check ✅ This fits comfortably within your 2-node core pool (8vCPU / 32GiB total). Workers Node Pool — Resource Budget Check Each sync job creates 3 pods: 
orchestrator
+ 
source
+ 
destination
. With your current 
global.jobs
settings (4Gi request per container): • Per sync: ~3 × 4Gi = 12Gi memory + ~3 × 500m = 1500m CPU requested • Per node (16GiB): ~1 concurrent sync per node comfortably • 3 nodes: ~3 concurrent syncs before autoscaling kicks in > ⚠️ For CDC workloads, memory usage can spike significantly. Given the orchestrator OOM bug at 1Gi, consider reducing 
global.jobs.resources.requests.memory
to 
2Gi
and keeping limits at 
8Gi
to allow more syncs to be scheduled while still permitting bursts. CDC-Specific Workaround for Orchestrator OOM Since the orchestrator memory bug is unresolved [GitHub Issue], you can apply a connection-level override via SQL for CDC connections:
Copy code
UPDATE connection
SET resource_requirements = jsonb_build_object(
  'cpu_request', '0.5',
  'cpu_limit', '2',
  'memory_request', '4Gi',
  'memory_limit', '8Gi'
)
WHERE id = '<your-cdc-connection-id>';
> Note: This applies to source/destination containers. The orchestrator container memory fix requires either the upstream bug fix or a Kubernetes Mutating Webhook workaround as described in the issue. [GitHub Issue] Summary of Key Recommendations 1. Explicitly define resources for 
workloadLauncher
and 
bootloader
— they don't inherit 
global.jobs
. 2. Add 
nodeSelector
 to all services to enforce core/workers pool separation. 3. Be aware of the orchestrator OOM bug — apply the SQL workaround for CDC connections. 4. Tune 
maxSyncWorkers
 based on your node capacity to avoid over-scheduling. 5. Monitor CPU and memory usage after deployment and adjust limits accordingly. [Scaling Airbyte] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
p
@kapa.ai how can I see what all pods are engaged in CDC workflow ? Help me with kubectl command
k
Based on the knowledge sources, Airbyte workload pods carry specific labels that you can use to filter and identify pods. Here are the relevant 
kubectl
commands: Find All Active Job Pods
Copy code
kubectl get pods -l airbyte=job-pod --all-namespaces
Find Pods by Job Type (Sync = CDC or Batch)
Copy code
# Get all sync/replication pods
kubectl get pods -l job_type=sync --all-namespaces

# Get all sync pods with their labels visible
kubectl get pods -l job_type=sync --all-namespaces --show-labels
Find Pods for a Specific Connection
Copy code
# If you know your CDC connection ID
kubectl get pods -l connection_id=<your-cdc-connection-id> --all-namespaces
Inspect Logs of the Orchestrator Container (Richest CDC Debug Source)
Copy code
# Get orchestrator logs for a specific pod
kubectl logs <pod-name> -c orchestrator -n <namespace>
Verify Pod Labels on a Specific Pod
Copy code
kubectl get pod <pod-name> -n <namespace> --show-labels
> ⚠️ Important caveat: The knowledge sources do not mention a label specifically for CDC vs. batch distinction. Both CDC and batch syncs run as 
job_type=sync
pods. [Log Collection] To distinguish CDC pods specifically, you would need to: 1. Know your CDC `connection_id`(s) from the Airbyte UI 2. Filter by that 
connection_id
label using the command above The available pod labels you can filter on are: So the most practical approach for CDC is filtering by 
source_image_name
if you know your CDC source connector:
Copy code
# Example: find all MySQL CDC sync pods
kubectl get pods -l source_image_name=source-mysql,job_type=sync --all-namespaces --show-labels
👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
Open in Slack
PreviousNext
Powered by