https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> Running abctl on EC2 instance, Airb...
# ask-ai
y
@kapa.ai Running abctl on EC2 instance, Airbyte is unable to run two concurrent syncs. One sync appears to be waiting for another to complete:
Copy code
2025-02-06 17:53:00 platform > Executing worker wrapper. Airbyte version: 1.2.0
3
2025-02-06 17:53:00 platform > Creating workload df6abdf3-c7d7-4c66-af44-1deb43a24633_409_0_sync
4
2025-02-06 17:53:01 platform > Workload df6abdf3-c7d7-4c66-af44-1deb43a24633_409_0_sync is pending
5
2025-02-06 17:53:01 platform > APPLY Stage: CLAIM — (workloadId = df6abdf3-c7d7-4c66-af44-1deb43a24633_409_0_sync) — (dataplaneId = local)
6
2025-02-06 17:53:01 platform > Claimed: true for df6abdf3-c7d7-4c66-af44-1deb43a24633_409_0_sync via API for local
7
2025-02-06 17:53:01 platform > APPLY Stage: CHECK_STATUS — (workloadId = df6abdf3-c7d7-4c66-af44-1deb43a24633_409_0_sync) — (dataplaneId = local)
8
2025-02-06 17:53:01 platform > No pod found running for workload df6abdf3-c7d7-4c66-af44-1deb43a24633_409_0_sync
9
2025-02-06 17:53:01 platform > APPLY Stage: BUILD — (workloadId = df6abdf3-c7d7-4c66-af44-1deb43a24633_409_0_sync) — (dataplaneId = local)
10
2025-02-06 17:53:01 platform > APPLY Stage: MUTEX — (workloadId = df6abdf3-c7d7-4c66-af44-1deb43a24633_409_0_sync) — (dataplaneId = local)
11
2025-02-06 17:53:01 platform > Mutex key: df6abdf3-c7d7-4c66-af44-1deb43a24633 specified for workload: df6abdf3-c7d7-4c66-af44-1deb43a24633_409_0_sync. Attempting to delete existing pods...
12
2025-02-06 17:53:01 platform > Mutex key: df6abdf3-c7d7-4c66-af44-1deb43a24633 specified for workload: df6abdf3-c7d7-4c66-af44-1deb43a24633_409_0_sync found no existing pods. Continuing...
13
2025-02-06 17:53:01 platform > APPLY Stage: LAUNCH — (workloadId = df6abdf3-c7d7-4c66-af44-1deb43a24633_409_0_sync) — (dataplaneId = local)
14
2025-02-06 17:53:01 platform > [initContainer] image: airbyte/workload-init-container:1.2.0 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=2Gi, cpu=2}, additionalProperties={})
15
2025-02-06 17:53:01 platform > Launching replication pod: replication-job-409-attempt-0 with containers:
16
2025-02-06 17:53:01 platform > [source] image: airbyte/source-mysql:3.7.3 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=1Gi, cpu=1}, additionalProperties={})
17
2025-02-06 17:53:01 platform > [destination] image: airbyte/destination-snowflake:3.15.2 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=1Gi, cpu=1}, additionalProperties={})
18
2025-02-06 17:53:01 platform > [orchestrator] image: airbyte/container-orchestrator:1.2.0 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=2Gi, cpu=2}, additionalProperties={})
19
2025-02-06 17:55:01 platform > Workload df6abdf3-c7d7-4c66-af44-1deb43a24633_409_0_sync is claimed
20
2025-02-06 17:57:01 platform > Workload df6abdf3-c7d7-4c66-af44-1deb43a24633_409_0_sync is claimed
k
Based on the provided logs and knowledge sources, it appears that Airbyte is indeed having difficulty running concurrent syncs on your EC2 instance. This issue is likely related to resource constraints or configuration settings. Let's explore some possible reasons and solutions: 1. Resource Limitations: The logs show that the sync job is requesting significant resources:
Copy code
[initContainer] image: airbyte/workload-init-container:1.2.0 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=2Gi, cpu=2}, additionalProperties={})
[source] image: airbyte/source-mysql:3.7.3 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=1Gi, cpu=1}, additionalProperties={})
[destination] image: airbyte/destination-snowflake:3.15.2 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=1Gi, cpu=1}, additionalProperties={})
[orchestrator] image: airbyte/container-orchestrator:1.2.0 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=2Gi, cpu=2}, additionalProperties={})
Your EC2 instance may not have enough resources to run multiple syncs concurrently. Consider scaling up your EC2 instance or adjusting the resource requests and limits. 2. Concurrent Sync Limits: According to the [Scaling Airbyte After Installation](https://docs.airbyte.com/enterprise-setup/scaling-airbyte) documentation, there are configurable limits to the number of syncs that can be run at once. You may need to adjust these settings:
Copy code
yaml
worker:
  extraEnvs:
    - name: MAX_SYNC_WORKERS
      value: ## e.g. 5
    - name: MAX_CHECK_WORKERS
      value: ## e.g. 5
3. Worker Replicas: If you intend to run many syncs simultaneously, you might need to increase the number of worker replicas:
Copy code
yaml
worker:
  replicaCount: ## e.g. 2
4. Mutex Key: The logs show that a mutex key is being used:
Copy code
Mutex key: df6abdf3-c7d7-4c66-af44-1deb43a24633 specified for workload: df6abdf3-c7d7-4c66-af44-1deb43a24633_409_0_sync. Attempting to delete existing pods...
This suggests that Airbyte is enforcing a mutex to prevent concurrent syncs for the same connection. This behavior is intentional to avoid conflicts, but it may be causing the perceived inability to run concurrent syncs. To address this issue, you can try the following: 1. Increase the resources of your EC2 instance to accommodate multiple concurrent syncs. 2. Adjust the 
MAX_SYNC_WORKERS
and 
MAX_CHECK_WORKERS
settings to allow more concurrent syncs. 3. Increase the 
worker.replicaCount
if you're running Airbyte on Kubernetes. 4. Ensure that you're not trying to run multiple syncs for the same connection simultaneously, as the mutex will prevent this.
Based on the provided logs and knowledge sources, it appears that Airbyte is indeed having difficulty running concurrent syncs on your EC2 instance. This issue is likely related to resource constraints or configuration settings. Let's explore some possible reasons and solutions: 1. Resource Limitations: The logs show that the sync job is requesting significant resources:
Copy code
[initContainer] image: airbyte/workload-init-container:1.2.0 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=2Gi, cpu=2}, additionalProperties={}) [source] image: airbyte/source-mysql:3.7.3 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=1Gi, cpu=1}, additionalProperties={}) [destination] image: airbyte/destination-snowflake:3.15.2 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=1Gi, cpu=1}, additionalProperties={}) [orchestrator] image: airbyte/container-orchestrator:1.2.0 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=2Gi, cpu=2}, additionalProperties={})
Your EC2 instance may not have enough resources to run multiple syncs concurrently. Consider scaling up your EC2 instance or adjusting the resource requests and limits. 2. Concurrent Sync Limits: According to the [Scaling Airbyte After Installation](https://docs.airbyte.com/enterprise-setup/scaling-airbyte) documentation, there are configurable limits to the number of syncs that can be run at once. You may need to adjust these settings:
Copy code
yaml worker:   extraEnvs:     - name: MAX_SYNC_WORKERS       value: ## e.g. 5     - name: MAX_CHECK_WORKERS       value: ## e.g. 5
3. Worker Replicas: If you intend to run many syncs simultaneously, you might need to increase the number of worker replicas:
Copy code
yaml worker:   replicaCount: ## e.g. 2
4. Mutex Key: The logs show that a mutex key is being used:
Copy code
Mutex key: df6abdf3-c7d7-4c66-af44-1deb43a24633 specified for workload: df6abdf3-c7d7-4c66-af44-1deb43a24633_409_0_sync. Attempting to delete existing pods...
This suggests that Airbyte is enforcing a mutex to prevent concurrent syncs for the same connection. This behavior is intentional to avoid conflicts, but it may be causing the perceived inability to run concurrent syncs. To address this issue, you can try the following: 1. Increase the resources of your EC2 instance to accommodate multiple concurrent syncs. 2. Adjust the 
MAX_SYNC_WORKERS
and 
MAX_CHECK_WORKERS
settings to allow more concurrent syncs. 3. Increase the 
worker.replicaCount
if you're running Airbyte on Kubernetes. 4. Ensure that you're not trying to run multiple syncs for the same connection simultaneously, as the mutex will prevent this. If these steps don't resolve the issue, you may need to investigate further by examining the full logs and your specific Airbyte configuration. For more detailed guidance, you might want to consult the [Airbyte documentation](https://docs.airbyte.com/) or reach out to Airbyte support for assistance with your specific setup.
Open in Slack
PreviousNext
Powered by