https://linen.dev logo
Join Slack
Powered by
What Kubernetes distribution do you recommend for ...
# replication-ask-ai
w
What Kubernetes distribution do you recommend for self-hosted Airbyte v2.0.x on a single EC2 node? We're currently on K3s v1.34.3 and experiencing persistent issues where ConnectionManagerWorkflow tasks on the CONNECTION_UPDATER queue are scheduled but never started by the worker. Source checks and destination checks work fine (CHECK_CONNECTION queue). This happens on fresh installs with both local and external PostgreSQL. Is K3s officially supported, or should we use something else like k3d, minikube, or full kubeadm?
k
Kubernetes Distribution Support for Airbyte v2.0.x Based on the official documentation, here's what Airbyte supports for self-managed deployments: Officially Supported Distributions For Self-Managed Enterprise (v2.0.x), Airbyte supports: ā€¢ Amazon EKS on EC2 ā€¢ Google Kubernetes Engine (GKE) on Google Compute Engine ā€¢ Azure Kubernetes Service (AKS) GKE Autopilot is supported, but Amazon EKS on Fargate is explicitly not supported. [v2.0 Enterprise Prerequisites] For non-Enterprise deployments, the documentation mentions that Airbyte has been tested with kind, k3s, and colima for local/single-node setups, and also works on single VMs (e.g., EC2 or GCE) with Kubernetes installed. [v2.0 Deploying Airbyte] K3s on a Single EC2 Node K3s is mentioned as a tested environment for local/development deployments, but it is not listed as an officially supported distribution for production. The documentation does not provide guidance on K3s for production use cases or troubleshooting K3s-specific issues. Regarding Your CONNECTION_UPDATER Issue The knowledge sources do not contain enough information to diagnose your specific issue with 
ConnectionManagerWorkflow
tasks on the 
CONNECTION_UPDATER
queue not being picked up by the worker. There is a related open GitHub issue describing a similar symptom (Temporal workflow tasks stuck/not started), but it does not provide a resolution. [GitHub Issue] Recommendation If you need a production-grade, officially supported setup on AWS, Airbyte recommends using Amazon EKS on EC2 (minimum 6 nodes, 2+ availability zones, memory-optimized instances like M7i/M7g with at least 2 cores and 8 GB RAM). [v2.0 Enterprise Prerequisites] For your specific Temporal/worker issue on K3s, I'd recommend reaching out to the Airbyte GitHub issues or community forums, as the knowledge sources don't contain enough detail to confirm whether K3s is the root cause or provide a fix. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
w
@kapa.ai After fresh Airbyte v2.0.x install, ConnectionManagerWorkflow is created in Temporal but WorkflowTaskStarted never fires. Source checks and destination checks work fine. Connection creation works. Only 'Sync now' fails. We fixed the dataplane group mismatch. What else in the Airbyte database needs to be configured correctly for ConnectionManagerWorkflow to process?
k
Based on the knowledge sources available, I can share what's documented about database configuration issues affecting sync workflows after fixing the dataplane group mismatch. What the Knowledge Sources Cover The knowledge sources focus primarily on the dataplane group mismatch as the root cause of syncs being stuck. Beyond that fix, the documented database elements relevant to 
ConnectionManagerWorkflow
processing are: 1. Dataplane Group Mismatch (Already Fixed) Ensure the 
workspace.dataplane_group_id
matches the 
dataplane.dataplane_group_id
that the workload-launcher is registered under. You can verify with: [Connector-Sidecar Issue]
Copy code
SELECT w.id AS workspace_id,
       w.dataplane_group_id AS workspace_group,
       d.id AS dataplane_id,
       d.dataplane_group_id AS dataplane_group,
       d.enabled
FROM workspace w
JOIN dataplane d ON d.enabled = true
WHERE w.dataplane_group_id <> d.dataplane_group_id;
Result should be zero rows. 2. Stale Temporal Workflows (NonDeterministicException) A separate contributing factor documented is stale Temporal 
ConnectionManagerWorkflow
instances carried over from previous versions causing 
NonDeterministicException
errors. These need to be terminated separately via 
tctl
. [Dataplane Group Mismatch Issue] 3. Workload Status Deadlock There is a documented scenario where workloads get stuck in 
launched
status permanently, preventing new claims. You can check: [Services Stuck Issue]
Copy code
SELECT id, status, termination_reason 
FROM workload 
WHERE status NOT IN ('success', 'failure', 'cancelled');
Limitation The knowledge sources do not contain enough information to provide a comprehensive checklist of all database configurations required for 
ConnectionManagerWorkflow
to process 
Sync now
specifically (as opposed to source/destination checks). Your symptom ā€” where checks work but syncs don't ā€” is not explicitly diagnosed beyond the dataplane group mismatch in the available documentation. I'd recommend checking the Airbyte Community Slack or filing a GitHub issue with your specific logs, as this appears to be a nuanced case beyond what the available knowledge sources cover. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
The knowledge sources don't contain enough information to give a definitive answer about all database configurations required for 
ConnectionManagerWorkflow
to process correctly beyond the dataplane group mismatch. However, based on what the knowledge sources do cover, here are the confirmed database-level issues that can cause syncs to fail while other operations (source/destination checks) continue to work: Confirmed Database Issues After v2.0.x Install/Upgrade 1. Dataplane Group Mismatch (Already Fixed) You've addressed this, but for completeness: workloads get routed to the wrong 
dataplane_group_id
, causing them to never be claimed. [Dataplane Group Mismatch Issue] 2. Workspace dataplane_group_id Mismatch Even after fixing the 
dataplane
table, the workspace table also needs to reference the correct 
dataplane_group_id
. Jobs are dispatched based on the workspace's group assignment, so if the workspace still points to the old/wrong group, syncs will still fail:
Copy code
-- Check workspace group assignment
SELECT id, organization_id, dataplane_group_id FROM public.workspace;

-- Verify it matches the active dataplane group
SELECT * FROM public.dataplane_group;
SELECT * FROM public.dataplane WHERE enabled = true;
The workspace's 
dataplane_group_id
must match the group that the active, enabled dataplane belongs to. [Services Stuck Issue; Dataplane Group Mismatch Issue] 3. Workloads Stuck in "Launched" Status A separate deadlock condition can occur where workloads remain in 
launched
status in the 
workload
table indefinitely, preventing new claims:
Copy code
SELECT id, status, termination_reason FROM workload WHERE status = 'pending' OR status = 'launched';
If workloads are stuck in 
launched
, the launcher cannot claim them again, and 
CHECK_STATUS
never runs to update them ā€” creating a deadlock. [Services Stuck Issue] 4. Stale Temporal Workflows (NonDeterministicException) The knowledge sources also note that stale 
ConnectionManagerWorkflow
instances carried over from previous versions can cause 
NonDeterministicException
errors in the worker, which may prevent workflow tasks from being processed. These need to be terminated via 
tctl
. [Dataplane Group Mismatch Issue] Beyond these points, the knowledge sources do not provide further detail on additional database configuration requirements specific to 
ConnectionManagerWorkflow
processing. I'd recommend checking the Airbyte Community Slack or filing a GitHub issue for deeper investigation. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
The knowledge sources I have available focus primarily on the dataplane group mismatch issue and do not contain information specific to why 
ConnectionManagerWorkflow
tasks on the 
CONNECTION_UPDATER
queue would fail while source/destination checks succeed after the dataplane fix is applied. Based strictly on what the knowledge sources document, here are the confirmed database-level checks relevant to your situation: Verified Database Checks 1. Workspace Table dataplane_group_id Even after fixing the 
dataplane
table, the workspace must also reference the correct group. Syncs are dispatched based on the workspace's group assignment:
Copy code
-- Verify workspace points to the correct active group
SELECT id, organization_id, dataplane_group_id FROM public.workspace;
SELECT * FROM public.dataplane_group;
SELECT * FROM public.dataplane WHERE enabled = true;
All three should reference the same 
dataplane_group_id
. [Services Stuck Issue; Dataplane Group Mismatch Issue] 2. Workloads Stuck in "Launched" or "Pending" Status Check for a deadlock where workloads never reach a terminal state:
Copy code
SELECT id, status, termination_reason FROM workload WHERE status IN ('pending', 'launched');
If workloads are stuck in 
launched
, the launcher cannot claim them again, 
CHECK_STATUS
never runs, and the system deadlocks. [Services Stuck Issue] 3. Stale Temporal Workflows Stale 
ConnectionManagerWorkflow
instances from previous versions can cause 
NonDeterministicException
errors in the worker, preventing workflow tasks from being processed. These need to be terminated via 
tctl
. [Dataplane Group Mismatch Issue] Beyond these points, the knowledge sources do not contain enough information to explain why 
ConnectionManagerWorkflow
specifically would fail while check operations succeed on a fresh install after the dataplane fix. Your symptom is a more specific case than what is documented here. I'd recommend joining the Airbyte Community Slack or filing a GitHub issue with your Temporal and worker logs for deeper investigation. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
The knowledge sources I have available do not contain information that goes beyond what I've already shared in our conversation. I've now answered this same question four times, and the knowledge sources simply don't document additional database configuration requirements specific to 
ConnectionManagerWorkflow
processing on the 
CONNECTION_UPDATER
queue beyond what's already been covered. To summarize what the knowledge sources confirm: 1. Workspace 
dataplane_group_id
 must match the active dataplane's group ā€” verify with the query from my previous responses. [Services Stuck Issue; Dataplane Group Mismatch Issue] 2. Workloads stuck in `pending`/`launched` status can deadlock the system. [Services Stuck Issue] 3. Stale Temporal workflows causing 
NonDeterministicException
need to be terminated via 
tctl
. [Dataplane Group Mismatch Issue] Your specific symptom ā€” 
WorkflowTaskStarted
never firing on 
CONNECTION_UPDATER
while 
CHECK_CONNECTION
works fine on a fresh install ā€” is not diagnosed in the available knowledge sources beyond these points. For a resolution, I'd strongly recommend: ā€¢ Posting in the Airbyte Community Slack with your Temporal and worker logs ā€¢ Opening a GitHub issue at airbytehq/airbyte with full reproduction details šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
2 Views
Open in Slack
PreviousNext
Powered by