https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> Im trying to setup a new source via...
# replication-ask-ai
m
@kapa.ai Im trying to setup a new source via UI but it's not setting up. It takes too long and raises timeout. If I try to create new connections or destinations via UI. The same happens. I am regarding logs but no pods are launched when clicking "Setup" button
k
This is a known issue that can have several root causes. Based on the knowledge sources, here are the most likely causes and fixes: No Pods Launched on Connection/Source Setup 1. Dataplane Group Misconfiguration (Most Common for Kubernetes/Helm v2+) If you upgraded from Airbyte v1 to v2, the 
dataplane_group_id
on your workspace may be pointing to a non-existent or wrong dataplane group. This causes connection tests and checks to hang silently with no pods being scheduled. [GitHub Issue] Fix option A ā€” Add this to your Helm 
values.yaml
before upgrading:
Copy code
global:
  dataplaneGroups:
    defaultDataplaneGroupName: "AUTO"
Fix option B ā€” Update the 
dataplane_group_id
in the database directly:
Copy code
-- Check available dataplane groups
SELECT * FROM public.dataplane_group;

-- Update workspace to point to the correct enabled dataplane group
UPDATE public.workspace SET dataplane_group_id = '<correct_id>';
[GitHub Issue] 2. Schema Discovery Timeout If you have a source with many tables/columns, the schema discovery process can exceed Airbyte's default timeout. You can increase it in `values.yaml`: [Scaling Airbyte]
Copy code
server:
  httpIdleTimeout: 20m
  extraEnv:
    - name: READ_TIMEOUT
      value: 30m
3. Temporal Service Connectivity Issues The worker may be unable to communicate with the Temporal service, causing jobs to never start. Check your worker and Temporal pod logs for errors like:
Copy code
Connection refused: airbyte-temporal-svc/...:7233
[Forum Post] 4. RBAC / Pod Creation Permissions (Kubernetes) If running on Kubernetes, the workload launcher may lack permissions to create pods. Ensure your RBAC role includes 
create
, 
update
, 
patch
, 
delete
on 
pods
resources. [GitHub Issue] Recommended Debugging Steps 1. Check the workload-launcher and worker pod logs for errors. 2. Check the cron pod logs ā€” it monitors workloads and may show cancellation reasons. 3. Verify your Airbyte version and whether you recently upgraded from v1 to v2. 4. Confirm the 
dataplane_group
table in your database has an enabled entry matching your workspace. If you share your Airbyte version and deployment method (Docker Compose, Helm, abctl), it would help narrow down the cause further. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
m
@kapa.ai We have deployed with this command:
Copy code
abctl local install --values airbyte_values_prod.yaml --secret airbyte_secrets_prod.yml --chart-version 1.9.2
k
Based on your deployment details, you're using abctl with chart version 1.9.2 (Helm v1 series). This matches a known issue reported by multiple users. Most Likely Cause: Dataplane Group Mismatch A user in the knowledge sources specifically reported this exact problem on chart version 1.9.2 (helm_v1 2.01): > "Check your `dataplane_group_id`'s in the 
dataplane
and 
workspace
tables. Make sure they match! No sidecar job (connector check or operation) would launch. I could see a job create line in the logs but nothing regarding the failure. No job pod was ever launched." [GitHub Issue] Diagnostic Query Run this SQL against your Airbyte database to check for a mismatch:
Copy code
SELECT w.id AS workspace_id,
       w.dataplane_group_id AS workspace_group,
       d.id AS dataplane_id,
       d.dataplane_group_id AS dataplane_group,
       d.enabled
FROM workspace w
JOIN dataplane d ON d.enabled = true
WHERE w.dataplane_group_id <> d.dataplane_group_id;
The result should return zero rows. If it returns any rows, there is a mismatch that needs to be corrected. [GitHub Issue] Fix 1. Identify the correct 
dataplane_group_id
from the 
dataplane
table where 
enabled = true
2. Update the workspace to use that ID:
Copy code
UPDATE public.workspace
SET dataplane_group_id = '<correct_dataplane_group_id>';
3. Restart the workload launcher:
Copy code
kubectl rollout restart deployment/airbyte-abctl-workload-launcher -n airbyte-abctl
After the restart, you should see pods being launched when setting up sources/destinations. [GitHub Issue] > Note: I should be transparent that the knowledge sources don't contain exhaustive details specific to chart version 1.9.2. If the above fix doesn't resolve the issue, I'd recommend checking the Airbyte Community Slack for further assistance. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
m
What should I do exactly ? I have this row for example: 3ac09613-99b1-482e-a097-8bc3c9c794da | dcfc7b69-5d93-4b7a-841e-cd50e3b067e5 | a65d890e-6128-463d-ba91-8df332ff1011 | e6b168c7-7a3a-4ce6-8c67-bd07ae64e36f | t @kapa.ai
k
Based on your output, the columns map as follows: Your workspace is pointing to 
dcfc7b69-...
but the active dataplane is registered under 
e6b168c7-...
. You need to update the workspace to match the active dataplane group. Steps 1. Update the workspace's dataplane group:
Copy code
UPDATE public.workspace
SET dataplane_group_id = 'e6b168c7-7a3a-4ce6-8c67-bd07ae64e36f'
WHERE id = '3ac09613-99b1-482e-a097-8bc3c9c794da';
2. Verify the fix (should now return zero rows):
Copy code
SELECT w.id AS workspace_id,
       w.dataplane_group_id AS workspace_group,
       d.id AS dataplane_id,
       d.dataplane_group_id AS dataplane_group,
       d.enabled
FROM workspace w
JOIN dataplane d ON d.enabled = true
WHERE w.dataplane_group_id <https://github.com/airbytehq/airbyte/issues/70451|GitHub Issue> d.dataplane_group_id;
3. Restart the workload launcher:
Copy code
kubectl rollout restart deployment/airbyte-abctl-workload-launcher -n airbyte-abctl
After the restart, try setting up a source again ā€” pods should now be launched correctly. [āš ļø] > šŸ‘ Important: Take a database backup before running any 
UPDATE
statements. šŸ‘Ž 0 button ā„¹ļø 0 button Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
m
@kapa.ai I am getting this error while restarting: kubectl rollout restart deployment/airbyte-abctl-workload-launcher -n airbyte-abctl E0728 143218.905038 4101367 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"127.0.0.1/api?timeout=32s\": dial tcp 127.0.0.136757 connect: connection refused" E0728 143218.906726 4101367 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"127.0.0.1/api?timeout=32s\": dial tcp 127.0.0.136757 connect: connection refused" E0728 143218.908242 4101367 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"127.0.0.1/api?timeout=32s\": dial tcp 127.0.0.136757 connect: connection refused" E0728 143218.909839 4101367 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"127.0.0.1/api?timeout=32s\": dial tcp 127.0.0.136757 connect: connection refused" The connection to the server 127.0.0.1:36757 was refused - did you specify the right host or port?
k
The error means 
kubectl
is not pointing to the correct Kubernetes cluster. Since you deployed with 
abctl
, the cluster uses a custom kubeconfig file located at 
~/.airbyte/abctl/abctl.kubeconfig
. Fix: Use the correct kubeconfig Run the restart command with the explicit kubeconfig path:
Copy code
kubectl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig rollout restart deployment/airbyte-abctl-workload-launcher -n airbyte-abctl
Or alternatively, export it for your current session so all 
kubectl
commands work correctly:
Copy code
export KUBECONFIG=~/.airbyte/abctl/abctl.kubeconfig
kubectl rollout restart deployment/airbyte-abctl-workload-launcher -n airbyte-abctl
You can verify it's working by listing the pods first:
Copy code
kubectl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig get pods -n airbyte-abctl
[abctl FAQ] šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
m
@kapa.ai Why is this happening? We haven't restarted anything, and the affected Workspace is brand new.
k
Based on the knowledge sources, this is a known bug that affects both upgrades and fresh installs. A user reported the exact same situation ā€” a brand new workspace on a clean install: > "I am also experiencing this issue on a fresh install of Airbyte 2.0.1 via abctl (not an upgrade). On a clean 
abctl local install
, the bootloader creates two 
dataplane_group
rows ā€” workloads are routed to the 
AUTO
group, but the active dataplane registers under the unnamed group." [GitHub Issue] Root Cause The bootloader creates two 
dataplane_group
entries on every install: ā€¢ New workspaces get assigned to the 
AUTO
 group ā€¢ But the workload-launcher registers under the unnamed group ā€¢ Since they don't match, the launcher never picks up the workload ā†’ no pods are ever created [GitHub Issue] Important Note The database fix you already applied (updating 
dataplane_group_id
on the workspace) is the correct workaround. However, be aware that: > "This fix needs to be reapplied after every 
abctl local install
run as the bootloader recreates the mismatch on each startup." [GitHub Issue] This is an open bug that has been escalated to the Airbyte internal team but does not yet have a permanent fix in the versions reported. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
m
@kapa.ai Right now, I create the workspaces via API like this: { "name": "The Bath" } Should I include other params in body JSON ?
k
Based on the knowledge sources, yes ā€” you should include the 
dataResidency
parameter in your workspace creation request to explicitly assign it to the correct dataplane group from the start. Recommended Workspace Creation Body
Copy code
{
  "name": "The Bath",
  "dataResidency": "auto"
}
The 
dataResidency
field maps to the dataplane group name (
AUTO
) that the workload-launcher is registered under. [Enterprise Setup Docs] Why This Matters Without specifying 
dataResidency
, the bootloader may assign your new workspace to the wrong dataplane group by default ā€” which is exactly the bug you've been hitting. Explicitly setting it to 
"auto"
ensures the workspace is routed to the correct 
AUTO
dataplane group. [GitHub Issue] Full API Call Example
Copy code
curl -X POST "https://<your-airbyte-host>/api/public/v1/workspaces" \
  --header "Authorization: Bearer $TOKEN" \
  --header "Content-Type: application/json" \
  -d '{
    "name": "The Bath",
    "dataResidency": "auto"
  }'
[Enterprise Setup Docs] > āš ļø I should be transparent that the knowledge sources don't explicitly confirm whether adding 
dataResidency: "auto"
fully prevents the mismatch bug in your specific version (chart 1.9.2). You may still need to run the diagnostic SQL query after creating new workspaces to verify the 
dataplane_group_id
is correctly assigned. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
2 Views
Open in Slack
PreviousNext
Powered by