<@U04S8PLJWMT> after updating to airbyte 2.0.1, I ...
# ask-ai
e
@kapa.ai after updating to airbyte 2.0.1, I started getting timeout errors when upgrading my source and destination connector versions. What could be happening?
k
After updating to Airbyte 2.0.1, if you are experiencing timeout errors when upgrading your source and destination connector versions, there are a few possible causes and solutions based on the knowledge sources: 1. Schema Discovery Timeouts: When configuring connectors—especially with sources that have hundreds or thousands of tables and columns—the schema discovery process (
discover
mechanism) can take a long time and may exceed Airbyte's default timeout settings. If this is the case, you should increase the timeout limits in your Airbyte deployment configuration. For Helm-based deployments, you can adjust the
HTTP_IDLE_TIMEOUT
and
READ_TIMEOUT
values in your
values.yaml
file as shown below:
Copy code
server:        extraEnv:          - name: HTTP_IDLE_TIMEOUT            value: 20m          - name: READ_TIMEOUT            value: 30m
Or for Helm chart V2:
Copy code
server:        httpIdleTimeout: 20m        extraEnv:          - name: READ_TIMEOUT            value: 30m
This should help if the timeouts are occurring during schema discovery or connector upgrades involving large or complex sources and destinations [Scaling Airbyte After Installation](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#schema-discovery-timeouts). 2. Connector Version Compatibility: Timeout or other errors can also occur if there are compatibility issues between the Airbyte platform version and the connector versions. After a major upgrade, it is important to ensure that your connectors are updated to versions compatible with your Airbyte instance. If you are running into issues, try updating your connectors to the latest available versions, as fixes for such issues are often included in newer releases [Upgrade to java-connector-base image 2.0.1](https://github.com/airbytehq/airbyte/issues/54706). 3. Resource Constraints: Timeout errors can also be related to insufficient resources (CPU, memory) allocated to Airbyte services, especially after an upgrade. Review your deployment's resource limits and consider increasing them if you are working with large datasets or complex syncs [Salesforce connections stuck and subsequent workflow runs failing](https://discuss.airbyte.io/t/1612). If you have already increased timeouts and updated connectors but still experience issues, please provide more details or logs for further troubleshooting.
e
Do I need all that just to bump my source connector version? @kapa.ai
k
A message from kapa.ai
No, you do not always need to adjust timeouts or resources just to bump your source connector version. The standard process for upgrading a connector version in Airbyte is typically straightforward: 1. Review the changelog for the connector to understand what has changed and if any action is required on your part. 2. Opt-in to the update via the settings page in Airbyte. For major version updates, you may need to accept the update for each individual connection, allowing you to review and adjust settings if needed. 3. Follow any migration guide provided for the connector, especially for major version changes, as these may require additional steps such as refreshing schemas or clearing data for affected streams. Timeout or resource adjustments are only necessary if you encounter specific issues (like timeouts during schema discovery or syncs), which is not the default expectation for a simple version bump. Most users can upgrade connectors without changing deployment settings unless they run into performance-related errors during the process. If you do not experience such issues, you can proceed with the upgrade as described in the documentation without additional configuration changes [Managing Connector Updates in Airbyte](https://docs.airbyte.com/platform/managing-airbyte/connector-updates#actions-to-take-in-response-to-connector-updates). If you do encounter errors (such as timeouts), then you may need to consider the troubleshooting steps mentioned earlier. Otherwise, the upgrade process should be routine.
e
why does this happen? ERROR i.a.w.l.StartupApplicationEventListener(onApplicationEvent$lambda$1):43 - Failed to retrieve and resume claimed workloads, exiting. │ │ io.airbyte.api.client.ApiException: Unauthorized │ │ at io.airbyte.workload.api.client.WorkloadApiClient.workloadList(WorkloadApiClient.kt:288) │ │ at io.airbyte.workload.launcher.ClaimedProcessor.getWorkloadList$lambda$9(ClaimedProcessor.kt:112) │ │ at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243) │ │ at dev.failsafe.Functions.lambda$get$0(Functions.java:46) │ │ at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74) │ │ at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187) │ │ at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376) │ │ at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112) │ │ at io.airbyte.workload.launcher.ClaimedProcessor.getWorkloadList(ClaimedProcessor.kt:110) │ │ at io.airbyte.workload.launcher.ClaimedProcessor.retrieveAndProcess(ClaimedProcessor.kt:57) │ │ at io.airbyte.workload.launcher.StartupApplicationEventListener.onApplicationEvent$lambda$1(StartupApplicationEventListener.kt:39) @kapa.ai
k
A message from kapa.ai
The error
Copy code
io.airbyte.api.client.ApiException: Unauthorized
when the workload launcher tries to retrieve and resume claimed workloads typically means there is an authentication issue between the Airbyte workload launcher and the Workload API server. According to the knowledge sources, this often happens when the authentication token (WORKLOAD_API_BEARER_TOKEN) or the header (WORKLOAD_API_AUTH_HEADER) is not properly set or is mismatched between the relevant Airbyte components. For example, if the worker and the workload API server are using different tokens, or if the token is missing or expired, you will see "Unauthorized" errors like this one. The solution is to ensure that the same token and header are configured for all relevant deployments and then restart them. Here’s a relevant workaround from a user: > The issue is because Workload API calls were rejected with 401 (error below) - WORKLOAD_API_BEARER_TOKEN (and header) wasn’t aligned between the Worker and the Workload API Server pods. > ... > Set the same auth header and bearer token on all three deployments, then restart: > - Create a token first - TOKEN=$(openssl rand -hex 24) > - Set the environment variables for all relevant deployments to use this token > [source](https://github.com/airbytehq/airbyte/issues/65567) So, this error is most likely due to a misconfiguration or mismatch of authentication credentials between Airbyte services. Double-check your environment variables and ensure all components use the same token and header.
e
Failed to retrieve and resume claimed workloads, exiting. @kapa.ai
k
A message from kapa.ai
The error message "Failed to retrieve and resume claimed workloads, exiting." in Airbyte is typically associated with authentication issues between the workload launcher and the Workload API server. Specifically, this often occurs when the authentication token (such as
WORKLOAD_API_BEARER_TOKEN
) or the authentication header is not properly set or is mismatched between the relevant Airbyte components. When these tokens or headers do not match, the API calls are rejected with an "Unauthorized" error, preventing the workload launcher from retrieving workloads and causing it to exit. To resolve this, ensure that the same authentication token and header are configured for all relevant Airbyte deployments (such as the worker and the workload API server), and then restart the affected services. This approach has been confirmed as a solution in user reports: > "The issue is because Workload API calls were rejected with 401... WORKLOAD_API_BEARER_TOKEN (and header) wasn’t aligned between the Worker and the Workload API Server pods. Set the same auth header and bearer token on all three deployments, then restart." [See details here](https://github.com/airbytehq/airbyte/issues/65567) If you continue to see this error after aligning the tokens and headers, double-check your environment variable configuration for typos or mismatches.
e
how can I ensure this? I'm deploying airbyte 2.0.1 using helm chart v1 1.9.1 @kapa.ai
k
A message from kapa.ai
To ensure that the
WORKLOAD_API_BEARER_TOKEN
and authentication header are properly aligned across your Airbyte components when deploying Airbyte 2.0.1 with Helm chart v1 (1.9.1), you need to set the same token and header values for all relevant deployments (such as the worker and the workload API server). Based on the knowledge sources, here’s how you can do it: 1. Generate a token (for example, using OpenSSL):
Copy code
bash    TOKEN=$(openssl rand -hex 24)
2. Set the environment variables for both the worker and the workload API server deployments to use this token and the same header (commonly
Authorization
):
Copy code
bash    kubectl -n <your-namespace> set env deployment/airbyte-worker WORKLOAD_API_AUTH_HEADER=Authorization WORKLOAD_API_BEARER_TOKEN=$TOKEN    kubectl -n <your-namespace> set env deployment/airbyte-workload-api-server WORKLOAD_API_AUTH_HEADER=Authorization WORKLOAD_API_BEARER_TOKEN=$TOKEN
Replace
<your-namespace>
with the namespace where Airbyte is deployed (commonly
airbyte
). 3. Restart the deployments to apply the changes:
Copy code
bash    kubectl -n <your-namespace> rollout restart deployment/airbyte-worker    kubectl -n <your-namespace> rollout restart deployment/airbyte-workload-api-server
This process ensures that both services use the same authentication credentials, which should resolve the "Failed to retrieve and resume claimed workloads, exiting." error due to unauthorized API calls [see details](https://github.com/airbytehq/airbyte/issues/65567). Note: The knowledge sources do not provide a Helm-native way (e.g., via
values.yaml
) to set these specific environment variables for Helm chart v1. The workaround above uses
kubectl set env
after deployment, which is a common approach referenced in user reports. If you continue to experience issues, double-check for typos and ensure all relevant pods are restarted. If your deployment uses different pod or deployment names, adjust the commands accordingly.
e
but why do I have to set all that up? @kapa.ai I need the minimum solution to this
k
A message from kapa.ai
You have to set up the
WORKLOAD_API_BEARER_TOKEN
and
WORKLOAD_API_AUTH_HEADER
manually because, in Airbyte 2.0.1 with Helm chart v1, there is a known issue where these authentication values are not automatically aligned between the worker and workload API server deployments. If these values are not the same, the services cannot authenticate with each other, resulting in "Unauthorized" errors and failures to resume workloads. The minimum solution is: 1. Generate a token (if you don't already have one):
Copy code
bash    TOKEN=$(openssl rand -hex 24)
2. Set the same token and header on both deployments:
Copy code
bash    kubectl -n <your-namespace> set env deployment/airbyte-worker WORKLOAD_API_AUTH_HEADER=Authorization WORKLOAD_API_BEARER_TOKEN=$TOKEN    kubectl -n <your-namespace> set env deployment/airbyte-workload-api-server WORKLOAD_API_AUTH_HEADER=Authorization WORKLOAD_API_BEARER_TOKEN=$TOKEN
3. Restart both deployments:
Copy code
bash    kubectl -n <your-namespace> rollout restart deployment/airbyte-worker    kubectl -n <your-namespace> rollout restart deployment/airbyte-workload-api-server
This manual step is necessary due to the way authentication is handled in this version and chart combination. If you skip it, the services will not be able to communicate securely, leading to the errors you are seeing [as described here](https://github.com/airbytehq/airbyte/issues/65567). There is no simpler solution documented in the knowledge sources for Helm chart v1 and Airbyte 2.0.1.
e
@kapa.ai I cant run kubectl commands. All changes I do I need to do to my values.yml
k
A message from kapa.ai
Thank you for clarifying that you cannot use
kubectl
and need to make all changes via your
values.yaml
. Based on the knowledge sources, for Helm chart V2, there are clear mappings for setting environment variables like
WORKLOAD_API_BEARER_TOKEN
and
WORKLOAD_API_AUTH_HEADER
directly in
values.yaml
(for example, using
global.api.authHeaderName
and
global.api.authHeaderValue
). However, for Helm chart V1, the knowledge sources do not explicitly document a way to set these specific environment variables directly in
values.yaml
for the worker and workload API server. The sources provide full references for
values.yaml
for both chart versions, but the relevant environment variable mappings are only detailed for chart V2, not V1. For V1, the workaround shown in the sources is to use
kubectl set env
after deployment, which you have stated is not possible in your environment [see source](https://github.com/airbytehq/airbyte/issues/65567). Summary: I'm sorry, but the knowledge sources do not contain enough information to confirm a supported way to set
WORKLOAD_API_BEARER_TOKEN
and
WORKLOAD_API_AUTH_HEADER
via
values.yaml
in Helm chart V1. The documented solutions rely on post-deployment
kubectl
commands, and there is no explicit documentation for configuring these values in
values.yaml
for chart V1. If you are able to upgrade to Helm chart V2, you will have more flexibility and documented support for these settings in
values.yaml
 [as shown here](https://docs.airbyte.com/platform/next/deploying-airbyte/chart-v2-community#update-your-valuesyaml-file). For Helm chart V1, the knowledge sources do not provide a minimum solution that does not involve
kubectl
.
e
should I update the dataplane entries on the db? @kapa.ai
k
A message from kapa.ai
You should generally not manually update the dataplane entries in the database unless you have a specific issue that is directly related to dataplane configuration and you understand the implications. The knowledge sources show that some users have encountered issues after upgrading (such as jobs not being launched or being stuck) due to mismatches or duplicate entries in the
dataplane_group
table. In those cases, a workaround was to update the
dataplane_group_id
in the
workspace
table to match the correct dataplane group, but this is not the standard or recommended approach for most users. Manual database changes can lead to further issues if not done carefully. Only consider this if you are experiencing a known issue (like jobs being assigned to the wrong dataplane group after an upgrade) and you have verified that this is the root cause. Even then, proceed with caution and back up your database first. Here’s a relevant user experience: > "After investigation, we found out looking inside the airbyte database and running
SELECT * FROM public.dataplane_group
that 2 dataplane_group were existing (1 from before the upgrade to 2.0, 1 after the upgrade) and the jobs were being pushed to dataplane A, but the workload launcher was listening to dataplane_group B. A workaround was to update the right dataplane_group_id in the workspace table." [See discussion](https://github.com/airbytehq/airbyte/issues/68115) In summary: Only update dataplane entries in the database if you are facing a specific, documented issue related to dataplane groups after an upgrade, and you are confident this is the cause. Otherwise, avoid manual DB changes.
e
@kapa.ai how to check if jobs are assigned to the wrong dataplane?
k
A message from kapa.ai
To check if jobs are assigned to the wrong dataplane in Airbyte, you should inspect the relevant tables in your Airbyte database: 1. Check the
dataplane_group
table
to see all existing dataplane groups:
Copy code
sql    SELECT * FROM public.dataplane_group;
2. Check the
workspace
table
to see which
dataplane_group_id
is assigned to each workspace:
Copy code
sql    SELECT id, organization_id, dataplane_group_id FROM public.workspace;
3. Check the
dataplane
table
to see which dataplanes belong to which group:
Copy code
sql    SELECT id, dataplane_group_id, name FROM public.dataplane;
4. Check the
workload
table
to see which
dataplane_id
is handling each workload/job:
Copy code
sql    SELECT id, dataplane_id, status FROM public.workload;
By comparing these tables, you can determine if jobs (workloads) are being assigned to a dataplane that does not match the expected
dataplane_group_id
for your workspace. This approach was used by users who discovered that jobs were being pushed to one dataplane while the workload launcher was listening to another, often after an upgrade that created duplicate or mismatched dataplane groups [as described here](https://github.com/airbytehq/airbyte/issues/68115). If you find mismatches, it may indicate that jobs are assigned to the wrong dataplane.