https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> I upgraded the airbyte version from...
# replication-ask-ai
a
@kapa.ai I upgraded the airbyte version from 1.6.0 to 1.7.1 but the bootloader pod is failing
@kapa.ai here are the error details
Copy code
2026-04-06 06:12:43,426 [main]	ERROR	i.a.b.ApplicationKt(main):28 - Unable to bootstrap Airbyte environment.
io.fabric8.kubernetes.client.KubernetesClientException: The timeout of 10000 ms has been exceeded when getting a connection to 172.20.0.1:443
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:509)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleCreate(OperationSupport.java:340)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:754)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:98)
	at io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:42)
	at io.airbyte.bootloader.AuthKubernetesSecretInitializer.checkAccessToSecrets(AuthKubernetesSecretInitializer.kt:56)
	at io.airbyte.bootloader.Bootloader.load(Bootloader.kt:75)
	at io.airbyte.bootloader.ApplicationKt.main(Application.kt:25)
Caused by: io.vertx.core.impl.NoStackTraceTimeoutException: The timeout of 10000 ms has been exceeded when getting a connection to 172.20.0.1:443
k
The error you're seeing is a timeout when the bootloader tries to connect to the Kubernetes API server (172.20.0.1:443) to check/create the 
airbyte-auth-secrets
secret. This is happening in 
AuthKubernetesSecretInitializer.checkAccessToSecrets
, which was introduced as part of the 1.6 breaking change requiring service account permissions to include access to the 
secrets
resource. There are two likely causes: 1. Missing Service Account Permissions (Most Common) Starting with Airbyte 1.6, the service account must have access to the 
secrets
resource. If you're upgrading from 1.5.x and haven't updated permissions, the bootloader will fail. [upgrade service account] If using Airbyte's default service account (
airbyte-sa
), run:
Copy code
kubectl -n <namespace> patch role airbyte-admin-role --type='json' -p='[{"op": "replace", "path": "/rules/0/resources", "value": ["jobs", "pods", "pods/log", "pods/exec", "pods/attach", "secrets"]}]'
If using your own service account, update the role to include 
secrets
in the resources:
Copy code
rules:
  - apiGroups: ["*"]
    resources: ["jobs", "pods", "pods/log", "pods/exec", "pods/attach", "secrets"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
Then retry the upgrade. [upgrade service account] 2. Kubernetes API Server Connectivity / Timeout Issue The timeout (
10000ms exceeded when getting a connection to 172.20.0.1:443
) can also be caused by the Kubernetes API server being slow or unreachable from the bootloader pod. A similar issue was reported on EKS where the bootloader intermittently timed out while executing a 
PUT
against the secrets API. [EKS install issue] In that case, the issue was eventually traced to Kong's admission webhook interfering with secret operations. The fix was:
Copy code
admissionWebhook:
  matchPolicy: Equivalent
  enabled: true
  filterSecrets: true
Recommended steps: 1. First, verify and update service account permissions as described above. 2. Check API server health and network connectivity from the bootloader pod. 3. If you have Kong or other admission webhooks, check if they are intercepting secret operations. 4. If the error is intermittent, retrying the install/upgrade has been reported to eventually succeed. [EKS install issue] šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
The error in your logs shows the bootloader is timing out trying to reach the Kubernetes API server (
172.20.0.1:443
) when attempting to create/check access to secrets:
Copy code
The timeout of 10000 ms has been exceeded when getting a connection to 172.20.0.1:443
This is similar to a known issue where the bootloader fails while trying to interact with Kubernetes secrets. [EKS install issue] Here are the most likely causes and steps to investigate: 1. Service Account Permissions (Most Likely for 1.6+ Upgrades) Airbyte 1.6 introduced a breaking change requiring the service account to have access to the 
secrets
resource. If you're upgrading from 1.5.x, this is the most probable cause. [upgrade service account] If using Airbyte's default service account (
airbyte-sa
), run:
Copy code
kubectl -n <namespace> patch role airbyte-admin-role --type='json' -p='[{"op": "replace", "path": "/rules/0/resources", "value": ["jobs", "pods", "pods/log", "pods/exec", "pods/attach", "secrets"]}]'
If using your own service account, update the role to include `secrets`:
Copy code
rules:
  - apiGroups: ["*"]
    resources: ["jobs", "pods", "pods/log", "pods/exec", "pods/attach", "secrets"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
2. Network/Connectivity Issue to Kubernetes API The timeout to 
172.20.0.1:443
(the Kubernetes API server) suggests a network policy or firewall may be blocking the bootloader pod from reaching the API server. Check: ā€¢ Network policies in your namespace ā€¢ Security groups (if on AWS EKS) allowing pod-to-API-server traffic 3. Intermittent Issue ā€” Retry In some cases, this error has been reported as intermittent. One user on AWS EKS resolved it by configuring Kong's admission webhook correctly. [EKS install issue] Retrying the installation/upgrade has also worked for some users. [bootloader hang issue] Start by checking your service account permissions, as that is the documented breaking change introduced in 1.6 that directly causes bootloader failures during upgrades. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai FYi the version was recently downgraded from 1.7.0 to 1.6.0 and didnt had this issue then, also the service account is custom but it already has the all the permissions mentioned above
k
Given that: ā€¢ Your custom service account already has the required 
secrets
permissions ā€¢ You previously ran 1.6.0 without issues (downgraded from 1.7.0 to 1.6.0 successfully) ā€¢ The error is a timeout connecting to the Kubernetes API server (not a permissions denial) This points more strongly to the Kubernetes API server connectivity/timeout issue rather than a permissions problem. Based on the knowledge sources, a very similar timeout error was reported on AWS EKS with Helm 1.6.2:
Copy code
The timeout period of 10000ms has been exceeded while executing PUT /api/v1/namespaces/airbyte/secrets/airbyte-auth-secrets
In that case, the root cause was Kong's admission webhook intercepting secret operations. The fix was configuring Kong's webhook: [EKS install issue]
Copy code
admissionWebhook:
  matchPolicy: Equivalent
  enabled: true
  filterSecrets: true
Suggested next steps: 1. Check if you have Kong or any other admission webhooks in your cluster that might be intercepting requests to the Kubernetes secrets API. 2. Check network policies ā€” ensure the bootloader pod can reach the Kubernetes API server at 
172.20.0.1:443
without restrictions. 3. Retry the upgrade ā€” the EKS issue report noted that the error was intermittent and eventually succeeded after several retries. [EKS install issue] Beyond these points, the knowledge sources don't contain additional specific information about why this timeout would appear specifically when upgrading from 1.6.0 to 1.7.1 but not on 1.6.0 itself. I'd recommend checking the Airbyte Community Slack for more targeted help. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
2 Views
Open in Slack
PreviousNext
Powered by