https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> I attempted to upgrade the Airbyte ...
# replication-ask-ai
m
@kapa.ai I attempted to upgrade the Airbyte version and then reverted it to the previous version. Now, when the bootloader pod starts, it fails with the following error:
Copy code
Search...
	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:66)
	at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:259)
	at io.vertx.core.impl.future.Composition$1.onSuccess(Composition.java:62)
	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:66)
	at io.vertx.core.impl.future.FutureImpl.addListener(FutureImpl.java:231)
	at io.vertx.core.impl.future.Composition.onSuccess(Composition.java:43)
	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:66)
	at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:259)
	at io.vertx.core.Promise.complete(Promise.java:66)
	at io.vertx.core.net.impl.NetClientImpl.lambda$connected$9(NetClientImpl.java:342)
	at io.vertx.core.net.impl.VertxHandler.setConnection(VertxHandler.java:82)
	at io.vertx.core.net.impl.VertxHandler.handlerAdded(VertxHandler.java:88)
	at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:1130)
	at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:558)
	at io.netty.channel.DefaultChannelPipeline.internalAdd(DefaultChannelPipeline.java:203)
	at io.netty.channel.DefaultChannelPipeline.addLast(DefaultChannelPipeline.java:227)
	at io.netty.channel.DefaultChannelPipeline.addLast(DefaultChannelPipeline.java:222)
	at io.vertx.core.net.impl.NetClientImpl.connected(NetClientImpl.java:344)
	at io.vertx.core.net.impl.NetClientImpl.lambda$connectInternal2$3(NetClientImpl.java:306)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:270)
	at io.vertx.core.net.impl.ChannelProvider$1.userEventTriggered(ChannelProvider.java:123)
	at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:398)
	at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376)
	at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:368)
	at io.netty.handler.ssl.SslHandler.setHandshakeSuccess(SslHandler.java:1991)
	at io.netty.handler.ssl.SslHandler.wrapNonAppData(SslHandler.java:1017)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1551)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1378)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1427)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)
2025-12-17T06:59:26.953915929Z
k
This stack trace alone doesn’t appear in the provided knowledge sources, and they don’t contain a concrete, documented fix for this exact bootloader failure pattern. The documents mainly cover other upgrade/rollback problems such as: - Bootloader failures on upgrade with only a generic “bootloader failed” message and advice to re-run with 
--verbose
and inspect full bootloader logs.[[Migration failing](https://github.com/airbytehq/airbyte/issues/55251)] - Bootloader/installation instability that sometimes resolves only after many retries, often related to networking or image-pull issues.[[bootloader hangs](https://github.com/airbytehq/airbyte/issues/58086)] - Version‑specific migration problems (e.g., Slack connector schema, DB migrations), which are resolved by adjusting connectors or following required upgrade paths, but not by simple rollback.[[Upgrade fails](https://discuss.airbyte.io/t/1684); [Upgrade not working 0.29](https://discuss.airbyte.io/t/517)] Because of that, I cannot reliably tell from the knowledge sources: - What Airbyte version(s) you upgraded to and rolled back from - Whether this trace is masking a database migration mismatch, a Temporal/DB connectivity issue, or something else - Any officially documented rollback procedure that guarantees the bootloader will succeed after a failed or partial upgrade The consistent guidance in similar threads is: 1. Get the full bootloader logs at higher verbosity and inspect the first error that mentions migrations, database schema, or connector definitions, not just the Netty/Vert.x stack trace. This is what support repeatedly requested in other migration‑failure cases.[[Upgrade fails](https://discuss.airbyte.io/t/1684); [Migration failing](https://github.com/airbytehq/airbyte/issues/55251)] 2. Avoid relying on rollback alone: some versions require specific upgrade paths and DB migrations that aren’t safely reversible just by swapping images or compose/Helm files.[[Upgrade not working 0.29](https://discuss.airbyte.io/t/517)] 3. If you can share more context, you’ll need at least: - Old Airbyte version - Target version you attempted - Deployment type (Docker / Helm / abctl, external DB or bundled DB) - The earliest clear error line in the bootloader log about migrations, schema, or configuration Without that additional information, the knowledge sources do not contain enough detail to give a specific fix for your current error.
m
@kapa.ai, here are the logs for the bootloader pod:
Copy code
ERROR	i.a.b.ApplicationKt(main):28 - Unable to bootstrap Airbyte environment.io.fabric8.kubernetes.client.KubernetesClientException: The timeout of 10000 ms has been exceeded when getting a connection to 168.40.0.1:443	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:509)	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524)	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleCreate(OperationSupport.java:340)	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:754)	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:98)	at io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:42)	at io.airbyte.bootloader.AuthKubernetesSecretInitializer.checkAccessToSecrets(AuthKubernetesSecretInitializer.kt:56)	at io.airbyte.bootloader.Bootloader.load(Bootloader.kt:74)	at io.airbyte.bootloader.ApplicationKt.main(Application.kt:25)Caused by: io.vertx.core.impl.NoStackTraceTimeoutException: The timeout of 10000 ms has been exceeded when getting a connection to 168.40.0.1:443
k
A message from kapa.ai
The error shows the bootloader is timing out when trying to talk to the Kubernetes API while checking access to secrets:
Copy code
text io.fabric8.kubernetes.client.KubernetesClientException:   The timeout of 10000 ms has been exceeded when getting a connection to 168.40.0.1:443 ... at io.airbyte.bootloader.AuthKubernetesSecretInitializer.checkAccessToSecrets(AuthKubernetesSecretInitializer.kt:56)
In 1.6+ the bootloader always tries to access the 
secrets
resource on the Kubernetes API at startup; if it can’t, it fails the bootstrap.[Update SA](https://docs.airbyte.com/platform/enterprise-setup/upgrade-service-account) However, your specific failure is a network timeout to the API server IP, not a simple “forbidden” RBAC error. From the docs and related issues, there are two things to check: --- ### 1. Confirm the Kubernetes API is reachable from the pod The timeout error matches the open GitHub issue where the bootloader fails with:
Copy code
text The timeout period of 10000ms has been exceeded while executing PUT /api/v1/namespaces/airbyte/secrets/airbyte-auth-secrets for server null Caused by: io.vertx.core.impl.NoStackTraceTimeoutException
which was ultimately due to network restrictions/proxy issues between the pod and the cluster’s API server.[Bootloader timeout](https://github.com/airbytehq/airbyte/issues/60318); [Bootloader hangs](https://github.com/airbytehq/airbyte/issues/58086) Actions (outside of Airbyte, in your cluster): - Check the API server IP (
168.40.0.1
) is actually your cluster’s API endpoint and reachable from worker nodes (security groups, firewalls, proxy, VPN, etc.). - From another pod in the same namespace, try:
Copy code
bash   apk add --no-cache curl   curl -k <https://168.40.0.1:443>
If this hangs or fails, fix cluster networking first. If you’re using a corporate proxy or are in a restricted network (like the Iran example), ensure the Airbyte namespace/pods have proper proxy/NO_PROXY configuration so they can reach the K8s API directly.[Bootloader hangs](https://github.com/airbytehq/airbyte/issues/58086) --- ### 2. Ensure service account has 
secrets
permissions (for 1.6+) Once connectivity is fixed, you must have RBAC for 
secrets
or the bootloader will still fail on this same check.[Update SA](https://docs.airbyte.com/platform/enterprise-setup/upgrade-service-account) - If using Airbyte’s default service account (
airbyte-sa
), run:
Copy code
bash   kubectl -n <namespace> patch role airbyte-admin-role \     --type='json' \     -p='[{"op": "replace", "path": "/rules/0/resources", "value": ["jobs","pods","pods/log","pods/exec","pods/attach","secrets"]}]'
- If using your own service account, the Role in that namespace should include:
Copy code
yaml   apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>   kind: Role   metadata:     name: <role-name>   rules:     - apiGroups: ["*"]       resources: ["jobs","pods","pods/log","pods/exec","pods/attach","secrets"]       verbs: ["get","list","watch","create","update","patch","delete"]
Then restart the bootloader pod. --- If, after confirming: 1. The pod can reach 
168.40.0.1:443
, and 2. The service account has 
secrets
permissions, the bootloader still times out with the same message, the knowledge sources don’t provide a more specific, documented remedy. In that case, you’d likely need to open a new GitHub issue or forum topic and include: - Full bootloader logs from startup - Airbyte version(s) before/after your attempted upgrade - How you deploy (Helm vs abctl) and your cluster/Kubernetes version.
2 Views
Open in Slack
PreviousNext
Powered by