https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> i see this error in the logs Error ...
# replication-ask-ai
a
@kapa.ai i see this error in the logs Error from server: Get "https://172.19.0.2:10250/containerLogs/airbyte-abctl/airbyte-abctl-server-6b77cd774b-hqlwx/airbyte-server-container": dial tcp 172.19.0.210250 connect: connection refused what is the issue?
k
The error indicates that 
kubectl
(or 
abctl
, which uses Kubernetes under the hood) is trying to talk to the kubelet on 
172.19.0.2:10250
to fetch the pod logs, but the TCP connection is being refused by that node. In Airbyte’s troubleshooting docs, a similar “connection refused” situation is documented when 
abctl
can’t talk to the Kubernetes API/server, resulting in errors like: > 
unable to initialize local command: error communicating with kubernetes ... connect: connection refused
[[Troubleshoot abctl](https://docs.airbyte.com/platform/deploying-airbyte/troubleshoot-deploy#connection-refused)]. In both cases, “connection refused” means the Kubernetes endpoint 
abctl
expects to reach is not accepting connections (service not running, not listening on that port, or blocked). However, the provided knowledge sources do not give specific remediation steps for kubelet (10250) connection issues, only the general “connection refused” case above. Because of that, I can only say with confidence that: - The problem is a network/availability issue between `abctl`/`kubectl` and the Kubernetes node at 
172.19.0.2
(the node is refusing connections on 10250). - It’s analogous in nature to the documented “connection refused” error where 
abctl
cannot communicate with Kubernetes. [[Troubleshoot abctl](https://docs.airbyte.com/platform/deploying-airbyte/troubleshoot-deploy#connection-refused)] The knowledge sources do not explicitly state the exact root cause or step‑by‑step fix for your specific kubelet log endpoint error.
a
@kapa.ai how do i restart the server?
k
A message from kapa.ai
To restart the Airbyte server when using 
abctl
, you restart the relevant Kubernetes deployments in the 
airbyte-abctl
namespace with 
kubectl
. From the docs, 
kubectl
is used to manage the 
abctl
-created cluster and deployments, and you can inspect pods and logs in the 
airbyte-abctl
namespace. [[Developing locally 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#prerequisites)] A pattern for restarting services is shown for the worker and workload API: [[Issue 66692](https://github.com/airbytehq/airbyte/issues/66692)]
Copy code
bash kubectl -n airbyte-abctl rollout restart deployment airbyte-abctl-server kubectl -n airbyte-abctl rollout status deployment airbyte-abctl-server
If you also need to restart other components involved in handling requests (as in the 502 examples), you can restart them similarly, for example:
Copy code
bash kubectl -n airbyte-abctl rollout restart deployment airbyte-abctl-worker kubectl -n airbyte-abctl rollout restart deployment airbyte-abctl-workload-api-server  kubectl -n airbyte-abctl rollout status deployment airbyte-abctl-worker kubectl -n airbyte-abctl rollout status deployment airbyte-abctl-workload-api-server
a
@kapa.ai now i see 2025-12-12 054347,897 [main] INFO i.m.c.e.DefaultEnvironment(<init>):170 - Established active environments: [k8s, control-plane, edition-community] 2025-12-12 054348,680 [main] WARN c.z.h.HikariConfig(validateNumerics):1135 - HikariPool-1 - idleTimeout is close to or more than maxLifetime, disabling it. 2025-12-12 054348,692 [main] INFO c.z.h.HikariDataSource(<init>):79 - HikariPool-1 - Starting... 2025-12-12 054348,714 [main] INFO c.z.h.HikariDataSource(<init>):81 - HikariPool-1 - Start completed. 2025-12-12 054349,178 [main] INFO i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):163 - Waiting for temporal server... 2025-12-12 054349,179 [main] WARN i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):174 - Waiting for namespace default to be initialized in temporal... 2025-12-12 054351,669 [main] INFO i.t.s.WorkflowServiceStubsImpl(<init>):86 - Created WorkflowServiceStubs for channel: ManagedChannelOrphanWrapper{delegate=ManagedChannelImpl{logId=1, target=airbyte- abctl-temporal:7233}} 2025-12-12 054351,932 [main] WARN i.a.c.t.TemporalUtils(getTemporalClientWhenConnected):184 - Ignoring exception while trying to request Temporal namespace: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:351) at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:332) at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:174) at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.getSystemInfo(WorkflowServiceGrpc.java:6065) at io.temporal.serviceclient.SystemInfoInterceptor.getServerCapabilitiesOrThrow(SystemInfoInterceptor.java:132) at io.temporal.serviceclient.SystemInfoInterceptor$1.start(SystemInfoInterceptor.java:86) at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:418) at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:394) and Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: airbyte-abctl-temporal/10.96.77.137:7233 Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused at io.grpc.netty.shaded.io.netty.channel.unix.Errors.newConnectException0(Errors.java:166) at io.grpc.netty.shaded.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131) at io.grpc.netty.shaded.io.netty.channel.unix.Socket.finishConnect(Socket.java:359) at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:715) at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:692) at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:491) at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399) at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994) at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:1583)
k
A message from kapa.ai
These logs show that the Airbyte server has started, opened its database pool (Hikari), and is now failing to connect to the Temporal service at `airbyte-abctl-temporal:7233`: - 
Waiting for temporal server...
- 
Waiting for namespace default to be initialized in temporal...
- Then repeated 
UNAVAILABLE: io exception
and 
Connection refused: airbyte-abctl-temporal/10.96.77.137:7233
. This pattern (gRPC 
UNAVAILABLE
/ 
Connection refused
to 
airbyte-temporal
) matches several documented issues where Airbyte components cannot talk to the Temporal pod/service; in those cases workers, server, or other pods keep restarting or returning 502s because Temporal isn’t reachable. [[GKE connection refused](https://github.com/airbytehq/airbyte/issues/50948); [Server–Temporal connection](https://github.com/airbytehq/airbyte/issues/68655)] From the knowledge sources, the only clear conclusion is: - The Airbyte server process is up, but it cannot establish a gRPC connection to the Temporal endpoint (
airbyte-abctl-temporal:7233
), so scheduling/executing jobs will not work. - This is a connectivity/availability problem on the Temporal side (pod not ready, not listening, or service misconfigured), not an application‑level configuration error inside the server. The docs and issues provided do not include a definitive, step‑by‑step fix for this specific 
airbyte-abctl-temporal
“connection refused” situation when using 
abctl
. They only show that such errors mean Temporal is not reachable and cause Airbyte components to fail. [[Temporal connection refused](https://github.com/airbytehq/airbyte/issues/50948); [Temporal UNAVAILABLE](https://github.com/airbytehq/airbyte/issues/68655)] So, based strictly on the sources: - Your problem is: Airbyte server cannot connect to the Temporal service (
airbyte-abctl-temporal:7233
), which is refusing connections. - The knowledge sources do not explicitly document how to repair this for 
abctl
deployments (e.g., whether it is a pod crash, service selector issue, or cluster networking problem), so I cannot reliably give you more specific remediation steps from the provided material.
a
@kapa.ai this is the temporal pod log Error from server: Get "https://172.19.0.2:10250/containerLogs/airbyte-abctl/airbyte-abctl-temporal-bb495795b-x99jh/airbyte-temporal": dial tcp 172.19.0.210250 connect: connection refused
k
A message from kapa.ai
That message means Kubernetes cannot reach the node’s kubelet to stream the Temporal pod logs: > 
Get "<https://172.19.0.2:10250/containerLogs/...>": dial tcp 172.19.0.2:10250: connect: connection refused
So: - The **Temporal pod log itself is not being returned**; instead the API server is failing to connect to the kubelet on 
172.19.0.2:10250
. - This is the same class of “connection refused” problem documented for 
abctl
, where it can’t talk to the Kubernetes endpoint it expects. [[Troubleshoot abctl](https://docs.airbyte.com/platform/deploying-airbyte/troubleshoot-deploy#connection-refused)] Based on the provided sources, the only firm conclusion is: - This error indicates a Kubernetes node / kubelet connectivity issue, not a Temporal‑specific log error. - Because the sources don’t include remediation steps for kubelet 
10250
failures, I can’t reliably outline the exact fix (e.g., restarting kubelet, checking node health, firewall rules) from the given material.
a
@kapa.ai 2025-12-12 070626,033 [io-executor-thread-1] ERROR i.a.c.s.e.h.IdNotFoundExceptionHandler(handle):33 - Not found exception class NotFoundKnownExceptionInfo { id: null message: Id not found: Could not find attempt stats for job_id: 31966 and attempt no: 0 exceptionClassName: io.airbyte.commons.server.errors.IdNotFoundKnownException exceptionStack: [io.airbyte.commons.server.errors.IdNotFoundKnownException: Id not found: Could not find attempt stats for job_id: 31966 and attempt no: 0, at io.airbyte.commons.server.errors.handlers.IdNotFoundExceptionHandler.handle(IdNotFoundExceptionHandler.kt:32), at io.airbyte.commons.server.errors.handlers.IdNotFoundExceptionHandler.handle(IdNotFoundExceptionHandler.kt:24), at io.micronaut.http.server.RequestLifecycle.lambda$handlerExceptionHandler$10(RequestLifecycle.java:314), at io.micronaut.http.server.RequestLifecycle.handlerExceptionHandler(RequestLifecycle.java:326), at io.micronaut.http.server.RequestLifecycle.onErrorNoFilter(RequestLifecycle.java:253), at io.micronaut.http.server.RequestLifecycle.lambda$onErrorNoFilter$2(RequestLifecycle.java:215), at io.micronaut.core.execution.ImperativeExecutionFlowImpl.onErrorResume(ImperativeExecutionFlowImpl.java:112), at io.micronaut.core.execution.DelayedExecutionFlowImpl$OnErrorResume.apply(DelayedExecutionFlowImpl.java:393), at io.micronaut.core.execution.DelayedExecutionFlowImpl.work(DelayedExecutionFlowImpl.java:58), at io.micronaut.core.execution.DelayedExecutionFlowImpl.completeLazy(DelayedExecutionFlowImpl.java:80), at io.micronaut.core.execution.DelayedExecutionFlowImpl.completeExceptionally(DelayedExecutionFlowImpl.java:104), at io.micronaut.core.execution.DelayedExecutionFlow.complete(DelayedExecutionFlow.java:100), at io.micronaut.core.execution.ImperativeExecutionFlowImpl.onComplete(ImperativeExecutionFlowImpl.java:132), at io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:92), at io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:232), at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144), at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642), at java.base/java.lang.Thread.run(Thread.java:1583), Caused by: io.airbyte.commons.server.errors.IdNotFoundKnownException: Could not find attempt stats for job_id: 31966 and attempt no: 0, at io.airbyte.commons.server.handlers.AttemptHandler.getAttemptCombinedStats(AttemptHandler.kt:251), at io.airbyte.server.apis.controllers.AttemptApiController.getAttemptCombinedStats$lambda$2(AttemptApiController.kt:66), at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.kt:32), at io.airbyte.server.apis.controllers.AttemptApiController.getAttemptCombinedStats(AttemptApiController.kt:65), at io.airbyte.server.apis.controllers.$AttemptApiController$Definition$Exec.dispatch(Unknown Source), at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invokeUnsafe(AbstractExecutableMethodsDefinition.java:461), at io.micronaut.context.DefaultBeanContext$BeanContextUnsafeExecutionHandle.invokeUnsafe(DefaultBeanContext.java:4438), at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:272), at io.micronaut.web.router.DefaultUriRouteMatch.execute(DefaultUriRouteMatch.java:38), at io.micronaut.http.server.RouteExecutor.executeRouteAndConvertBody(RouteExecutor.java:465), at io.micronaut.http.server.RouteExecutor.lambda$callRoute$5(RouteExecutor.java:442), ... 5 more] rootCauseExceptionClassName: java.lang.Class rootCauseExceptionStack: [io.airbyte.commons.server.errors.IdNotFoundKnownException: Could not find attempt stats for job_id: 31966 and attempt no: 0, at io.airbyte.commons.server.handlers.AttemptHandler.getAttemptCombinedStats(AttemptHandler.kt:251), at io.airbyte.server.apis.controllers.AttemptApiController.getAttemptCombinedStats$lambda$2(AttemptApiController.kt:66), at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.kt:32), at io.airbyte.server.apis.controllers.AttemptApiController.getAttemptCombinedStats(AttemptApiController.kt:65), at io.airbyte.server.apis.controllers.$AttemptApiController$Definition$Exec.dispatch(Unknown Source), at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invokeUnsafe(AbstractExecutableMethodsDefinition.java:461), at io.micronaut.context.DefaultBeanContext$BeanContextUnsafeExecutionHandle.invokeUnsafe(DefaultBeanContext.java:4438), at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:272), at io.micronaut.web.router.DefaultUriRouteMatch.execute(DefaultUriRouteMatch.java:38), at io.micronaut.http.server.RouteExecutor.executeRouteAndConvertBody(RouteExecutor.java:465), at io.micronaut.http.server.RouteExecutor.lambda$callRoute$5(RouteExecutor.java:442), at io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:92), at io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:232), at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144), at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642), at java.base/java.lang.Thread.run(Thread.java:1583)] }
k
A message from kapa.ai
This error means Airbyte’s API tried to fetch attempt stats for a specific job attempt (
job_id: 31966
, 
attempt no: 0
), but there is no corresponding stats record, so the server throws 
IdNotFoundKnownException
instead of returning data. From the logs and issues you shared, this is the same underlying situation as the repeated warnings: > 
Missing stats for job <id> attempt <n>
[[Jobs DB schema](https://docs.airbyte.com/platform/understanding-airbyte/database-data-catalog#jobs-database); [JobHistoryHandler warnings](https://discuss.airbyte.io/t/4016)] Key points based on the sources: - Airbyte stores job + attempt metadata in the jobs and attempts tables (status, timestamps, etc.), and separately stores attempt stats which some handlers use to build responses for the UI/API. [[Jobs DB schema](https://docs.airbyte.com/platform/understanding-airbyte/database-data-catalog#jobs-database)] - When those stats are missing for a given 
(job_id, attempt_number)
, server-side code logs “Missing stats for job … attempt …” and, for certain API calls (like 
getAttemptCombinedStats
), now raises 
IdNotFoundKnownException: Could not find attempt stats …
. [[JobHistoryHandler warnings](https://discuss.airbyte.io/t/4016); [Missing stats noise](https://discuss.airbyte.io/t/3804)] - In the forum threads, this is treated as log noise / missing stats, not as something that necessarily breaks syncs; there is no documented data loss or corruption associated with it, only incomplete statistics and noisy logs. [[JobHistoryHandler warnings](https://discuss.airbyte.io/t/4016); [Missing stats noise](https://discuss.airbyte.io/t/3804)] What this implies for you: - The sync for job 31966 may still have run; but Airbyte cannot return combined stats for attempt 0 because those stats are absent. - The knowledge sources do not document a supported way to “rebuild” or backfill attempt stats, nor a configuration toggle to suppress this specific 
IdNotFoundKnownException
. They only show that: - The stats are read from persistence and may be missing. [[JobHistoryHandler warnings](https://discuss.airbyte.io/t/4016)] - Users see repeated “Missing stats for job … attempt …” messages in logs. [[Missing stats noise](https://discuss.airbyte.io/t/3804)] Because of that, I can’t give you a confirmed repair procedure (like SQL fixes or config changes) from the provided material. From what we can say based on the sources: - This error indicates missing attempt stats for that job/attempt, which affects UI/API stats views for that run. - There’s no explicit evidence in the docs/issues provided that this alone causes jobs to hang or fail; it mainly affects observability and triggers noisy warnings/errors in logs. If this is blocking you functionally (e.g., UI fails to load pages), the next documented step would be to gather the job/attempt IDs and open an issue or support ticket, as users did in the linked GitHub and forum threads when they encountered similar “Missing stats” behavior. [[JobHistoryHandler warnings](https://discuss.airbyte.io/t/4016); [Missing stats noise](https://discuss.airbyte.io/t/3804)]
a
@kapa.ai Failed to pull image "airbyte/source-postgres:3.7.0": failed to pull and unpack image "docker.io/airbyte/source-postgres:3.7.0": failed to copy: httpReadSeeker: failed open: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://mirror.gcr.io/v2/token?scope=repository%3Aairbyte%2Fsource-postgres%3Apull&amp;scope=repository%3Aartifact-registry-docker-cache%2Fvirtual-docker%2Fairbyte%2Fsource-postgres%3Apull&amp;service=mirror.gcr.io: 403 Forbidden
k
A message from kapa.ai
This error means Kubernetes is trying to pull the connector image from Docker Hub via 
<http://mirror.gcr.io|mirror.gcr.io>
, but the token request to 
<http://mirror.gcr.io|mirror.gcr.io>
is being rejected with 
403 Forbidden
, so the image can’t be pulled: > failed to fetch anonymous token … `https://mirror.gcr.io/v2/token?...`: 403 Forbidden You can work around this in two main ways, depending on how you deploy Airbyte: --- ### 1. Kubernetes + Helm: use a custom / proxy registry Configure Airbyte to pull all platform and connector images from a custom or proxy registry that you control (e.g., your own Docker Hub mirror, GCR/Artifact Registry, ECR, ACR). In `values.yaml`:
Copy code
yaml global:   image:     registry: <http://ghcr.io/NAMESPACE|ghcr.io/NAMESPACE>
This tells Airbyte to use 
<http://ghcr.io/NAMESPACE|ghcr.io/NAMESPACE>
instead of 
<http://docker.io|docker.io>
when pulling images (including 
airbyte/source-postgres
). [[Custom image registry](https://docs.airbyte.com/platform/deploying-airbyte/integrations/custom-image-registries#step-1-customize-airbyte-to-use-your-image-registry)] If the registry is private, create an image‑pull secret and reference it:
Copy code
bash kubectl create secret generic regcred \   --from-file=.dockerconfigjson=<path/to/.docker/config.json> \   --type=<http://kubernetes.io/dockerconfigjson|kubernetes.io/dockerconfigjson>
Copy code
yaml global:   image:     registry: <http://ghcr.io/NAMESPACE|ghcr.io/NAMESPACE>   imagePullSecrets:     - name: regcred
[[Custom image registry](https://docs.airbyte.com/platform/deploying-airbyte/integrations/custom-image-registries#step-1-customize-airbyte-to-use-your-image-registry)] Make sure your Airbyte Helm chart is recent (≥ v1.2.0); older chart versions had issues where the custom registry setting did not apply to connectors, causing them to still pull from 
<http://docker.io|docker.io>
and hit rate/authorization limits. Upgrading to ≥ v1.5.1 is confirmed to fix this behavior. [[Custom registry connectors](https://github.com/airbytehq/airbyte/discussions/57509)] --- ### 2. Docker Compose or custom connectors from a private registry If you are using Docker Compose and/or custom connectors in a private registry (e.g., GCR, ECR, ACR, private Docker Hub): - Authenticate both: - The environment where you build/push the image, and - The host where Airbyte runs (so it can pull the image). For example, with ECR:
Copy code
bash   aws ecr get-login-password --region <region> \     | docker login --username AWS --password-stdin <aws_account_id>.dkr.ecr.<region>.<http://amazonaws.com|amazonaws.com>
Similar flows exist for GCP Artifact Registry, Azure Container Registry, Docker Hub service accounts, or a self‑hosted registry. [[Uploading custom connectors](https://docs.airbyte.com/platform/operator-guides/using-custom-connectors#3-adhere-to-airbytes-docker-image-requirements)] - For Kubernetes worker jobs that need to pull from another private registry, create a Kubernetes 
Secret
for your registry credentials and set:
Copy code
bash   # name of your image-pull secret   JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_SECRET=<secret-name>
on the 
airbyte-worker
pod, so job pods can authenticate when pulling connector images. [[Uploading custom connectors](https://docs.airbyte.com/platform/operator-guides/using-custom-connectors#3-adhere-to-airbytes-docker-image-requirements)] --- In short, the 403 is coming from 
<http://mirror.gcr.io|mirror.gcr.io>
when trying to anonymously proxy Docker Hub. To resolve it with the documented approaches, route image pulls through a registry/proxy you control (via 
global.image.registry
) and/or ensure proper authentication with your private registry and Kubernetes/Docker environment.
3 Views
Open in Slack
PreviousNext
Powered by