https://linen.dev logo
Join Slack
Powered by
Hi <@U04S8PLJWMT> getting this error from worker
# replication-ask-ai
i
Hi @kapa.ai getting this error from worker
k
From the snippets you shared, all the errors are coming from the worker, but they’re from different situations. To help you meaningfully, I need the exact error message you’re seeing now (a few lines before and after it from the worker logs). Here are some examples from the provided sources so you can match what you’re seeing: 1. “Could not find image: airbyte/source-postgres:…” / “Unable to get image” This happens when the worker can’t find or pull a connector image (e.g. source‑postgres) even though the UI is up. [[Unable image](https://discuss.airbyte.io/t/1803)][[Unable image-logs](https://discuss.airbyte.io/t/1803)] In that case you’ll see log lines like: - 
Checking if airbyte/source-postgres:0.4.30 exists...
- 
airbyte/source-postgres:0.4.30 not found locally. Attempting to pull the image...
- 
Image does not exist.
- 
WorkerException: Could not find image: airbyte/source-postgres:0.4.30
2. “Running the launcher replication-orchestrator failed” / K8s pod errors On Kubernetes, you might see errors like: - 
WorkerException: Running the launcher replication-orchestrator failed
- With a cause like 
Failed to create pod orchestrator-... pre-existing pod exists...
or missing secret 
airbyte-secret
. [[Launcher failed](https://discuss.airbyte.io/t/3823)][[Worker not started](https://discuss.airbyte.io/t/3054)] 3. Normalization / dbt errors - 
WorkerException: Normalization Failed.
or - 
WorkerException: Dbt Transformation Failed.
[[Hubspot norm](https://discuss.airbyte.io/t/617)][[dbt with snowflake](https://discuss.airbyte.io/t/285)] 4. Container‑orchestrator / S3 state issues - Startup failure in worker bean initialization or JSON config validation around S3 state storage. [[Worker not started](https://discuss.airbyte.io/t/3054)] Right now, the knowledge sources don’t include a generic “this error from worker” without detail, so I can’t pinpoint the cause of your specific problem. Please paste: - The full error line from the worker (including 
WorkerException
or 
RuntimeException
text). - 10–20 lines around it from the 
airbyte-worker
logs. Once I see the exact message, I can map it to one of the known patterns above and suggest concrete steps.
i
Hi @kapa.ai 2025-12-13 233137,584 [Activity Executor taskQueue="CONNECTION_UPDATER", namespace="default": 8195] WARN i.a.a.c.c.ClientConfigurationSupport(generateDefaultRetryPolicy$lambda$5):79 - Retry attempt 1 of 5. Last response: null 2025-12-13 233139,841 [Activity Executor taskQueue="CONNECTION_UPDATER", namespace="default": 8195] WARN i.a.a.c.c.ClientConfigurationSupport(generateDefaultRetryPolicy$lambda$5):79 - Retry attempt 2 of 5. Last response: null 2025-12-13 233142,192 [Activity Executor taskQueue="CONNECTION_UPDATER", namespace="default": 8195] WARN i.a.a.c.c.ClientConfigurationSupport(generateDefaultRetryPolicy$lambda$5):79 - Retry attempt 3 of 5. Last response: null Dec 13, 2025 113142 PM okhttp3.internal.platform.Platform log WARNING: A connection to http://airbyte-airbyte-server-svc:8001/ was leaked. Did you forget to close a response body? To see where this was allocated, set the OkHttpClient logger level to FINE: Logger.getLogger(OkHttpClient.class.getName()).setLevel(Level.FINE); 2025-12-13 233145,108 [Activity Executor taskQueue="CONNECTION_UPDATER", namespace="default": 8195] WARN i.a.a.c.c.ClientConfigurationSupport(generateDefaultRetryPolicy$lambda$5):79 - Retry attempt 4 of 5. Last response: null 2025-12-13 233148,397 [Activity Executor taskQueue="CONNECTION_UPDATER", namespace="default": 8195] WARN i.a.a.c.c.ClientConfigurationSupport(generateDefaultRetryPolicy$lambda$5):79 - Retry attempt 5 of 5. Last response: null 2025-12-13 233148,461 [Activity Executor taskQueue="CONNECTION_UPDATER", namespace="default": 8195] ERROR i.a.a.c.c.ClientConfigurationSupport(generateDefaultRetryPolicy$lambda$7):95 - Retry attempts exceeded. java.io.IOException: HTTP error: 500 Internal Server Error at io.airbyte.api.client.interceptor.ThrowOn5xxInterceptor.intercept(ThrowOn5xxInterceptor.kt:16) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) at dev.failsafe.okhttp.FailsafeCall.lambda$execute$0(FailsafeCall.java:117) at dev.failsafe.Functions.lambda$get$0(Functions.java:46) at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74) at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187) at dev.failsafe.CallImpl.execute(CallImpl.java:33) at dev.failsafe.okhttp.FailsafeCall.execute(FailsafeCall.java:121) at io.airbyte.api.client.generated.AttemptApi.createNewAttemptNumberWithHttpInfo(AttemptApi.kt:891) at io.airbyte.api.client.generated.AttemptApi.createNewAttemptNumber(AttemptApi.kt:74) at io.airbyte.workers.temporal.scheduling.activities.JobCreationAndStatusUpdateActivityImpl.createNewAttemptNumber(JobCreationAndStatusUpdateActivityImpl.kt:88) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107) at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124) at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:290) at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:254) at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:217) at io.temporal.internal.worker.PollTaskExecutor.lambda$process$1(PollTaskExecutor.java:96) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583) 2025-12-13 233148,462 [Activity Executor taskQueue="CONNECTION_UPDATER", namespace="default": 8195] ERROR i.a.a.c.c.ClientConfigurationSupport(generateDefaultRetryPolicy$lambda$3):63 - Failed to call unknown. Last response: null java.io.IOException: HTTP error: 500 Internal Server Error at io.airbyte.api.client.interceptor.ThrowOn5xxInterceptor.intercept(ThrowOn5xxInterceptor.kt:16) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) at dev.failsafe.okhttp.FailsafeCall.lambda$execute$0(FailsafeCall.java:117) at dev.failsafe.Functions.lambda$get$0(Functions.java:46) at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74) at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187) at dev.failsafe.CallImpl.execute(CallImpl.java:33) at dev.failsafe.okhttp.FailsafeCall.execute(FailsafeCall.java:121) at io.airbyte.api.client.generated.AttemptApi.createNewAttemptNumberWithHttpInfo(AttemptApi.kt:891) at io.airbyte.api.client.generated.AttemptApi.createNewAttemptNumber(AttemptApi.kt:74) at io.airbyte.workers.temporal.scheduling.activities.JobCreationAndStatusUpdateActivityImpl.createNewAttemptNumber(JobCreationAndStatusUpdateActivityImpl.kt:88) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107) at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124) at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:290) at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:254) at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:217) at io.temporal.internal.worker.PollTaskExecutor.lambda$process$1(PollTaskExecutor.java:96) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583) 2025-12-13 233148,462 [Activity Executor taskQueue="CONNECTION_UPDATER", namespace="default": 8195] ERROR i.a.w.t.s.a.JobCreationAndStatusUpdateActivityImpl(createNewAttemptNumber):98 - createNewAttemptNumber for job 61921 failed with exception: HTTP error: 500 Internal Server Error java.io.IOException: HTTP error: 500 Internal Server Error at io.airbyte.api.client.interceptor.ThrowOn5xxInterceptor.intercept(ThrowOn5xxInterceptor.kt:16) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) at dev.failsafe.okhttp.FailsafeCall.lambda$execute$0(FailsafeCall.java:117) at dev.failsafe.Functions.lambda$get$0(Functions.java:46) at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74) at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187) at dev.failsafe.CallImpl.execute(CallImpl.java:33) at dev.failsafe.okhttp.FailsafeCall.execute(FailsafeCall.java:121) at io.airbyte.api.client.generated.AttemptApi.createNewAttemptNumberWithHttpInfo(AttemptApi.kt:891) at io.airbyte.api.client.generated.AttemptApi.createNewAttemptNumber(AttemptApi.kt:74) at io.airbyte.workers.temporal.scheduling.activities.JobCreationAndStatusUpdateActivityImpl.createNewAttemptNumber(JobCreationAndStatusUpdateActivityImpl.kt:88) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107) at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124) at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:290) at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:254) at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:217) at io.temporal.internal.worker.PollTaskExecutor.lambda$process$1(PollTaskExecutor.java:96) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583) More log lines aronud it
k
A message from kapa.ai
These worker logs show that the worker cannot successfully call the Airbyte Server’s Attempt API (
AttemptApi.createNewAttemptNumber
), and after 5 retries the call still returns HTTP 500 Internal Server Error from 
<http://airbyte-airbyte-server-svc:8001>
. The problem is on the server side, not in the worker itself. Very similar 5xx / 
ThrowOn5xxInterceptor
patterns have been seen when: - The server is unhealthy or cannot reach its dependencies (e.g. workload API, storage, or database). [[EKS 500 when creating sources](https://github.com/airbytehq/airbyte/issues/48809)] - There is a temporary server‑side bug or bad cache, sometimes resolved by restarting the 
airbyte-server
pod. [[Unexpected 500 cache](https://github.com/airbytehq/airbyte/issues/54058)] - In newer Helm/chart setups, the workload‑launcher or server cannot be reached and returns errors when the worker calls the internal API. [[2.0.0 chart server 500/connect](https://github.com/airbytehq/airbyte/issues/68134)] What to do next: 1. Check 
airbyte-server
logs at the same time window (around 
2025-12-13 23:31:37–23:31:49
). You should see the underlying exception for the 500 error (e.g. DB error, NPE, or another stack trace). That error is what needs fixing. 2. Verify internal service connectivity: - From inside the worker pod, curl 
<http://airbyte-airbyte-server-svc:8001/health>
(or equivalent health endpoint your version exposes). - Confirm DNS/service name (
airbyte-airbyte-server-svc
) and port 
8001
match the server service definition. [[EKS internal_api_host/workload_api_host](https://github.com/airbytehq/airbyte/issues/48809)] 3. Check server configuration / env vars: - Database connection (JDBC URL, credentials). - Storage / MinIO / S3 endpoints if you use them; past 5xx issues have been tied to storage connectivity. [[502 when creating connection – Minio issue](https://discuss.airbyte.io/t/1207)] - Internal & workload API host/envs (e.g. 
INTERNAL_API_HOST
, 
WORKLOAD_API_HOST
) are correctly set. [[EKS internal_api_host](https://github.com/airbytehq/airbyte/issues/48809)] 4. Restart 
airbyte-server
: - There is at least one report where the 500 was caused by a cache problem and restarting 
airbyte-server
resolved it. [[Unexpected 500 cache](https://github.com/airbytehq/airbyte/issues/54058)] 5. If using Helm / Kubernetes: - Ensure 
airbyte-airbyte-server-svc
and 
airbyte-workload-launcher
are both up and ready; in some chart versions server issues prevent the workload‑launcher from starting and surface as 5xx on internal calls. [[2.0.0 chart multiple issues](https://github.com/airbytehq/airbyte/issues/68134)] Because the knowledge sources do not include the exact 500 stack trace from your 
airbyte-server
logs, I can’t say precisely what is failing inside the server. The key is to: - Look at the server logs for the corresponding request/stack trace, - Fix that underlying server error (db, storage, misconfig, etc.), - Then retry the connection/job; the worker should stop throwing this 500 once the server responds normally. If you can share the matching 
airbyte-server
log snippet around 233148, I can help you interpret that specific 500.
i
@kapa.ai i see this in server logs 2025-12-14 091740,350 [io-executor-thread-1] INFO i.g.o.k.s.i.Slf4jLogger(info):82 - Attempt stats: {} 2025-12-14 091740,350 [io-executor-thread-1] INFO i.g.o.k.s.i.Slf4jLogger(info):82 - Hydrating job 61921 2025-12-14 091740,350 [io-executor-thread-1] WARN i.g.o.k.s.i.Slf4jLogger(warn):113 - Missing stats for job 61921 attempt 0 2025-12-14 091740,674 [io-executor-thread-6] ERROR i.a.s.a.ApiHelper(execute):47 - Unexpected Exception io.airbyte.commons.server.errors.IdNotFoundKnownException: Could not find attempt stats for job_id: 61921 and attempt no: 0 at io.airbyte.commons.server.handlers.AttemptHandler.getAttemptCombinedStats(AttemptHandler.kt:251) at io.airbyte.server.apis.controllers.AttemptApiController.getAttemptCombinedStats$lambda$2(AttemptApiController.kt:66) at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.kt:32) at io.airbyte.server.apis.controllers.AttemptApiController.getAttemptCombinedStats(AttemptApiController.kt:65) at io.airbyte.server.apis.controllers.$AttemptApiController$Definition$Exec.dispatch(Unknown Source) at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invokeUnsafe(AbstractExecutableMethodsDefinition.java:461) at io.micronaut.context.DefaultBeanContext$BeanContextUnsafeExecutionHandle.invokeUnsafe(DefaultBeanContext.java:4438) at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:272) at io.micronaut.web.router.DefaultUriRouteMatch.execute(DefaultUriRouteMatch.java:38) at io.micronaut.http.server.RouteExecutor.executeRouteAndConvertBody(RouteExecutor.java:465) at io.micronaut.http.server.RouteExecutor.lambda$callRoute$5(RouteExecutor.java:442) at io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:92) at io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:232) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583) 2025-12-14 091740,675 [io-executor-thread-6] ERROR i.a.c.s.e.h.IdNotFoundExceptionHandler(handle):33 - Not found exception class NotFoundKnownExceptionInfo { id: null message: Id not found: Could not find attempt stats for job_id: 61921 and attempt no: 0 exceptionClassName: io.airbyte.commons.server.errors.IdNotFoundKnownException
k
A message from kapa.ai
The server log shows the actual cause of the worker’s 500s:
Copy code
text io.airbyte.commons.server.errors.IdNotFoundKnownException: Could not find attempt stats for job_id: 61921 and attempt no: 0 ... ERROR i.a.c.s.e.h.IdNotFoundExceptionHandler(handle):33 - Not found exception message: Id not found: Could not find attempt stats for job_id: 61921 and attempt no: 0
So: - The worker calls 
AttemptApi.createNewAttemptNumber
(or related stats APIs). - The server tries to hydrate job 61921 and fetch attempt stats for attempt 0. - No stats exist for that attempt, so the server throws 
IdNotFoundKnownException
, which becomes a 500 to the worker. This “Missing stats for job … attempt …” pattern is documented as a warning when stats are absent in persistence; it appears in similar form for job history and stats hydration. [[JobHistoryHandler missing stats](https://discuss.airbyte.io/t/4016)] From the knowledge we have: - There is no explicit fix documented for this exact 
IdNotFoundKnownException
in 
AttemptHandler.getAttemptCombinedStats
. - A related 500 issue caused by stale cache was solved simply by restarting 
airbyte-server
. [[Unexpected 500 cache](https://github.com/airbytehq/airbyte/issues/54058)] Given that: 1. Try a full restart of 
airbyte-server
 (and, if you’re on Kubernetes, let the pod be recreated), then retry the job. This has resolved unexplained 500s in at least one similar case. [[Unexpected 500 cache](https://github.com/airbytehq/airbyte/issues/54058)] 2. Check if the job/attempt really exists in the UI: - Does job 61921 show up in the Jobs page? - Does it list attempt 0? If the job exists but stats don’t, the server is trying to fetch stats that were never written (or were lost). The knowledge sources do not state how to repair missing attempt‑stats rows, so I cannot recommend a safe DB manual fix. 3. Workaround to move forward: - Cancel job 61921 (if it’s hanging). - Trigger a new sync / new job, so a fresh job ID and attempt are created. - If new jobs also fail with the same pattern, it points to a systematic problem in how attempt stats are being stored or read; unfortunately the sources above don’t document configuration‑level causes or migrations related to this specific handler. Because the provided documents don’t include a known bug or migration note specifically about 
AttemptHandler.getAttemptCombinedStats
or this 
IdNotFoundKnownException
, I can’t give a definitive root‑cause beyond “server cannot find attempt stats for that job/attempt in persistence.” If restarting 
airbyte-server
and re‑running a new job still leads to the same error for new job IDs, the next step (not covered in the docs) would be to open a GitHub issue with: - Airbyte version - Deployment type (Docker / K8s, Helm chart version) - The worker + server stack traces you posted.
Open in Slack
PreviousNext
Powered by