Hi <@U04S8PLJWMT> getting this error 97 [Activit...
# ask-ai
i
Hi @kapa.ai getting this error 97 [Activity Executor taskQueue="CONNECTION_UPDATER", namespace="default": 2040] ERROR i.a.w.t.s.a.JobCreationAndStatusUpdateActivityImpl(createNewAttemptNumber):98 - createNewAttemptNumber for job 60048 failed with exception: HTTP error: 500 Internal Server Error
k
The log line you’re seeing is a generic “HTTP 500 Internal Server Error” surfaced by Airbyte’s orchestration layer when an internal HTTP call fails; it doesn’t by itself pinpoint the root cause. From the provided knowledge sources, there are two main patterns where HTTP 500s commonly appear: 1. Enterprise source stubs fetch failure (UI/API 500): In several issues, a 500 appears when Airbyte tries to call
/api/v1/source_definitions/list_enterprise_source_stubs
and fails with: >
Internal Server Error: java.io.IOException: HTTP error fetching enterprise sources
[[HTTP 500 enterprise](https://github.com/airbytehq/airbyte/issues/53609); [UI 500 error](https://github.com/airbytehq/airbyte/issues/54058)]. A temporary workaround used by OSS users was: - Host a file containing just
[]
(empty JSON array). - Set the
ENTERPRISE_SOURCE_STUBS_URL
environment variable to point to that file, so Airbyte stops trying (and failing) to fetch real enterprise stubs. [[HTTP 500 enterprise](https://github.com/airbytehq/airbyte/issues/53609)] In that specific case, the underlying endpoint cache was later fixed and the issue resolved by the team. [[HTTP 500 enterprise](https://github.com/airbytehq/airbyte/issues/53609); [UI 500 error](https://github.com/airbytehq/airbyte/issues/54058)] 2. Connector/underlying API returning 500 during sync: Some connectors (for example
source-posthog
) can start failing because the remote API endpoint begins returning 500s or timeouts (e.g., deprecated endpoints). [[PostHog 500](https://github.com/airbytehq/airbyte/issues/60293)] In those cases, the Airbyte job logs for the specific sync/check attempt will usually show the external HTTP 500 and possibly connector‑specific stack traces. Given your single log line:
Copy code
text createNewAttemptNumber for job 60048 failed with exception: HTTP error: 500 Internal Server Error
the knowledge sources do not give a direct mapping from that exact message to a single known bug or config problem. I can’t reliably say which of the above patterns (or something else) you’re hitting without more context. How to narrow it down (based on available info): 1. Check if it’s the enterprise stubs issue: - Look in
airbyte-server
logs around job
60048
for any call to
/api/v1/source_definitions/list_enterprise_source_stubs
returning 500 with message
HTTP error fetching enterprise sources
. [[HTTP 500 enterprise](https://github.com/airbytehq/airbyte/issues/53609); [UI 500 error](https://github.com/airbytehq/airbyte/issues/54058)] - If that’s present and you’re on an OSS/self‑hosted deployment, you can: - Temporarily set
ENTERPRISE_SOURCE_STUBS_URL
to a hosted
[]
file as described above, then restart
airbyte-server
. [[HTTP 500 enterprise](https://github.com/airbytehq/airbyte/issues/53609)] - Or upgrade to a version where this endpoint/cache issue is already fixed (the linked issues are closed as resolved). [[HTTP 500 enterprise](https://github.com/airbytehq/airbyte/issues/53609); [UI 500 error](https://github.com/airbytehq/airbyte/issues/54058)] 2. If the 500 is from a source/destination API: - Open the full job logs for job
60048
in the UI and check: - Which connector is in use. - Whether the logs show remote API 500s or timeouts similar to the PostHog case. [[PostHog 500](https://github.com/airbytehq/airbyte/issues/60293)] - If so, adjusting error‑handling on HTTP 500s may help in connector‑builder connectors, by treating 500s as retryable instead of immediate failure (via HTTP response filters). [[HTTP codes handling](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/error-handling#http-codes)] - Example idea (for config‑based/Builder connectors): add a response filter with
http_codes: [500]
and
action: RETRY
so that transient 500s are retried instead of failing the job. [[HttpResponseFilter](https://docs.airbyte.com/platform/1.8/connector-development/config-based/understanding-the-yaml-file/reference#/definitions/HttpResponseFilter); [HTTP codes handling](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/error-handling#http-codes)] Because the sources you provided don’t include your exact deployment version, connector name, or the surrounding stack trace, I can’t be more specific than the above. If you can share: - The Airbyte version and deployment type (Docker, Kubernetes, abctl, etc.). - The connector and action (test connection vs. sync). - A larger excerpt of the server/job logs around job
60048
. then I can map it more directly to the closest documented issue or configuration option from the knowledge sources.
i
@kapa.ai seeing things like this also 2-01 205801,597 [Activity Executor taskQueue="CONNECTION_UPDATER", namespace="default": 2040] WARN i.t.i.a.ActivityTaskExecutors$BaseActivityTaskExecutor(execute):126 - Activity failure. ActivityId=1cebdea1-8611-3e69-93fc-313645a0981e, activityType=CreateNewAttemptNumber, attempt=3 io.airbyte.commons.temporal.exception.RetryableException: java.io.IOException: HTTP error: 500 Internal Server Error at io.airbyte.workers.temporal.scheduling.activities.JobCreationAndStatusUpdateActivityImpl.createNewAttemptNumber(JobCreationAndStatusUpdateActivityImpl.kt:99) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64) at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43) at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107) at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124) at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:290) at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:254) at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:217) at io.temporal.internal.worker.PollTaskExecutor.lambda$process$1(PollTaskExecutor.java:96) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: java.io.IOException: HTTP error: 500 Internal Server Error at io.airbyte.api.client.interceptor.ThrowOn5xxInterceptor.intercept(ThrowOn5xxInterceptor.kt:16) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) at dev.failsafe.okhttp.FailsafeCall.lambda$execute$0(FailsafeCall.java:117) at dev.failsafe.Functions.lambda$get$0(Functions.java:46) at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74) at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187) at dev.failsafe.CallImpl.execute(CallImpl.java:33) at dev.failsafe.okhttp.FailsafeCall.execute(FailsafeCall.java:121) at io.airbyte.api.client.generated.AttemptApi.createNewAttemptNumberWithHttpInfo(AttemptApi.kt:891) at io.airbyte.api.client.generated.AttemptApi.createNewAttemptNumber(AttemptApi.kt:74) at io.airbyte.workers.temporal.scheduling.activities.JobCreationAndStatusUpdateActivityImpl.createNewAttemptNumber(JobCreationAndStatusUpdateActivityImpl.kt:88) ... 13 common frames omitted Dec 01, 2025 90005 PM okhttp3.internal.platform.Platform log WARNING: A connection to http://airbyte-airbyte-server-svc:8001/ was leaked. Did you forget to close a response body? To see where this was allocated, set the OkHttpClient logger level to FINE: Logger.getLogger(OkHttpClient.class.getName()).setLevel(Level.FINE); Dec 01, 2025 90005 PM okhttp3.internal.platform.Platform log WARNING: A connection to http://airbyte-airbyte-server-svc:8001/ was leaked. Did you forget to close a response body? To see where this was allocated, set the OkHttpClient logger level to FINE: Logger.getLogger(OkHttpClient.class.getName()).setLevel(Level.FINE); Dec 01, 2025 90005 PM okhttp3.internal.platform.Platform log WARNING: A connection to http://airbyte-airbyte-server-svc:8001/ was leaked. Did you forget to close a response body? To see where this was allocated, set the OkHttpClient logger level to FINE: Logger.getLogger(OkHttpClient.class.getName()).setLevel(Level.FINE); Dec 01, 2025 90005 PM okhttp3.internal.platform.Platform log WARNING: A connection to http://airbyte-airbyte-server-svc:8001/ was leaked. Did you forget to close a response body? To see where this was allocated, set the OkHttpClient logger level to FINE: Logger.getLogger(OkHttpClient.class.getName()).setLevel(Level.FINE); Dec 01, 2025 90005 PM okhttp3.internal.platform.Platform log WARNING: A connection to http://airbyte-airbyte-server-svc:8001/ was leaked. Did you forget to close a response body? To see where this was allocated, set the OkHttpClient logger level to FINE: Logger.getLogger(OkHttpClient.class.getName()).setLevel(Level.FINE); Dec 01, 2025 90005 PM okhttp3.internal.platform.Platform log WARNING: A connection to http://airbyte-airbyte-server-svc:8001/ was leaked. Did you forget to close a response body? To see where this was allocated, set the OkHttpClient logger level to FINE: Logger.getLogger(OkHttpClient.class.getName()).setLevel(Level.FINE);
k
A message from kapa.ai
The stack trace shows: - Temporal activity
CreateNewAttemptNumber
is calling the Airbyte Server Attempt API and receiving HTTP 500. - The error is raised by
ThrowOn5xxInterceptor
inside the Airbyte API client, so the server itself is returning 500 for
AttemptApi.createNewAttemptNumber
. [[v1.6.0 deploy 5xx](https://github.com/airbytehq/airbyte/issues/58587)] - OkHttp warns about “connection leaked… did you forget to close a response body?” for
http://airbyte-airbyte-server-svc:8001/
, which indicates client connections to the server are not being closed cleanly, potentially under heavy/failed calls. From the knowledge you provided, there are similar patterns: - Internal HTTP 500s from Airbyte server (e.g. when fetching enterprise connector stubs, or from cache issues) manifested as generic 500 errors in the UI/API and were ultimately solved by fixing the server bug or restarting/upgrading the server. [[HTTP 500 enterprise](https://github.com/airbytehq/airbyte/issues/53609); [Unexpected 500 cache](https://github.com/airbytehq/airbyte/issues/54058)] - A 5xx interceptor in the abctl / platform client surfaces server‐side 500 errors in the same way you see in your logs. [[v1.6.0 deploy 5xx](https://github.com/airbytehq/airbyte/issues/58587)] However, the sources you shared do not contain: - The implementation details of
AttemptApi.createNewAttemptNumber
. - A known bug specifically about
JobCreationAndStatusUpdateActivityImpl.createNewAttemptNumber
returning 500. - A documented fix or configuration for the “connection leaked” warnings in this context. So I can’t point to a definitive root cause or a version‑specific bug for this exact stack trace from the knowledge set. Based on what is documented, the most relevant, low‑risk steps are: 1. Inspect the Airbyte Server logs around the 500s Look at the
airbyte-server
pod logs at the timestamps matching these Temporal activity errors. In similar cases, the server logs clearly showed the deeper reason (e.g. failing to fetch enterprise stubs, cache inconsistency). [[HTTP 500 enterprise](https://github.com/airbytehq/airbyte/issues/53609); [Unexpected 500 cache](https://github.com/airbytehq/airbyte/issues/54058)] 2. Restart airbyte-server For at least one internal 500 issue, maintainers indicated it was related to a cache issue and suggested restarting the
airbyte-server
to clear the problem. [[Unexpected 500 cache](https://github.com/airbytehq/airbyte/issues/54058)] If you haven’t already, restart only the
airbyte-server
deployment/pod and see if the
CreateNewAttemptNumber
calls start succeeding. 3. Check/upgrade to a version where similar 500s are marked as fixed Some 500s (e.g. enterprise stub fetching) were resolved in subsequent releases and the corresponding issues are closed. [[HTTP 500 enterprise](https://github.com/airbytehq/airbyte/issues/53609); [Unexpected 500 cache](https://github.com/airbytehq/airbyte/issues/54058)] If you’re on an older version than those fixes, upgrading Airbyte (and redeploying cleanly) is aligned with what support recommended in similar threads. [[K8s 502/health](https://discuss.airbyte.io/t/594); [K8s services errors](https://discuss.airbyte.io/t/2861)] 4. Verify deployment health (Kubernetes) In other Kubernetes deployments where UI/API operations failed (502/500s, Temporal workflow errors), the root issue was an unhealthy deployment (Temporal or server not reachable, pods spread across nodes causing volume/log issues, etc.). [[K8s 502/health](https://discuss.airbyte.io/t/594); [Gateway timeout / logs missing](https://discuss.airbyte.io/t/376)] Basic checks you can do, consistent with those threads: - All core pods (
airbyte-server
,
airbyte-worker
, Temporal, DB) are
Running
and ready. - Temporal is reachable from worker pods (no
Connection refused
to Temporal). [[K8s 502/health](https://discuss.airbyte.io/t/594)] - No obvious out‑of‑memory or crashloop patterns in server/worker logs. Because the knowledge sources don’t describe this exact
CreateNewAttemptNumber
500 or the OkHttp leak warnings as a known issue with a documented fix, I can’t go beyond these general, evidence‑backed steps. If you can pull and share (in your environment or with Airbyte support) the
airbyte-server
logs corresponding to one of these 500s, that should reveal the real server‑side exception, which is the next necessary step to resolve it.