<@U04S8PLJWMT> I have a Airbyte 1.8.1 K8S installa...
# ask-ai
j
@kapa.ai I have a Airbyte 1.8.1 K8S installation and one of my connections is breaking when reaching 300Mb of records read:
Copy code
2025-11-28 17:25:52 replication-orchestrator INFO Records read: 315000 (299 MB)
2025-11-28 17:26:14 replication-orchestrator INFO Records read: 320000 (305 MB)
2025-11-28 17:26:20 destination ERROR Killed
2025-11-28 17:26:20 replication-orchestrator ERROR DestinationReader error: 
2025-11-28 17:26:20 replication-orchestrator INFO DestinationReader finished.
k
This log pattern (
destination ERROR Killed
with no Airbyte stack trace) is consistent with the destination container being killed by the platform (Kubernetes) rather than Airbyte itself, most commonly due to out‑of‑memory (OOM) or resource limits. From the shared references, similar symptoms appear when: * The JVM/container for a connector is OOMKilled or forcibly terminated by the runtime, often with large datasets or wide rows. [[Cockroach OOM](https://discuss.airbyte.io/t/1261); [PG→PG OOM kill](https://discuss.airbyte.io/t/4539)] * Destination connectors buffer up to ~10k records per batch, which can blow up memory usage with large rows. [[Memory request/limit](https://discuss.airbyte.io/t/2214)] * On Kubernetes, jobs can hang or fail when a destination hits resource ceilings, especially with S3 or large‑record destinations. [[S3 Data Lake stalling](https://github.com/airbytehq/airbyte/issues/56926)] Because the knowledge sources you provided don’t include your specific destination type or your pod limits, they don’t contain enough information to pinpoint the exact root cause in your setup. However, they strongly indicate a resource/OOM problem at relatively small data volumes for some connectors. Given what is documented, you can: 1. Check Kubernetes events and pod status for the destination pod at the time of failure: * Look for
OOMKilled
or similar in
kubectl describe pod ...
. * This is exactly what users see in similar failures. [[Cockroach OOM](https://discuss.airbyte.io/t/1261); [PG→PG OOM kill](https://discuss.airbyte.io/t/4539)] 2. Review and increase job/pod memory limits: * Airbyte jobs use environment variables like
JOB_MAIN_CONTAINER_MEMORY_REQUEST
and
JOB_MAIN_CONTAINER_MEMORY_LIMIT
to cap worker memory; with big rows, these may be too low. [[Memory request/limit](https://discuss.airbyte.io/t/2214)] * If the destination pod is running with a low
resources.limits.memory
, increase it and re‑run the sync. 3. Re‑run with a smaller dataset: * As in the Postgres→Postgres example, a small (~300 MB) dataset succeeded while an ~8 GB one failed, helping confirm an OOM boundary. [[PG→PG OOM kill](https://discuss.airbyte.io/t/4539)] * If your connection always fails around ~300 MB, that strongly supports a memory cap as the limiting factor. 4. If the issue is with S3 Data Lake or another known large‑row destination, be aware there are open issues around memory leaks / stalling on large syncs, and gathering full pod logs plus
kubectl describe
output will be important if you open a GitHub issue. [[S3 Data Lake stalling](https://github.com/airbytehq/airbyte/issues/56926)] Because the sources you shared don’t document a generic “300 MB” limit or a specific bug in Airbyte 1.8.1, I can’t say more precisely what’s causing your specific failure beyond “likely resource/kill from Kubernetes”. The next concrete step is to: * capture the destination pod logs and
kubectl describe
for that pod around the failure, and * confirm whether the container is
OOMKilled
or otherwise terminated by K8s. If you can share which destination connector you’re using and the pod’s resource configuration, I can map that more directly to the documented behaviors above.
j
@kapa.ai I'm using AWS Datalake v0.1.58 for destination and Zendesk Support v4.10.12 as source, these are the sync logs up until the start of replication:
Copy code
2025-11-28 17:03:36 platform INFO APPLY Stage: BUILD — (workloadId=75a0f6df-41d1-47ec-bd01-6b1f9ae93545_25156_0_check)
2025-11-28 17:03:36 platform INFO APPLY Stage: CLAIM — (workloadId=75a0f6df-41d1-47ec-bd01-6b1f9ae93545_25156_0_check)
2025-11-28 17:03:36 platform INFO Claimed: true for workload 75a0f6df-41d1-47ec-bd01-6b1f9ae93545_25156_0_check via API in dataplane AUTOe067ebd9-9266-4cf3-a59d-87c949337c8a (87ecdcd2-c5a4-4233-a6cf-f61c659b87ec)
2025-11-28 17:03:36 platform INFO APPLY Stage: LOAD_SHED — (workloadId=75a0f6df-41d1-47ec-bd01-6b1f9ae93545_25156_0_check)
2025-11-28 17:03:36 platform INFO APPLY Stage: CHECK_STATUS — (workloadId=75a0f6df-41d1-47ec-bd01-6b1f9ae93545_25156_0_check)
2025-11-28 17:03:36 platform INFO No pod found running for workload 75a0f6df-41d1-47ec-bd01-6b1f9ae93545_25156_0_check
2025-11-28 17:03:36 platform INFO APPLY Stage: MUTEX — (workloadId=75a0f6df-41d1-47ec-bd01-6b1f9ae93545_25156_0_check)
2025-11-28 17:03:36 platform INFO No mutex key specified for workload: 75a0f6df-41d1-47ec-bd01-6b1f9ae93545_25156_0_check. Continuing...
2025-11-28 17:03:36 platform INFO APPLY Stage: ARCHITECTURE — (workloadId=75a0f6df-41d1-47ec-bd01-6b1f9ae93545_25156_0_check)
2025-11-28 17:03:36 platform INFO APPLY Stage: LAUNCH — (workloadId=75a0f6df-41d1-47ec-bd01-6b1f9ae93545_25156_0_check)
2025-11-28 17:03:36 platform INFO [initContainer] image: airbyte/workload-init-container:1.8.1 resources: ResourceRequirements(claims=[], limits={}, requests={}, additionalProperties={})
2025-11-28 17:03:44 platform INFO Attempting to update workload: 75a0f6df-41d1-47ec-bd01-6b1f9ae93545_25156_0_check to LAUNCHED.
2025-11-28 17:03:44 platform INFO Pipeline completed for workload: 75a0f6df-41d1-47ec-bd01-6b1f9ae93545_25156_0_check.
2025-11-28 17:03:48 platform INFO 
2025-11-28 17:03:48 platform INFO ----- START CHECK -----
2025-11-28 17:03:48 platform INFO 
2025-11-28 17:03:49 platform INFO Connector exited, processing output
2025-11-28 17:03:49 platform INFO Output file jobOutput.json found
2025-11-28 17:03:49 platform INFO Connector exited with exit code 0
2025-11-28 17:03:49 platform INFO Reading messages from protocol version 0.2.0
2025-11-28 17:03:50 platform INFO Check succeeded
2025-11-28 17:03:50 platform INFO Checking for optional control message...
2025-11-28 17:03:50 platform INFO Writing output of 75a0f6df-41d1-47ec-bd01-6b1f9ae93545_25156_0_check to the doc store
2025-11-28 17:03:50 platform INFO Marking workload 75a0f6df-41d1-47ec-bd01-6b1f9ae93545_25156_0_check as successful
2025-11-28 17:03:50 platform INFO 
2025-11-28 17:03:50 platform INFO Deliberately exiting process with code 0.
2025-11-28 17:03:50 platform INFO ----- END CHECK -----
2025-11-28 17:03:50 platform INFO 
2025-11-28 17:03:51 platform INFO APPLY Stage: BUILD — (workloadId=424b7ff2-07cd-4aaf-b956-986ae67f6e41_25156_0_check)
2025-11-28 17:03:51 platform INFO APPLY Stage: CLAIM — (workloadId=424b7ff2-07cd-4aaf-b956-986ae67f6e41_25156_0_check)
2025-11-28 17:03:51 platform INFO Claimed: true for workload 424b7ff2-07cd-4aaf-b956-986ae67f6e41_25156_0_check via API in dataplane AUTOe067ebd9-9266-4cf3-a59d-87c949337c8a (87ecdcd2-c5a4-4233-a6cf-f61c659b87ec)
2025-11-28 17:03:51 platform INFO APPLY Stage: LOAD_SHED — (workloadId=424b7ff2-07cd-4aaf-b956-986ae67f6e41_25156_0_check)
2025-11-28 17:03:51 platform INFO APPLY Stage: CHECK_STATUS — (workloadId=424b7ff2-07cd-4aaf-b956-986ae67f6e41_25156_0_check)
2025-11-28 17:03:51 platform INFO No pod found running for workload 424b7ff2-07cd-4aaf-b956-986ae67f6e41_25156_0_check
2025-11-28 17:03:51 platform INFO APPLY Stage: MUTEX — (workloadId=424b7ff2-07cd-4aaf-b956-986ae67f6e41_25156_0_check)
2025-11-28 17:03:51 platform INFO No mutex key specified for workload: 424b7ff2-07cd-4aaf-b956-986ae67f6e41_25156_0_check. Continuing...
2025-11-28 17:03:51 platform INFO APPLY Stage: ARCHITECTURE — (workloadId=424b7ff2-07cd-4aaf-b956-986ae67f6e41_25156_0_check)
2025-11-28 17:03:51 platform INFO APPLY Stage: LAUNCH — (workloadId=424b7ff2-07cd-4aaf-b956-986ae67f6e41_25156_0_check)
2025-11-28 17:03:51 platform INFO [initContainer] image: airbyte/workload-init-container:1.8.1 resources: ResourceRequirements(claims=[], limits={}, requests={}, additionalProperties={})
2025-11-28 17:04:00 platform INFO Attempting to update workload: 424b7ff2-07cd-4aaf-b956-986ae67f6e41_25156_0_check to LAUNCHED.
2025-11-28 17:04:00 platform INFO Pipeline completed for workload: 424b7ff2-07cd-4aaf-b956-986ae67f6e41_25156_0_check.
2025-11-28 17:04:04 platform INFO 
2025-11-28 17:04:04 platform INFO ----- START CHECK -----
2025-11-28 17:04:04 platform INFO 
2025-11-28 17:04:09 platform INFO Connector exited, processing output
2025-11-28 17:04:09 platform INFO Output file jobOutput.json found
2025-11-28 17:04:09 platform INFO Connector exited with exit code 0
2025-11-28 17:04:09 platform INFO Reading messages from protocol version 0.2.0
2025-11-28 17:04:09 platform INFO Resetting table landing_prd.airbyte_test_jpRay
2025-11-28 17:04:09 platform INFO Resetting table landing_prd.airbyte_test_jpRay
2025-11-28 17:04:09 platform INFO Deleting table landing_prd.airbyte_test_jpRay
2025-11-28 17:04:09 platform INFO Deleting objects in s3://totvs-cda-useast1-landing-prd/landing_prd/airbyte_test_jpRay/
2025-11-28 17:04:09 platform INFO Checking for optional control message...
2025-11-28 17:04:09 platform INFO Writing output of 424b7ff2-07cd-4aaf-b956-986ae67f6e41_25156_0_check to the doc store
2025-11-28 17:04:09 platform INFO Marking workload 424b7ff2-07cd-4aaf-b956-986ae67f6e41_25156_0_check as successful
2025-11-28 17:04:10 platform INFO 
2025-11-28 17:04:10 platform INFO Deliberately exiting process with code 0.
2025-11-28 17:04:10 platform INFO ----- END CHECK -----
2025-11-28 17:04:10 platform INFO 
2025-11-28 17:04:12 platform INFO APPLY Stage: BUILD — (workloadId=f82abdad-4c0d-438c-b4bd-eb658ddfade0_25156_0_sync)
2025-11-28 17:04:12 platform INFO APPLY Stage: CLAIM — (workloadId=f82abdad-4c0d-438c-b4bd-eb658ddfade0_25156_0_sync)
2025-11-28 17:04:12 platform INFO Claimed: true for workload f82abdad-4c0d-438c-b4bd-eb658ddfade0_25156_0_sync via API in dataplane AUTOe067ebd9-9266-4cf3-a59d-87c949337c8a (87ecdcd2-c5a4-4233-a6cf-f61c659b87ec)
2025-11-28 17:04:12 platform INFO APPLY Stage: LOAD_SHED — (workloadId=f82abdad-4c0d-438c-b4bd-eb658ddfade0_25156_0_sync)
2025-11-28 17:04:12 platform INFO APPLY Stage: CHECK_STATUS — (workloadId=f82abdad-4c0d-438c-b4bd-eb658ddfade0_25156_0_sync)
2025-11-28 17:04:12 platform INFO No pod found running for workload f82abdad-4c0d-438c-b4bd-eb658ddfade0_25156_0_sync
2025-11-28 17:04:12 platform INFO APPLY Stage: MUTEX — (workloadId=f82abdad-4c0d-438c-b4bd-eb658ddfade0_25156_0_sync)
2025-11-28 17:04:12 platform INFO Mutex key: f82abdad-4c0d-438c-b4bd-eb658ddfade0 specified for workload: f82abdad-4c0d-438c-b4bd-eb658ddfade0_25156_0_sync. Attempting to delete existing pods...
2025-11-28 17:04:12 platform INFO Existing pods for mutex key: f82abdad-4c0d-438c-b4bd-eb658ddfade0 deleted.
2025-11-28 17:04:12 platform INFO APPLY Stage: ARCHITECTURE — (workloadId=f82abdad-4c0d-438c-b4bd-eb658ddfade0_25156_0_sync)
2025-11-28 17:04:12 platform INFO APPLY Stage: LAUNCH — (workloadId=f82abdad-4c0d-438c-b4bd-eb658ddfade0_25156_0_sync)
2025-11-28 17:04:12 platform INFO [initContainer] image: airbyte/workload-init-container:1.8.1 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=2Gi, cpu=1}, additionalProperties={})
2025-11-28 17:04:12 platform INFO Launching replication pod: replication-job-25156-attempt-0 (selectors = {}) with containers:
2025-11-28 17:04:12 platform INFO [source] image: airbyte/source-zendesk-support:4.10.12 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=1Gi, cpu=0.2}, additionalProperties={})
2025-11-28 17:04:12 platform INFO [destination] image: airbyte/destination-aws-datalake:0.1.58 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=1Gi, cpu=0.2}, additionalProperties={})
2025-11-28 17:04:12 platform INFO [orchestrator] image: airbyte/container-orchestrator:1.8.1 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=2Gi, cpu=1}, additionalProperties={})
2025-11-28 17:04:27 platform INFO Attempting to update workload: f82abdad-4c0d-438c-b4bd-eb658ddfade0_25156_0_sync to LAUNCHED.
2025-11-28 17:04:27 platform INFO Pipeline completed for workload: f82abdad-4c0d-438c-b4bd-eb658ddfade0_25156_0_sync.
2025-11-28 17:04:40 replication-orchestrator INFO Running replication worker...
2025-11-28 17:04:40 replication-orchestrator INFO Starting replication worker. job id: 25156 attempt: 0
2025-11-28 17:04:40 replication-orchestrator INFO 
2025-11-28 17:04:40 replication-orchestrator INFO ----- START REPLICATION -----
k
A message from kapa.ai
The log snippet confirms that: - Both source and destination pods are launched with 2Gi memory limit / 1Gi request. [[Configuring resources](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-connector-resources#debugging)] - The sync then proceeds into replication (
----- START REPLICATION -----
) and later (from your earlier message) fails around 300 MB read with
destination ERROR Killed
and no Java stack trace. This pattern matches what’s seen when the destination container is being killed by Kubernetes (typically OOMKilled), not by Airbyte itself: the orchestrator just sees the destination disappear and logs a generic `Killed`/“DestinationReader error”. Similar symptoms occur in other large/long‑running syncs, including S3‑based destinations, where the destination pod hits its memory ceiling and is terminated by K8s. [[S3 Data Lake OOMKilled](https://github.com/airbytehq/airbyte/issues/70247); [large sync stalls/timeout](https://github.com/airbytehq/airbyte/issues/56926)] From the provided docs/issues, there is no specific known hard limit at ~300 MB or a Zendesk→AWS Datalake bug documented; the sources instead consistently point to resource limits / JVM heap sizing inside the destination container as the root cause for similar failures. [[Configuring resources](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-connector-resources#debugging); [PG→PG OOM‑like kill](https://discuss.airbyte.io/t/4539)] ### What you can verify next 1. Confirm OOMKilled in Kubernetes: - Run
kubectl describe pod <replication-job-... destination pod>
for the failing attempt and look under
Last State
/
Reason
. - In similar cases, users see
Reason: OOMKilled
for the destination, with source/orchestrator exiting cleanly. [[S3 Data Lake OOMKilled](https://github.com/airbytehq/airbyte/issues/70247)] 2. Increase destination memory limits: - Right now AWS Datalake is running with: -
limits={memory=2Gi, cpu=1}
-
requests={memory=1Gi, cpu=0.2}
- The Airbyte docs explicitly say you can and should bump connector resources when a job is running out of memory, and you can verify the new settings in logs just like the snippet you shared. [[Configuring resources](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-connector-resources#debugging)] - Try raising the destination memory limit (e.g., 4–8 Gi) and re‑run the sync. Keep source/orchestrator at current values initially to isolate the effect. 3. Re‑run and watch memory usage & logs: - After changing connector resources, confirm via the job logs that the new limits are applied (you should see updated
resources:
lines for the destination). [[Configuring resources](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-connector-resources#debugging)] - If the failure boundary moves (e.g., now fails at a higher GB count or no longer fails), that strongly confirms a memory‑limit issue. If, after increasing memory, the destination is still getting killed around similar data volumes and Kubernetes does not report
OOMKilled
, then we might instead be hitting a connector‑specific bug like the open AWS/S3 data‑lake issues where long‑running writes stall or leak memory; in that case, gathering: - full job logs, and -
kubectl describe
+
kubectl logs
for the destination pod and attaching them to a new or existing GitHub issue for
destination-aws-datalake
would be the next step, referencing the behavior seen in the S3 Data Lake issues. [[S3 Data Lake OOMKilled](https://github.com/airbytehq/airbyte/issues/70247); [Sync stalling](https://github.com/airbytehq/airbyte/issues/56926)] From the current knowledge sources I can’t be more specific than “very likely memory / K8s kill on the destination”, but increasing the destination’s memory limit and confirming the pod’s termination reason in Kubernetes is the most concrete, documented path forward.
j
@kapa.ai I'm trying to run a sync to get the job id, but I'm getting this erro on the status screen:
Copy code
message='java.io.IOException: HTTP error: 500 Internal Server Error', type='io.airbyte.commons.temporal.exception.RetryableException', nonRetryable=false
k
A message from kapa.ai
That error means the Temporal workflow retried because the Airbyte server returned an HTTP 500 while the job status was being fetched. It’s a platform error, not a connector‑specific one. From the provided sources, very similar 500s are reported when: - The API that lists enterprise sources fails with
Internal Server Error: java.io.IOException: HTTP error fetching enterprise sources
. [[Error 500 fetching sources](https://github.com/airbytehq/airbyte/issues/53609); [Same endpoint 500](https://github.com/airbytehq/airbyte/issues/54058)] - Kubernetes‑based deployments have backend services in a bad state and the UI shows generic 5xx (500/502) errors when configuring sources or connections. [[K8s test source 500](https://github.com/airbytehq/airbyte/issues/48809); [502 fixed by restarting ABCTL services](https://github.com/airbytehq/airbyte/issues/66692)] The knowledge sources do not give a single root cause for your exact message, but they do suggest a few concrete steps: 1. Check server / deployment health - Look at the
airbyte-server
(and, on 1.8.x, the
airbyte-abctl-*
components if you’re using the new control plane) pod logs around the time you see the 500. In similar cases, the server logs contain the underlying `IOException`/HTTP error. [[K8s test source 500](https://github.com/airbytehq/airbyte/issues/48809); [Error fetching sources](https://github.com/airbytehq/airbyte/issues/53609)] 2. Restart control‑plane services (if using Helm / ABCTL on K8s 1.8.x) In one 1.8.3 Kubernetes case, “Airbyte is temporarily unavailable. Please try again. (HTTP 502)” during source/destination configuration was solved by restarting the ABCTL worker and workload API server: [[1.8.3 502 fix](https://github.com/airbytehq/airbyte/issues/66692)]
Copy code
bash    kubectl -n airbyte-abctl rollout restart deployment airbyte-abctl-worker    kubectl -n airbyte-abctl rollout restart deployment airbyte-abctl-workload-api-server     kubectl -n airbyte-abctl rollout status deployment airbyte-abctl-worker    kubectl -n airbyte-abctl rollout status deployment airbyte-abctl-workload-api-server
Even though your version is 1.8.1, the same pattern may apply if you are using the ABCTL‑based deployment. 3. Re‑run the sync after services are healthy Since the exception is marked
RetryableException
and
nonRetryable=false
, once the backend is healthy again, starting a new sync should allow you to get the job id normally. Because the sources don’t show a specific Temporal/job‑status 500 bug in 1.8.1, I can’t point to an exact known defect. If, after checking logs and restarting relevant services, the 500 persists, the next step would be to open a GitHub issue with: - your deployment type (Helm / ABCTL / other), - Airbyte version (1.8.1), - and the server logs around the time of the 500, similar to how it was done in the linked issues. [[K8s test source 500](https://github.com/airbytehq/airbyte/issues/48809); [Error fetching sources](https://github.com/airbytehq/airbyte/issues/53609)]
j
@kapa.ai How can I see the airbyte-server pod logs?
k
A message from kapa.ai
On Kubernetes you can use
kubectl
to view the
airbyte-server
logs. If you installed Airbyte with
abctl
, the namespace is typically `airbyte-abctl`; you can list pods and then get logs as shown in the developing‑locally docs:
Copy code
bash # List pods in the abctl namespace kubectl get pods -n airbyte-abctl  # View logs for the server pod (replace with your actual pod name) kubectl logs -n airbyte-abctl airbyte-abctl-server-74465db7fd-gk25q
[[Developing locally](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#prerequisites)] If your deployment uses a different namespace or release name (for example,
airbyte
), adjust the namespace and pod name accordingly:
Copy code
bash kubectl get pods -n <your-namespace> kubectl logs -n <your-namespace> <airbyte-server-pod-name>
j
@kapa.ai These are my pods:
Copy code
[root@CDA-AIRBYTE-TESP3 ~]# kubectl get pods -n airbyte
NAME                                                READY   STATUS      RESTARTS   AGE
airbyte-airbyte-bootloader                          0/1     Completed   0          38d
airbyte-connector-builder-server-7567cbdf47-szsvd   1/1     Running     0          2d22h
airbyte-cron-66d465bd8b-8g8s2                       1/1     Running     0          2d22h
airbyte-db-0                                        1/1     Running     0          2d22h
airbyte-minio-0                                     1/1     Running     0          2d22h
airbyte-server-7dbb45cf5d-6szts                     1/1     Running     0          2d22h
airbyte-temporal-574df4f985-8fnxr                   1/1     Running     0          2d22h
airbyte-worker-58fbc95cf4-5krbn                     1/1     Running     0          2d22h
airbyte-workload-api-server-84d48b5465-xzp6v        1/1     Running     0          2d22h
airbyte-workload-launcher-6f7bbf4cd4-w9w72          1/1     Running     0          2d22h
replication-job-26647-attempt-0                     0/3     Completed   0          9m19s
replication-job-26648-attempt-0                     0/3     Completed   0          9m18s
replication-job-26649-attempt-0                     0/3     Completed   0          9m19s
And the logs are too big to analyze in my only ssh linux airbyte installation
k
A message from kapa.ai
Given your pod list, your
airbyte-server
pod is:
Copy code
text airbyte-server-7dbb45cf5d-6szts
To view its logs:
Copy code
bash kubectl logs -n airbyte airbyte-server-7dbb45cf5d-6szts
This follows the same pattern as the docs’ example, just using your namespace (
airbyte
) and pod name instead of the
airbyte-abctl-*
ones. [[Developing locally](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#prerequisites)] Since logs are large, a few options to make them manageable: - Show only the end of the log (often where the error is):
Copy code
bash   kubectl logs -n airbyte airbyte-server-7dbb45cf5d-6szts --tail=500
- Follow logs in real time while reproducing the error:
Copy code
bash   kubectl logs -n airbyte -f airbyte-server-7dbb45cf5d-6szts
- Filter for errors (if you have
grep
):
Copy code
bash   kubectl logs -n airbyte airbyte-server-7dbb45cf5d-6szts | grep -i "error"
If you can copy/paste just the section around the time of the 500 error (from the server log), I can help interpret it.
j
@kapa.ai The error is not happening as the sync is executed, it is showing on the main page of the connection. This is happening only on this connection, all others are running fine. When I try to run a sync on this connection nothing happens.
k
A message from kapa.ai
That symptom (connection card showing an error, clicking “Sync” does nothing, while other connections are fine) points to a connection‑level / backend error rather than a problem inside the sync job itself. From the materials you shared, we know: - When the UI shows `HTTP 500`–style messages on connection pages, it usually means the server API call behind that screen is failing, not that a job has started. Similar behavior appears when listing enterprise sources or configuring connections returns 500s. [[Error 500 fetching sources](https://github.com/airbytehq/airbyte/issues/53609); [HTTP 500 internalServerError](https://github.com/airbytehq/airbyte/issues/54058)] - In Kubernetes installations, these 5xx errors often come from a single bad resource or a backend service in an unhealthy state, while other connections continue to work. [[Kubernetes 502/504 on some connections](https://discuss.airbyte.io/t/2394); [502 when creating connection](https://discuss.airbyte.io/t/1207); [502 during check on K8s](https://discuss.airbyte.io/t/594)] However, the knowledge sources do not describe a specific, known bug for “one connection UI stuck, sync won’t start, message=`java.io.IOException: HTTP error: 500`”. So I can’t cite an exact root cause or one guaranteed fix. Based on the docs and issues, here’s what you can concretely do next: 1. Check server logs exactly when you open that connection’s page Since you already know how to access
airbyte-server
logs, run:
Copy code
bash    kubectl logs -n airbyte airbyte-server-7dbb45cf5d-6szts --tail=500 -f
Then: - Open the problematic connection’s Status page and wait for the error to appear. - Look for a stack trace or log line including that
java.io.IOException: HTTP error: 500
at that time. In similar 500 cases, the server logs include more details, e.g., failing to fetch enterprise sources or failing to reach another internal service. [[Error 500 fetching sources](https://github.com/airbytehq/airbyte/issues/53609); [HTTP 500 internalServerError](https://github.com/airbytehq/airbyte/issues/54058)] 2. Compare with a healthy connection While the log tail is still running: - Open a working connection’s Status page and trigger a sync. - Note which API endpoints and log messages appear. - Then open the broken connection and see what’s different (e.g., a particular endpoint or SQL query failing). The troubleshooting docs describe using the Status page to isolate failing connections, though they don’t go down to this UI/API level. [[Connection troubleshooting](https://docs.airbyte.com/platform/1.8/cloud/managing-airbyte-cloud/review-connection-status#connection-troubleshooting)] 3. If you see a clear 500 cause in the logs, act on that Examples from the sources: - If it’s an issue fetching enterprise sources (similar to
/source_definitions/list_enterprise_source_stubs
), the documented workaround is to point
ENTERPRISE_SOURCE_STUBS_URL
to an empty
[]
JSON
so that endpoint no longer fails. [[Error 500 fetching sources](https://github.com/airbytehq/airbyte/issues/53609)] - If it’s a connection to another internal service (Temporal, Minio, database, etc.), you may see connection‑refused or timeout errors similar to other K8s reports, and you’d need to address that service specifically. [[K8s 502/temporal not reachable](https://discuss.airbyte.io/t/594); [Minio connectivity causing 502](https://discuss.airbyte.io/t/1207)] 4. If the logs don’t make it obvious The sources you provided don’t include a documented fix for “single bad connection stuck in UI” on 1.8.1, so if the server logs show a generic 500 without a clear target (or you’re unsure how to interpret them), the best next step is: - Capture: - the exact error shown in the UI, - the matching server log chunk (a few dozen lines around the 500), - and your deployment/version details (K8s, Airbyte 1.8.1). - Open a GitHub issue or forum topic following the style in the linked 500‑error reports. [[Error 500 fetching sources](https://github.com/airbytehq/airbyte/issues/53609); [HTTP 500 internalServerError](https://github.com/airbytehq/airbyte/issues/54058)] Because the knowledge sources don’t describe this exact behavior on the connection main page, I can’t definitively say whether this is a corrupted job/connection record, a specific connector bug, or a platform defect in 1.8.1. The server log snippet around the 500 is what will let you or the Airbyte team move from “generic HTTP 500” to an actionable fix.
j
@kapa.ai This is the log of when I enter the connection page:
Copy code
2025-12-02 13:26:22,263 [io-executor-thread-2]  INFO    i.g.o.k.s.i.Slf4jLogger(info):82 - Attempt stats: {JobAttemptPair(id=25198, attemptNumber=0)=AttemptStats(combinedStats=io.airbyte.config.SyncStats@1c1fac87[bytesCommitted=1106718,bytesEmitted=322206940,destinationStateMessagesEmitted=<null>,destinationWriteEndTime=<null>,destinationWriteStartTime=<null>,estimatedBytes=<null>,estimatedRecords=<null>,meanSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBetweenStateMessageEmittedandCommitted=<null>,meanSecondsBetweenStateMessageEmittedandCommitted=<null>,recordsEmitted=328290,recordsCommitted=476,recordsFilteredOut=<null>,bytesFilteredOut=<null>,replicationEndTime=<null>,replicationStartTime=<null>,sourceReadEndTime=<null>,sourceReadStartTime=<null>,sourceStateMessagesEmitted=<null>,discoverSchemaEndTime=<null>,discoverSchemaStartTime=<null>,recordsRejected=<null>,additionalProperties={}], perStreamStats=[io.airbyte.config.StreamSyncStats@327e9ae0[streamName=users,streamNamespace=<null>,stats=io.airbyte.config.SyncStats@2ba24079[bytesCommitted=915530,bytesEmitted=915530,destinationStateMessagesEmitted=<null>,destinationWriteEndTime=<null>,destinationWriteStartTime=<null>,estimatedBytes=0,estimatedRecords=0,meanSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBetweenStateMessageEmittedandCommitted=<null>,meanSecondsBetweenStateMessageEmittedandCommitted=<null>,recordsEmitted=434,recordsCommitted=434,recordsFilteredOut=<null>,bytesFilteredOut=<null>,replicationEndTime=<null>,replicationStartTime=<null>,sourceReadEndTime=<null>,sourceReadStartTime=<null>,sourceStateMessagesEmitted=<null>,discoverSchemaEndTime=<null>,discoverSchemaStartTime=<null>,recordsRejected=<null>,additionalProperties={}],wasBackfilled=false,wasResumed=true,additionalProperties={}], io.airbyte.config.StreamSyncStats@4bbe3ac4[streamName=group_memberships,streamNamespace=<null>,stats=io.airbyte.config.SyncStats@2aa4dca0[bytesCommitted=944,bytesEmitted=944,destinationStateMessagesEmitted=<null>,destinationWriteEndTime=<null>,destinationWriteStartTime=<null>,estimatedBytes=0,estimatedRecords=0,meanSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBetweenStateMessageEmittedandCommitted=<null>,meanSecondsBetweenStateMessageEmittedandCommitted=<null>,recordsEmitted=4,recordsCommitted=4,recordsFilteredOut=<null>,bytesFilteredOut=<null>,replicationEndTime=<null>,replicationStartTime=<null>,sourceReadEndTime=<null>,sourceReadStartTime=<null>,sourceStateMessagesEmitted=<null>,discoverSchemaEndTime=<null>,discoverSchemaStartTime=<null>,recordsRejected=<null>,additionalProperties={}],wasBackfilled=false,wasResumed=true,additionalProperties={}], io.airbyte.config.StreamSyncStats@6ccbea8e[streamName=groups,streamNamespace=<null>,stats=io.airbyte.config.SyncStats@55d3dee6[bytesCommitted=0,bytesEmitted=0,destinationStateMessagesEmitted=<null>,destinationWriteEndTime=<null>,destinationWriteStartTime=<null>,estimatedBytes=0,estimatedRecords=0,meanSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBetweenStateMessageEmittedandCommitted=<null>,meanSecondsBetweenStateMessageEmittedandCommitted=<null>,recordsEmitted=0,recordsCommitted=0,recordsFilteredOut=<null>,bytesFilteredOut=<null>,replicationEndTime=<null>,replicationStartTime=<null>,sourceReadEndTime=<null>,sourceReadStartTime=<null>,sourceStateMessagesEmitted=<null>,discoverSchemaEndTime=<null>,discoverSchemaStartTime=<null>,recordsRejected=<null>,additionalProperties={}],wasBackfilled=false,wasResumed=true,additionalProperties={}], io.airbyte.config.StreamSyncStats@6f01a682[streamName=brands,streamNamespace=<null>,stats=io.airbyte.config.SyncStats@634c261e[bytesCommitted=10245,bytesEmitted=10245,destinationStateMessagesEmitted=<null>,destinationWriteEndTime=<null>,destinationWriteStartTime=<null>,estimatedBytes=0,estimatedRecords=0,meanSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBetweenStateMessageEmittedandCommitted=<null>,meanSecondsBetweenStateMessageEmittedandCommitted=<null>,recordsEmitted=6,recordsCommitted=6,recordsFilteredOut=<null>,bytesFilteredOut=<null>,replicationEndTime=<null>,replicationStartTime=<null>,sourceReadEndTime=<null>,sourceReadStartTime=<null>,sourceStateMessagesEmitted=<null>,discoverSchemaEndTime=<null>,discoverSchemaStartTime=<null>,recordsRejected=<null>,additionalProperties={}],wasBackfilled=false,wasResumed=false,additionalProperties={}], io.airbyte.config.StreamSyncStats@50603f63[streamName=sla_policies,streamNamespace=<null>,stats=io.airbyte.config.SyncStats@1b8fff67[bytesCommitted=22270,bytesEmitted=22270,destinationStateMessagesEmitted=<null>,destinationWriteEndTime=<null>,destinationWriteStartTime=<null>,estimatedBytes=0,estimatedRecords=0,meanSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBetweenStateMessageEmittedandCommitted=<null>,meanSecondsBetweenStateMessageEmittedandCommitted=<null>,recordsEmitted=1,recordsCommitted=1,recordsFilteredOut=<null>,bytesFilteredOut=<null>,replicationEndTime=<null>,replicationStartTime=<null>,sourceReadEndTime=<null>,sourceReadStartTime=<null>,sourceStateMessagesEmitted=<null>,discoverSchemaEndTime=<null>,discoverSchemaStartTime=<null>,recordsRejected=<null>,additionalProperties={}],wasBackfilled=false,wasResumed=true,additionalProperties={}], io.airbyte.config.StreamSyncStats@79586f22[streamName=ticket_metric_events,streamNamespace=<null>,stats=io.airbyte.config.SyncStats@7ba6cac1[bytesCommitted=0,bytesEmitted=43595047,destinationStateMessagesEmitted=<null>,destinationWriteEndTime=<null>,destinationWriteStartTime=<null>,estimatedBytes=0,estimatedRecords=0,meanSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBetweenStateMessageEmittedandCommitted=<null>,meanSecondsBetweenStateMessageEmittedandCommitted=<null>,recordsEmitted=321400,recordsCommitted=0,recordsFilteredOut=<null>,bytesFilteredOut=<null>,replicationEndTime=<null>,replicationStartTime=<null>,sourceReadEndTime=<null>,sourceReadStartTime=<null>,sourceStateMessagesEmitted=<null>,discoverSchemaEndTime=<null>,discoverSchemaStartTime=<null>,recordsRejected=<null>,additionalProperties={}],wasBackfilled=false,wasResumed=true,additionalProperties={}], io.airbyte.config.StreamSyncStats@2a6c69f1[streamName=organization_fields,streamNamespace=<null>,stats=io.airbyte.config.SyncStats@4efccf1c[bytesCommitted=0,bytesEmitted=0,destinationStateMessagesEmitted=<null>,destinationWriteEndTime=<null>,destinationWriteStartTime=<null>,estimatedBytes=0,estimatedRecords=0,meanSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBetweenStateMessageEmittedandCommitted=<null>,meanSecondsBetweenStateMessageEmittedandCommitted=<null>,recordsEmitted=0,recordsCommitted=0,recordsFilteredOut=<null>,bytesFilteredOut=<null>,replicationEndTime=<null>,replicationStartTime=<null>,sourceReadEndTime=<null>,sourceReadStartTime=<null>,sourceStateMessagesEmitted=<null>,discoverSchemaEndTime=<null>,discoverSchemaStartTime=<null>,recordsRejected=<null>,additionalProperties={}],wasBackfilled=false,wasResumed=true,additionalProperties={}], io.airbyte.config.StreamSyncStats@7467989e[streamName=organizations,streamNamespace=<null>,stats=io.airbyte.config.SyncStats@395ab411[bytesCommitted=157729,bytesEmitted=157729,destinationStateMessagesEmitted=<null>,destinationWriteEndTime=<null>,destinationWriteStartTime=<null>,estimatedBytes=0,estimatedRecords=0,meanSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBetweenStateMessageEmittedandCommitted=<null>,meanSecondsBetweenStateMessageEmittedandCommitted=<null>,recordsEmitted=31,recordsCommitted=31,recordsFilteredOut=<null>,bytesFilteredOut=<null>,replicationEndTime=<null>,replicationStartTime=<null>,sourceReadEndTime=<null>,sourceReadStartTime=<null>,sourceStateMessagesEmitted=<null>,discoverSchemaEndTime=<null>,discoverSchemaStartTime=<null>,recordsRejected=<null>,additionalProperties={}],wasBackfilled=false,wasResumed=true,additionalProperties={}], io.airbyte.config.StreamSyncStats@70b0712a[streamName=tickets,streamNamespace=<null>,stats=io.airbyte.config.SyncStats@589cc40f[bytesCommitted=0,bytesEmitted=277505175,destinationStateMessagesEmitted=<null>,destinationWriteEndTime=<null>,destinationWriteStartTime=<null>,estimatedBytes=0,estimatedRecords=0,meanSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBeforeSourceStateMessageEmitted=<null>,maxSecondsBetweenStateMessageEmittedandCommitted=<null>,meanSecondsBetweenStateMessageEmittedandCommitted=<null>,recordsEmitted=6414,recordsCommitted=0,recordsFilteredOut=<null>,bytesFilteredOut=<null>,replicationEndTime=<null>,replicationStartTime=<null>,sourceReadEndTime=<null>,sourceReadStartTime=<null>,sourceStateMessagesEmitted=<null>,discoverSchemaEndTime=<null>,discoverSchemaStartTime=<null>,recordsRejected=<null>,additionalProperties={}],wasBackfilled=false,wasResumed=true,additionalProperties={}]])}
2025-12-02 13:26:22,264 [io-executor-thread-2]  INFO    i.g.o.k.s.i.Slf4jLogger(info):82 - Hydrating job 25198
k
A message from kapa.ai
The log you shared shows that when you open the connection page, the server is: - Loading stats for job 25198:
Attempt stats: {JobAttemptPair(id=25198, attemptNumber=0)=...}
- Then:
Hydrating job 25198
So, the UI call that renders the Status page is trying to fetch and “hydrate” that job, and somewhere after this step the backend returns the
java.io.IOException: HTTP error: 500 Internal Server Error
you see in the UI. From the knowledge you provided: - Airbyte’s Status page always pulls recent sync/job history to show “Records loaded” and per‑stream stats. [[Connection troubleshooting](https://docs.airbyte.com/platform/1.8/cloud/managing-airbyte-cloud/review-connection-status#connection-troubleshooting)] - There is a known class of issues where a killed / stuck job causes the server’s job‑conversion code to blow up when the UI tries to read it, e.g. “Sync attempt with killed pod hangs eternally”, with stack traces in
JobConverter.getSynchronousJobRead
/
SchedulerHandler
. [[Killed pod hangs](https://github.com/airbytehq/airbyte/issues/48879)] - There are also open issues where large Zendesk Support syncs get stuck (e.g. around the
updated_at
cursor or very large responses), leaving jobs in a bad/unfinished state. [[Zendesk cursor issue](https://github.com/airbytehq/airbyte/issues/65543); [Zendesk gateway timeout](https://github.com/airbytehq/airbyte/issues/52059)] - And open issues where S3/AWS data lake–style destinations stall or are killed on large/long‑running syncs, leaving jobs with incomplete stats. [[S3 data lake stalling](https://github.com/airbytehq/airbyte/issues/56926)] Putting this together with your situation: - Job
25198
clearly has lots of bytes emitted but almost no bytes committed for the heavy streams like
tickets
and
ticket_metric_events
in the stats you pasted, which matches a failed or killed long‑running sync. - When the connection page tries to hydrate that job, something about that job record (possibly missing fields due to the abrupt kill) is causing a server‑side failure; that 500 is then wrapped as the
java.io.IOException: HTTP error: 500
you see. The knowledge sources do not show a documented, step‑by‑step fix for “broken connection page due to a bad job record”, so I can’t give you a guaranteed remediation path. But based on the similar “killed pod hangs job forever” report, the most realistic ways forward are: 1. Confirm the server‑side stack trace for job 25198 In
airbyte-server
logs, directly after the “Hydrating job 25198” line, you should see a Java exception that explains the 500 (likely somewhere in
JobConverter
,
SchedulerHandler
or the web backend handlers, similar to the killed‑pod issue). [[Killed pod hangs](https://github.com/airbytehq/airbyte/issues/48879)] That exact stack trace is what the Airbyte team used to identify the “killed pod hangs eternally” bug, and it’s what you’ll need if you open a bug report. 2. Work around the broken job The docs and issues you shared don’t document an admin‑level way to delete or “forget” a single bad job, so I can’t tell you a supported SQL statement or API call to clean up job
25198
. I can only infer, based on the killed‑pod issue, that this is likely what needs to happen. Practically, this means: - Capture: - the 500 you see in the UI, - the full server stack trace after “Hydrating job 25198”, - and the fact that this job corresponds to a Zendesk → AWS Datalake sync that was killed around ~300 MB. - Open a GitHub issue against the Airbyte platform (and optionally reference the similar “Sync attempt with killed pod hangs eternally” report). [[Killed pod hangs](https://github.com/airbytehq/airbyte/issues/48879)] 3. As a temporary mitigation Since only this connection is affected and other connections work: - You can create a new connection from the same source and destination, with identical configuration, to resume data loading while the bad job/connection is being investigated. - The knowledge you shared doesn’t state this explicitly as a workaround, but it is implied by the fact that other connections and jobs are independent in the job history model, and only this one is failing UI hydration. Because the available sources don’t include a documented “repair” operation for a single broken job/connection, I can’t provide a definitive in‑place fix. The next concrete step is to extract the stack trace right after “Hydrating job 25198” from
airbyte-server
logs and attach it to a GitHub issue so the team can address the underlying bug in job hydration for failed/killed long‑running syncs.
j
@kapa.ai Nothing shows on the logs after the hydration line
k
A message from kapa.ai
Given the logs you shared and the docs/issues available, here’s what we can say with confidence and what you can do next. 1. Hydration line with no error in server logs You see:
Copy code
text    ... INFO - Hydrating job 25198
and then nothing else in
airbyte-server
logs, while the UI shows:
Copy code
text    java.io.IOException: HTTP error: 500 Internal Server Error
None of the provided docs or issues describe a case where the server logs only “Hydrating job …” and then silently fails; in all the 500‑error reports we have, there is an explicit stack trace logged on the server side. [[Error 500 fetching sources](https://github.com/airbytehq/airbyte/issues/53609)] So, from the knowledge sources alone, there is no documented root cause matching “single connection UI broken, hydration logged, but no visible server stack trace”. 2. What is documented and still applies - The Status page always loads recent job history and per‑attempt stats when you open a connection. [[Connection troubleshooting](https://docs.airbyte.com/platform/1.8/cloud/managing-airbyte-cloud/review-connection-status#connection-troubleshooting)] - Your stats for job
25198
clearly show a large, unfinished sync (hundreds of MB emitted, almost nothing committed on
tickets
/
ticket_metric_events
), which aligns with the earlier
destination ERROR Killed
around ~300 MB. - There are known issues where large Zendesk Support syncs or S3‑style destinations get stuck or fail in odd ways, but they’re documented at the sync/job level, not as “connection page 500s”. [[Zendesk cursor issue](https://github.com/airbytehq/airbyte/issues/65543); [S3 data lake stalling](https://github.com/airbytehq/airbyte/issues/56926)] None of the shared docs describe tools to “repair” a single broken job or connection in‑place (e.g., via API/SQL), so I cannot provide a supported cleanup procedure based on these sources. 3. **Concrete options you can take now** Based on what is documented, and staying within that: - Create a new connection between the same Zendesk source and AWS Datalake destination and try a sync there. - Connections are independent; breaking one does not affect others, which is consistent with your observation that only this connection is failing while others run. The docs assume this when they talk about per‑connection troubleshooting on the Status page. [[Connection troubleshooting](https://docs.airbyte.com/platform/1.8/cloud/managing-airbyte-cloud/review-connection-status#connection-troubleshooting)] - This won’t fix the broken connection, but it should let you resume data loading. - Capture more targeted server logs when the UI shows the 500 - Even though you don’t see more lines immediately after “Hydrating job 25198”, it’s worth: 1. Running a short, tailed log:
Copy code
bash           kubectl logs -n airbyte airbyte-server-7dbb45cf5d-6szts --since=5m
2. Then opening the connection page again. - In prior 500‑error issues, the server logs did contain the underlying exception, which was key for the Airbyte team to fix it. [[Error 500 fetching sources](https://github.com/airbytehq/airbyte/issues/53609)] - Open an issue with the details - Since the knowledge sources don’t document any known bug that fully matches what you see (UI 500 only for a specific connection, hydration message but no visible stack trace), the recommended path is to open a GitHub issue with: - Airbyte version and deployment (1.8.1, Kubernetes), - Source/destination:
source-zendesk-support 4.10.12
→
destination-aws-datalake 0.1.58
, - The job stats snippet for job
25198
, - The fact that entering this connection’s page shows a 500 while other connections are fine, - Any additional
airbyte-server
log lines around “Hydrating job 25198”, even if they’re not obviously errors. - This follows the pattern used for other 500‑level platform bugs. [[Error 500 fetching sources](https://github.com/airbytehq/airbyte/issues/53609); [Similar internalServerError](https://github.com/airbytehq/airbyte/issues/54058)] Because the provided knowledge does not include a documented fix for this exact situation, I can’t honestly give you a precise remediation beyond: - work around it by recreating the connection, and - escalate it with the details above so the platform team can investigate the specific failure path for hydrating job
25198
.
j
@kapa.ai After a while these logs showed:
Copy code
2025-12-02 13:39:17,406 [io-executor-thread-6]  INFO    i.g.o.k.s.i.Slf4jLogger(info):82 - Hydrating job 25198
2025-12-02 13:40:00,923 [scheduler-executor-thread-24]  INFO    i.g.o.k.s.i.Slf4jLogger(info):86 - Found the following streams to reset for connection 223041f0-4364-4956-9769-58a9637dbbf0: []
2025-12-02 13:40:00,943 [scheduler-executor-thread-24]  INFO    i.a.c.h.ResourceRequirementsUtils(getResourceRequirementsForJobType):64 - Merged resource requirements. mergedResourceReqs=io.airbyte.config.ResourceRequirements@2ea6e1f2[cpuRequest=1,cpuLimit=2,memoryRequest=1Gi,memoryLimit=2Gi,ephemeralStorageRequest=<null>,ephemeralStorageLimit=<null>,additionalProperties={}] connectionResourceReqs=io.airbyte.config.ResourceRequirements@45eb3998[cpuRequest=<null>,cpuLimit=<null>,memoryRequest=<null>,memoryLimit=<null>,ephemeralStorageRequest=<null>,ephemeralStorageLimit=<null>,additionalProperties={}] actorResourceReqs=io.airbyte.config.ScopedResourceRequirements@2c383bd0[_default=<null>,jobSpecific=[],additionalProperties={}] actorDefinitionResourceReqs=null workerDefaultResourceReqs=io.airbyte.config.ResourceRequirements@607f2e06[cpuRequest=1,cpuLimit=2,memoryRequest=1Gi,memoryLimit=2Gi,ephemeralStorageRequest=<null>,ephemeralStorageLimit=<null>,additionalProperties={}] jobType=sync
2025-12-02 13:40:00,943 [scheduler-executor-thread-24]  INFO    i.a.c.h.ResourceRequirementsUtils(getResourceRequirementsForJobType):64 - Merged resource requirements. mergedResourceReqs=io.airbyte.config.ResourceRequirements@79cfea8a[cpuRequest=1,cpuLimit=2,memoryRequest=1Gi,memoryLimit=2Gi,ephemeralStorageRequest=<null>,ephemeralStorageLimit=<null>,additionalProperties={}] connectionResourceReqs=io.airbyte.config.ResourceRequirements@45eb3998[cpuRequest=<null>,cpuLimit=<null>,memoryRequest=<null>,memoryLimit=<null>,ephemeralStorageRequest=<null>,ephemeralStorageLimit=<null>,additionalProperties={}] actorResourceReqs=io.airbyte.config.ScopedResourceRequirements@65f42bd2[_default=<null>,jobSpecific=[],additionalProperties={}] actorDefinitionResourceReqs=null workerDefaultResourceReqs=io.airbyte.config.ResourceRequirements@8ff6a07[cpuRequest=1,cpuLimit=2,memoryRequest=1Gi,memoryLimit=2Gi,ephemeralStorageRequest=<null>,ephemeralStorageLimit=<null>,additionalProperties={}] jobType=sync
2025-12-02 13:40:00,943 [scheduler-executor-thread-24]  INFO    i.a.p.j.DefaultJobPersistence(enqueueJob):107 - enqueuing pending job for scope: 223041f0-4364-4956-9769-58a9637dbbf0
2025-12-02 13:40:00,950 [scheduler-executor-thread-24]  INFO    i.g.o.k.KLogger$DefaultImpls(info):534 - New job created, with id: 26961
2025-12-02 13:40:01,227 [io-executor-thread-6]  INFO    i.a.s.s.CommandService(createCommand):426 - The workload 4e962333-3041-40a9-858c-4e250cef2207_1764633600000_discover already exists for command discover_26961_0_4e962333-3041-40a9-858c-4e250cef2207, continuing
2025-12-02 13:40:01,435 [io-executor-thread-1]  INFO    i.a.c.s.h.ConnectionsHandler(applySchemaChange):1962 - Applying schema change for connection '223041f0-4364-4956-9769-58a9637dbbf0' only
2025-12-02 13:40:01,492 [io-executor-thread-1]  INFO    i.g.o.k.s.i.Slf4jLogger(info):86 - Propagating changes for connectionId: '223041f0-4364-4956-9769-58a9637dbbf0', new catalogId '128d9b39-8da9-4c55-973d-f93e5cc45712'
2025-12-02 13:40:01,492 [io-executor-thread-1]  INFO    i.a.c.s.h.h.ConnectionTimelineEventHelper(logSchemaChangeAutoPropagationEventInConnectionTimeline):287 - Diff is empty. Bypassing logging an event.
2025-12-02 13:40:01,492 [io-executor-thread-1]  INFO    i.a.c.s.h.ConnectionsHandler(applySchemaChange):2011 - Sending notification of schema auto propagation for connectionId: 'class ConnectionRead {
k
A message from kapa.ai
From these new logs we can see a few important things: 1. The UI can now hydrate job 25198 and is no longer “stuck” only on that step You see
Hydrating job 25198
and then further activity instead of nothing. That suggests the previous “blank after hydration + 500 in UI” was transient, not a permanent crash in the hydration code. The docs confirm that when you open the Status page, Airbyte loads the recent job stats to render the connection page, which is what you’re seeing. [[Connection troubleshooting](https://docs.airbyte.com/platform/1.8/cloud/managing-airbyte-cloud/review-connection-status#connection-troubleshooting)] 2. A new sync job was successfully created for this connection The key lines:
Copy code
text    Found the following streams to reset for connection 223041f0-4364-4956-9769-58a9637dbbf0: []    ... DefaultJobPersistence(enqueueJob):107 - enqueuing pending job for scope: 223041f0-4364-4956-9769-58a9637dbbf0    ... New job created, with id: 26961
This shows the scheduler was able to: - evaluate streams to reset (
[]
→ nothing to reset), - enqueue a sync job for this connection, - and create job 26961. So, from the platform’s perspective, the connection is no longer blocked from scheduling a sync. 3. **Schema‑change handling ran and auto‑propagation did not detect any actual diff**
Copy code
text    Applying schema change for connection '223041f0-4364-4956-9769-58a9637dbbf0' only    Propagating changes for connectionId ... new catalogId ...    Diff is empty. Bypassing logging an event.    Sending notification of schema auto propagation ...
This is the schema‑change management path. The docs describe that Airbyte tracks schema changes, can auto‑propagate them, and logs/acts based on the diff. [[Schema change management](https://docs.airbyte.com/platform/using-airbyte/schema-change-management)] Here, the diff is empty, so no effective schema changes are being applied. --- ### What this means for your original problems - The logs indicate that: - The connection page can now be opened without the server “stopping” right after
Hydrating job 25198
. - A new sync job (26961) has been scheduled for that connection. - The knowledge sources you provided do not include detailed behavior for exactly this sequence (hydration → 500 → later a new job being created), so I cannot say precisely why the earlier UI 500 happened, only that the system now appears to be accepting new syncs for that connection. Given what is documented: 1. Check that job 26961 actually runs to completion Use the Status page or the job history to see whether job 26961: - starts replication, - and whether it again fails around ~300 MB with
destination ERROR Killed
. The resource‑configuration docs explain how to inspect and adjust per‑job resources (memory/CPU) if it’s still being OOM‑killed. [[Jobs & resources](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-airbyte#jobs)] 2. If syncs now start but the UI still occasionally shows the 500 message The docs and issues you shared don’t document a known bug for “intermittent 500 on connection page while syncs do run”, so if that persists you’ll likely need to open a GitHub issue with: - the 500 message from the UI, - these server log snippets (showing hydration and job creation), - and your deployment details (K8s, Airbyte 1.8.1, this specific Zendesk→AWS Datalake connection). This follows the pattern used to diagnose other 500‑level backend errors. [[Error 500 fetching sources](https://github.com/airbytehq/airbyte/issues/53609)] Because the sources don’t contain a prescribed fix for this exact combination of symptoms, I can’t provide a more specific root cause. What we can see is that the platform is now successfully creating new jobs for the connection; the next step is to confirm whether those jobs still hit the original “destination killed around 300 MB” issue and, if so, continue focusing on increasing destination memory and/or reporting a potential destination‑aws‑datalake bug.
j
@kapa.ai
Copy code
2025-12-02 13:40:33,648 [io-executor-thread-8]  INFO    i.a.v.j.JsonSchemaValidator(test):119 - JSON schema validation failed.
errors: $.method: must be the constant value 'CDC', $: required property 'replication_slot' not found, $: required property 'publication' not found
2025-12-02 13:40:33,648 [io-executor-thread-8]  INFO    i.a.v.j.JsonSchemaValidator(test):119 - JSON schema validation failed.
errors: $.method: must be the constant value 'Xmin'
2025-12-02 13:40:33,648 [io-executor-thread-8]  INFO    i.a.v.j.JsonSchemaValidator(test):119 - JSON schema validation failed.
errors: $.credentials_title: does not have a value in the enumeration ["IAM Role"], $.credentials_title: must be the constant value 'IAM Role', $: required property 'role_arn' not found
2025-12-02 13:40:33,648 [io-executor-thread-8]  INFO    i.a.v.j.JsonSchemaValidator(test):119 - JSON schema validation failed.
errors: $.aws_access_key_id: object found, string expected, $.aws_secret_access_key: object found, string expected
2025-12-02 13:40:33,648 [io-executor-thread-8]  INFO    i.a.v.j.JsonSchemaValidator(test):119 - JSON schema validation failed.
errors: $.format_type: does not have a value in the enumeration ["JSONL"], $.compression_codec: does not have a value in the enumeration ["UNCOMPRESSED", "GZIP"]
2025-12-02 13:40:33,683 [io-executor-thread-9]  INFO    i.g.o.k.s.i.Slf4jLogger(info):90 - deleteStreamResetRecordsForJob was called for job 26961 with config type sync. Returning, as config type is not resetConnection.
k
A message from kapa.ai
Those log lines are Airbyte’s JSON schema validator telling you that the saved configuration of this connection no longer matches the connector specs for the involved connectors. From the messages: -
$.method: must be the constant value 'CDC'
and
required property 'replication_slot'/'publication' not found
→ A Postgres source/destination (or another connector with a
method
field) is now expecting CDC configuration fields that are missing in the stored config. -
$.method: must be the constant value 'Xmin'
→ Another branch of the same `oneOf`/`enum` schema expects
Xmin
, but your saved config has a different or missing value. -
$.credentials_title: does not have a value in the enumeration ["IAM Role"], must be the constant value 'IAM Role', required property 'role_arn' not found
→ The destination (AWS‑related) is now expecting IAM Role credentials with a
role_arn
field, but your config uses a different credentials shape. -
$.aws_access_key_id: object found, string expected
/
aws_secret_access_key: object found, string expected
→ Those fields are now defined as simple strings in the spec, but what’s stored in your connection config is an object (likely because of a previous version/schema where credentials were nested differently). -
$.format_type: does not have a value in the enumeration ["JSONL"]
and
$.compression_codec: does not have a value in the enumeration ["UNCOMPRESSED", "GZIP"]
→ The destination’s format/compression options have changed; the saved values are no longer valid per the current spec. In the Airbyte forum, similar “JSON schema validation failed” messages are explained as the saved config not matching the connector’s JSON schema, often after connector or platform updates. [[Understanding errors](https://discuss.airbyte.io/t/2314)] In other threads, the validator complains when fields no longer satisfy `enum`/`oneOf`/`required` constraints after a change in the JSON schema, exactly like what you’re seeing. [[JsonSchemaValidator oneOf](https://discuss.airbyte.io/t/914); [After update schema fails](https://discuss.airbyte.io/t/1456)] The practical fix, based on these sources, is: 1. Open the source and destination configuration in the UI and re‑save them - For each connector: - Go to its configuration page. - Make sure all required fields are filled according to the current spec (e.g., choose the correct auth/credentials method, supply
role_arn
if using IAM Role, pick a supported format/compression). - Save. This forces Airbyte to store a config that matches the connector’s current JSON schema. 2. Re‑configure this connection if necessary - If the connection was created under older versions of the connectors, you may need to: - Re‑select / re‑save the source; - Re‑select / re‑save the destination; - Then re‑save the connection so its stored config references the updated actor configs. The knowledge you provided does not contain an example with exactly this combination of fields (
credentials_title
,
format_type
, etc.), so I can’t point to a specific known bug in those connector versions. But it does clearly show that: - “JSON schema validation failed” here means the existing config is invalid against the current JSON schema, usually after an upgrade. [[Understanding errors](https://discuss.airbyte.io/t/2314); [Schema validator issues after update](https://discuss.airbyte.io/t/1456)] Once the configs are brought back in line with the specs and saved, these validation warnings should stop for this connection, and the sync/job logic can proceed.
j
@kapa.ai On the UI I have a job that started but not concluded or cancelled and after that every time I try to enable the connection and run a sync it doesnt show:
Copy code
Connection disabled

Default User disabled this connection
Dec 1, 2025, 7:07 PM

Connection enabled

Default User enabled this connection
Dec 1, 2025, 6:15 PM

Connection disabled

Default User disabled this connection
Nov 28, 2025, 8:19 PM

Connection enabled

Default User enabled this connection
Nov 28, 2025, 8:11 PM

Connection disabled

Default User disabled this connection
Nov 28, 2025, 7:48 PM

Connection enabled

Default User enabled this connection
Nov 28, 2025, 7:43 PM

Connection disabled

Default User disabled this connection
Nov 28, 2025, 7:40 PM

Connection enabled

Default User enabled this connection
Nov 28, 2025, 7:39 PM

Connection disabled

Default User disabled this connection
Nov 28, 2025, 7:26 PM

Connection enabled

Default User enabled this connection
Nov 28, 2025, 7:21 PM

Connection disabled

Default User disabled this connection
Nov 28, 2025, 7:10 PM

Sync started

Default User manually started a sync
k
A message from kapa.ai
It looks like this connection is “stuck” because of a previous job plus a now‑invalid configuration, and the UI isn’t starting new syncs even though you toggle enable/disable. From the materials you provided, there are two relevant patterns: 1. Stuck/hanging jobs preventing new syncs - Older versions of Airbyte had issues where a sync could remain in a perpetual “running” state and block further syncs until the job was cancelled or recovered. Airbyte 1.4 introduced a heartbeat service to auto‑recover hanging connections by creating a new attempt when a job is stuck in progress. [[Autorecovery for hanging connections](https://docs.airbyte.com/release_notes/v-1.4#-bug-fixes)] - In several Kubernetes reports, users see a job that never finishes; attempts to start a new sync either do nothing, or return “A sync is already running for …”. [[Sync already running](https://discuss.airbyte.io/t/3336); [All syncs hanging](https://discuss.airbyte.io/t/1916)] Your history shows only enable/disable events and a single “Sync started”, but no “Sync succeeded/failed/cancelled”, which matches a job that never reached a terminal state. 2. Invalid configuration blocking the connection - Your logs show repeated
JSON schema validation failed
errors for this connection’s config (wrong
method
, missing
replication_slot
, wrong credentials format, unsupported `format_type`/`compression_codec`, etc.). That means the saved source/destination settings for this connection no longer match the current connector specs. [[Understanding schema validation errors](https://discuss.airbyte.io/t/2314)] When a connection’s underlying actor configs are invalid against their JSON schema, the platform can reject operations on that connection. The docs don’t spell out “UI sync button does nothing” in this case, but this is consistent with the validator failures you’re seeing whenever this connection is touched. --- Given what’s documented, the most reliable way to “unstick” this connection is: 1. Fix the underlying source and destination configs For the Zendesk source and AWS Datalake destination used by this connection: - Open each connector in the UI. - Re‑enter credentials and options so that: - Required fields (e.g.,
role_arn
if using IAM Role; any `method`/CDC fields) are present. - Fields match the current enums (e.g.,
format_type
allowed values,
compression_codec
from
["UNCOMPRESSED","GZIP"]
). - Save each connector. This ensures the stored configs pass JSON schema validation. [[Understanding schema validation errors](https://discuss.airbyte.io/t/2314)] 2. Re‑save (or recreate) the connection Because the existing connection has a stuck job and was created with an older, now‑invalid config: - Try editing and re‑saving the connection after the actors are fixed. - If it still refuses to start syncs, create a new connection between the same (now‑valid) source and destination and copy the mappings/schedule. The docs and forum threads imply connections are independent; creating a new one is a common workaround when one connection’s internal state is bad. [[Connection troubleshooting](https://docs.airbyte.com/platform/1.8/cloud/managing-airbyte-cloud/review-connection-status#connection-troubleshooting)] 3. Cancel the stuck job if possible - In the connection’s Timeline/Status view, look for the last “Sync started” job and try cancelling it from the UI if the option is available. - In similar cases, manually cancelling the hanging job allowed new jobs to be scheduled. [[All syncs hanging](https://discuss.airbyte.io/t/1916); [Sync already running](https://discuss.airbyte.io/t/3336)] If, after: - correcting the source/destination configs (so schema validation passes), - re‑saving or recreating the connection, - and cancelling any visible running job, you still cannot get a new “Sync started” event to appear for this connection while others work, then the knowledge sources don’t provide enough information to identify a deeper platform bug in 1.8.1. In that case, the next step would be to open a GitHub issue including: - the connection id, - the job id that never completed (25198), - the JSON schema validation errors you shared, - and a description that enabling the connection and clicking “Sync now” produces no new job in the timeline.
j
@kapa.ai I created a new connection with the same configurations and state since the tables are incremental append, and now I can run the job, although I only got back to the previous destination killed error:
Copy code
2025-12-02 12:03:47 replication-orchestrator INFO Records read: 315000 (322 MB)
2025-12-02 12:04:05 destination ERROR Killed
2025-12-02 12:04:05 replication-orchestrator ERROR DestinationWriter error: 
2025-12-02 12:04:05 replication-orchestrator ERROR DestinationWriter error:
I want to try and change the resource requirements of the connection on the internal database for it to use 4Gi and 1 CPU on the highest level of precedence and to be sure the value is updated over the default installation. Right now on running a sync it is like this:
Copy code
2025-12-02 11:40:31 platform INFO [initContainer] image: airbyte/workload-init-container:1.8.1 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=2Gi, cpu=1}, additionalProperties={})
2025-12-02 11:40:31 platform INFO Launching replication pod: replication-job-26971-attempt-0 (selectors = {}) with containers:
2025-12-02 11:40:31 platform INFO [source] image: airbyte/source-zendesk-support:4.10.12 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=1Gi, cpu=0.2}, additionalProperties={})
2025-12-02 11:40:31 platform INFO [destination] image: airbyte/destination-aws-datalake:0.1.58 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=1Gi, cpu=0.2}, additionalProperties={})
2025-12-02 11:40:31 platform INFO [orchestrator] image: airbyte/container-orchestrator:1.8.1 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=2Gi, cpu=1}, additionalProperties={})
Write me a query for me to run on the internal database and update the required memory to 4GI and cpu to 1. This new connection id is b213af00-ea37-4bbe-805b-62f5216360d1
k
A message from kapa.ai
To set per-connection resource requirements at the highest precedence, Airbyte’s docs say to update the
connection.resource_requirements
column directly in the config database. [[1.8 conn-specific](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-connector-resources#configuring-connection-specific-requirements); [connection table](https://docs.airbyte.com/platform/1.8/understanding-airbyte/database-data-catalog#connection)] The documented pattern is:
Copy code
sql update connection set resource_requirements = '{"cpu_limit": "2", "cpu_request": "2", "memory_limit": "2048Mi", "memory_request": "2048Mi"}' where id = '<connection-id>';
For your connection
b213af00-ea37-4bbe-805b-62f5216360d1
, with 4Gi memory and 1 CPU, that becomes:
Copy code
sql UPDATE connection SET resource_requirements = '{   "cpu_limit": "1",   "cpu_request": "1",   "memory_limit": "4Gi",   "memory_request": "4Gi" }' WHERE id = 'b213af00-ea37-4bbe-805b-62f5216360d1';
This connection-specific setting has the highest precedence and will override instance-wide, connector-definition, and connector-specific defaults for sync jobs on this connection. [[Configuring connector resources](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-connector-resources#configuring-connection-specific-requirements)] After running it, trigger a new sync and confirm in the job logs that the
[source]
and
[destination]
resources:
lines now show the updated limits/requests. [[Debugging resources](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-connector-resources#debugging)]
j
@kapa.ai In the context of a sync pod, what difference would more or less cpu make?
k
A message from kapa.ai
In Airbyte’s model, CPU settings mainly affect how fast a sync can process data and how many syncs you can run in parallel, not whether a single sync succeeds. From the docs: - Each sync job runs in its own pod with separate containers for source, destination, and orchestrator, and the main scaling knobs are CPU and memory per connector pod. [[Scaling after install](https://docs.airbyte.com/platform/1.8/enterprise-setup/scaling-airbyte#concurrent-syncs)] - You can set CPU requests/limits at several levels (instance‑wide, definition‑level, connector‑specific, connection‑specific). The higher‑precedence ones override lower‑level defaults. [[Connector resources](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-connector-resources)] - The docs note that increasing job CPU and memory may allow for increased sync speeds when a deployment is under‑provisioned. [[Scaling after install](https://docs.airbyte.com/platform/1.8/enterprise-setup/scaling-airbyte#connector-cpu--memory-settings)] So, in practice for a single sync pod: - More CPU (higher limit/request) - Gives the containers more processing capacity if they are CPU‑bound (e.g., heavy JSON processing, compression, encryption). - Can speed up syncs that are currently limited by CPU. [[Scaling after install](https://docs.airbyte.com/platform/1.8/enterprise-setup/scaling-airbyte#connector-cpu--memory-settings)] - Increases the per‑pod footprint, which can reduce how many pods your cluster can schedule concurrently. - Less CPU - Reduces per‑pod resource footprint, which may let you run more syncs in parallel on the same cluster. [[Scaling after install](https://docs.airbyte.com/platform/1.8/enterprise-setup/scaling-airbyte#concurrent-syncs)] - If set too low, a CPU‑intensive connector may run more slowly, but the docs do not mention CPU throttling causing hard failures; failures in similar cases are almost always due to memory, not CPU. For your specific “`destination ERROR Killed` around 300 MB” issue: the Airbyte docs clearly call out memory (especially on the source, but also destination) as the main culprit for sync failures, because connectors can hold up to 10,000 records in memory. [[What to scale](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#memory); [Next what to scale](https://docs.airbyte.com/platform/next/operator-guides/scaling-airbyte#memory)] CPU changes will mainly influence speed; resolving “Killed” errors is typically about increasing memory limits for that connector.