https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> I have a Self-Managed Community Air...
# ask-ai
j
@kapa.ai I have a Self-Managed Community Airbyte 1.8.1 K8S installation, and I want to create a File source of a 2GB json file on a AWS bucket. When I try to create the source connector the "Test the source" part runs endlessly, never giving me any error. What could be causing this?
k
The behavior you describe (source “Test the source” hanging indefinitely with no error) matches known issues where: 1. The worker/connector pod is created, but hangs when writing output or talking to storage (S3 or doc store). In a similar OSS 1.8.5 case, the connector pod started correctly, but the request hung and only logged “Writing output … to the doc store”, with the client eventually timing out. [[Custom source API](https://github.com/airbytehq/airbyte/issues/68608)] 2. S3 credentials or permissions for Airbyte’s internal storage are misconfigured, so the connector-sidecar/worker cannot read or write to the S3 bucket where job outputs are stored. In another Kubernetes/EKS deployment, “Test the source” failed because the connector pod lacked valid AWS credentials (401 / “Unable to load AWS credentials from any provider in the chain”), which caused the job to error at the storage/Secrets Manager layer rather than on the connector itself. [[EKS HTTP 500](https://github.com/airbytehq/airbyte/issues/48809)] Given those references and the symptoms: - The issue is likely not the 2 GB JSON file itself, but Airbyte’s ability to write/read job output or secrets using your configured S3 / AWS credentials. - The fact that it “runs endlessly” with no UI error strongly suggests the worker or sidecar is blocked or failing when persisting the “check” result, rather than failing inside the File source code. What to check Based on the issues above, the most probable causes are: 1. S3 / storage configuration for the Airbyte deployment - Verify that the bucket configured for Airbyte’s own storage (not just your data file) exists and is reachable. - Confirm the IAM role / access keys used by the worker and connector-sidecar pods have: - 
s3:PutObject
, 
s3:GetObject
, 
s3:ListBucket
on the storage bucket/path Airbyte uses for job logs/outputs. [[Mixpanel S3 logs](https://discuss.airbyte.io/t/2529)] - Check for AWS SDK credential errors (
Unable to load AWS credentials
, 
Unauthorized (Service: null; Status Code: 401)
) in the logs of: - 
airbyte-worker
(or 
airbyte-abctl-worker
in abctl installs) - The job pod for the File source check (look for 
source-file-check-...
or similar). [[EKS HTTP 500](https://github.com/airbytehq/airbyte/issues/48809)] 2. Credentials propagation into job pods - In the EKS 1.2.0 issue, the sidecar container had proper 
AWS_ACCESS_KEY_ID
/ 
AWS_SECRET_ACCESS_KEY
, but the init/main containers did not, leading to failures when they tried to reach AWS services. [[EKS HTTP 500](https://github.com/airbytehq/airbyte/issues/48809)] - For your 1.8.1 K8S install, confirm that whatever AWS auth mechanism you use (env vars, IRSA, etc.) is actually available to: - The 
airbyte-worker
pod - The File source job pod (all containers inside it, including any sidecar/init) 3. Worker logs during “Test the source” - When you click “Test the source”, watch the 
airbyte-worker
logs in real time. In the similar OSS 1.8.5 case, the logs showed the connector watcher hanging exactly on “Writing output to the doc store”. [[Custom source API](https://github.com/airbytehq/airbyte/issues/68608)] - If you see that log line (or any S3/Secrets Manager error) and nothing after it, that confirms the hang is at the storage layer, not at reading the file. Because the knowledge sources do not include a specific bug tied to File source on 1.8.1 or to 2GB file size, I can’t point to a version-specific regression. The available information instead points to AWS/S3 credential or storage configuration issues causing the check job to hang or fail when persisting its result. If after confirming IAM/S3 and pod credentials you still see the hang and no clear log error, the next step would be to share: - The 
airbyte-worker
logs around the time of the check. - The full logs of the 
source-file-check-…
pod. Those details would be needed to narrow it down further; the current knowledge sources don’t provide more specific guidance.
j
@kapa.ai I made another test creating a source FIle connector for a file on the same s3 bucket as the one I'm having trouble during the check. This other test was normal, being able to test the connection and advancing the source connector to the next step, defining the destination. This other file was a csv with 1.1mb. Also, for the first file I got an error while trying to setup it again:
Copy code
Airbyte is temporarily unavailable. Please try again. (HTTP 502)
On console:
Copy code
core-eqb7qj4qcu.js:56 
 POST <http://10.100.25.52:8081/api/v1/scheduler/sources/check_connection> 502 (Bad Gateway)
k
A message from kapa.ai
The symptoms you describe match several known 502 / “Test the connection” issues, and they point to a platform / worker problem, not to the specific File connector config: - Other users see 
Airbyte is temporarily unavailable. Please try again. (HTTP 502)
when testing any source (CSV, S3, DB, ClickHouse) on recent versions, even with very small files, and the UI hangs at “Please wait a little bit more…”. [[Problem 1.8.3](https://github.com/airbytehq/airbyte/issues/66692); [source-file 502](https://github.com/airbytehq/airbyte/issues/65574)] - In Kubernetes/abctl installs, the error often comes from the worker ↔️ workload API path, not from the connector itself. When that breaks or is misconfigured, checks hang or return 502 even though job pods may be created and succeed. [[source-file 502](https://github.com/airbytehq/airbyte/issues/65574)] In those reports, two fixes consistently helped: 1. Restart the worker and workload-api deployments On 1.8.x abctl/K8s installs, users resolved this exact 502 on “Test connection” by restarting these two services: [[Problem 1.8.3](https://github.com/airbytehq/airbyte/issues/66692)]
Copy code
bash # adjust namespace if different kubectl -n airbyte-abctl rollout restart deployment airbyte-abctl-worker kubectl -n airbyte-abctl rollout restart deployment airbyte-abctl-workload-api-server  kubectl -n airbyte-abctl rollout status deployment airbyte-abctl-worker kubectl -n airbyte-abctl rollout status deployment airbyte-abctl-workload-api-server
2. Ensure the worker and workload-api share a valid WORKLOAD_API_BEARER_TOKEN In another 502 case, the worker could run connector pods successfully only after the bearer token was explicitly set on both deployments and they were restarted. [[source-file 502](https://github.com/airbytehq/airbyte/issues/65574)]
Copy code
bash kubectl set env deploy/airbyte-abctl-workload-api-server \   -n "$NAMESPACE" \   WORKLOAD_API_BEARER_TOKEN="$WORKLOAD_TOKEN"  kubectl set env deploy/airbyte-abctl-worker \   -n "$NAMESPACE" \   WORKLOAD_API_BEARER_TOKEN="$WORKLOAD_TOKEN"  kubectl rollout restart deploy/airbyte-abctl-workload-api-server -n "$NAMESPACE" kubectl rollout restart deploy/airbyte-abctl-worker -n "$NAMESPACE"
After this patch, users reported that multiple connectors (including File) started working again. [[source-file 502](https://github.com/airbytehq/airbyte/issues/65574)] --- Because your small CSV in the same bucket works, the S3 access path from the connector clearly functions. The 2 GB JSON file might be making the check slower and pushing some component into an error or timeout, but the knowledge sources do not provide a specific file‑size limit or bug for 2 GB JSON on 1.8.1. Given the available information, the most likely causes are: - A transient or configuration issue in airbyte-abctl-worker / workload-api-server leading to 502s on 
POST /api/v1/scheduler/sources/check_connection
. [[Problem 1.8.3](https://github.com/airbytehq/airbyte/issues/66692); [source-file 502](https://github.com/airbytehq/airbyte/issues/65574)] - Possibly a timeout or failure in the worker when handling a heavier check, but the sources do not document a confirmed bug for large JSON files. If, after applying the restart / bearer-token steps above, the 2 GB JSON check still hangs or returns 502 while smaller files work, the next step would be to inspect and share: - 
airbyte-abctl-worker
and 
airbyte-abctl-workload-api-server
logs during the failing check, and - logs from the specific 
source-file-check-...
pod (if it gets created). The current knowledge sources don’t include deeper guidance specific to large File-source checks on 1.8.1, so beyond these known fixes I can’t confidently suggest more targeted changes.
j
@kapa.ai The initial connection setup logs would appear on kubectl logs server-pod?
k
A message from kapa.ai
Yes. The “Test connection” / initial setup request is handled by the server pod, so its side of the flow will show up in that pod’s logs. For an abctl-based K8s deploy, you’d typically look at:
Copy code
bash kubectl logs -n airbyte-abctl <airbyte-abctl-server-pod-name>
The docs explicitly show using 
kubectl logs
on the 
airbyte-abctl-server-...
pod to inspect server behavior in a local/development abctl cluster. [[Developing locally 1.8](https://docs.airbyte.com/platform/1.8/contributing-to-airbyte/developing-locally#prerequisites)] Keep in mind: - These logs will show the HTTP request (e.g. 
/api/v1/scheduler/sources/check_connection
) and how the server handed it off (or any 5xx it returned). - The actual connector run (check job) logs will be on the worker and the job pod (e.g. 
source-file-check-...
), and you’d inspect those separately with 
kubectl logs
or by exec’ing into the pod. [[Browsing on Kubernetes](https://docs.airbyte.com/platform/1.8/operator-guides/browsing-output-logs#browsing-on-kubernetes)]
j
@kapa.ai Just got the 502 error on the connection setup but found nothing on the server logs:
Copy code
2025-12-10 21:20:34,160 [io-executor-thread-2]  INFO    i.a.v.j.JsonSchemaValidator(test):119 - JSON schema validation failed.
errors: $.method: must be the constant value 'CDC', $: required property 'replication_slot' not found, $: required property 'publication' not found
2025-12-10 21:20:34,160 [io-executor-thread-2]  INFO    i.a.v.j.JsonSchemaValidator(test):119 - JSON schema validation failed.
errors: $.method: must be the constant value 'Xmin'
2025-12-10 21:20:34,160 [io-executor-thread-2]  INFO    i.a.v.j.JsonSchemaValidator(test):119 - JSON schema validation failed.
errors: $.credentials_title: does not have a value in the enumeration ["IAM Role"], $.credentials_title: must be the constant value 'IAM Role', $: required property 'role_arn' not found
2025-12-10 21:20:34,160 [io-executor-thread-2]  INFO    i.a.v.j.JsonSchemaValidator(test):119 - JSON schema validation failed.
errors: $.aws_access_key_id: object found, string expected, $.aws_secret_access_key: object found, string expected
2025-12-10 21:20:34,160 [io-executor-thread-2]  INFO    i.a.v.j.JsonSchemaValidator(test):119 - JSON schema validation failed.
errors: $.format_type: does not have a value in the enumeration ["JSONL"], $.compression_codec: does not have a value in the enumeration ["UNCOMPRESSED", "GZIP"]
2025-12-10 21:20:34,201 [io-executor-thread-4]  INFO    i.g.o.k.s.i.Slf4jLogger(info):90 - deleteStreamResetRecordsForJob was called for job 31371 with config type sync. Returning, as config type is not resetConnection.
The command I run is:
Copy code
kubectl logs -n airbyte airbyte-server-7dbb45cf5d-6szts --tail=500 -f
k
A message from kapa.ai
Those 
JsonSchemaValidator
lines are just generic config-validation messages and are not specific to the 502 you’re seeing. They indicate that, at some point, configs were submitted that didn’t match the expected JSON schema (e.g. wrong replication method, wrong S3 auth fields, wrong file format), but they are not fatal on their own and don’t explain the gateway error. In the 1.8.x / Kubernetes reports that match your symptoms (
Airbyte is temporarily unavailable. Please try again. (HTTP 502)
on 
POST /api/v1/scheduler/sources/check_connection
), the root cause was not visible in the server logs at all. Instead, it was in the worker and workload API components: - In multiple abctl/K8s installs, “Test the connection” for any connector (CSV, S3, DB, ClickHouse) hung and then returned 502, while server logs didn’t show a clear error. The fix was to reconfigure/restart the worker and workload-api-server so they shared a valid 
WORKLOAD_API_BEARER_TOKEN
. [[Problem 1.8.3](https://github.com/airbytehq/airbyte/issues/66692); [source-file 502](https://github.com/airbytehq/airbyte/issues/65574); [workload 401](https://github.com/airbytehq/airbyte/issues/65567)] In other words: not seeing anything “bad” in 
airbyte-server
logs during the 502 is expected in this class of issues. The actual failure is usually: - Between server → worker → workload API, or - Inside the worker / job pod when it tries to create or manage the check job. Given what’s in the knowledge sources, the next useful steps are all outside the server pod: 1. Check worker and workload-api logs during the failing check Run these in parallel while you reproduce the 502 (adjust namespace and deployment names to your install):
Copy code
bash    kubectl logs -n airbyte-abctl deploy/airbyte-abctl-worker -f    kubectl logs -n airbyte-abctl deploy/airbyte-abctl-workload-api-server -f
In similar cases, users saw 401/unauthorized or other workload-API–related errors there, even when server logs looked normal. [[workload 401](https://github.com/airbytehq/airbyte/issues/65567)] 2. Make sure 
WORKLOAD_API_BEARER_TOKEN
is aligned on worker and workload-api Several 502 cases were fixed by explicitly setting the same bearer token on both deployments and restarting: [[Problem 1.8.3](https://github.com/airbytehq/airbyte/issues/66692); [source-file 502](https://github.com/airbytehq/airbyte/issues/65574)]
Copy code
bash    # Example for abctl namespace; adapt if yours is different    NAMESPACE=airbyte-abctl    BEARER_TOKEN=$(openssl rand -hex 16)     kubectl -n "$NAMESPACE" set env deployment/airbyte-abctl-worker \      WORKLOAD_API_BEARER_TOKEN="$BEARER_TOKEN" \      INTERNAL_API_AUTH_TOKEN="$BEARER_TOKEN"     kubectl -n "$NAMESPACE" set env deployment/airbyte-abctl-workload-api-server \      WORKLOAD_API_BEARER_TOKEN="$BEARER_TOKEN"     kubectl -n "$NAMESPACE" rollout restart deployment airbyte-abctl-worker    kubectl -n "$NAMESPACE" rollout restart deployment airbyte-abctl-workload-api-server
3. Confirm that a check job pod is actually being created During “Test the source”, you should see a pod like 
source-file-check-...
appear. If no pod shows up, the problem is between worker and workload API; if it does show up, then its own logs may contain S3/JSON/timeout errors.
Copy code
bash    kubectl get pods -n airbyte-abctl | grep source-file
The knowledge sources don’t show a documented case where a 2 GB JSON file alone causes this 502 pattern; all similar 1.8.x reports tie back to the worker/workload-API auth and job-launch path rather than the file content or size. [[Problem 1.8.3](https://github.com/airbytehq/airbyte/issues/66692); [source-file 502](https://github.com/airbytehq/airbyte/issues/65574)] So, the absence of a clear server-side error for your 502 is consistent with those issues; you’ll need to inspect and, if necessary, reconfigure the worker and workload-api components as above.
j
@kapa.ai I found a job with error on the pods list:
Copy code
[root@CDA-AIRBYTE-TESP3 ~]# kubectl get pods -n airbyte
NAME                                                             READY   STATUS      RESTARTS   AGE
airbyte-airbyte-bootloader                                       0/1     Completed   0          47d
airbyte-connector-builder-server-7567cbdf47-szsvd                1/1     Running     0          11d
airbyte-cron-66d465bd8b-8g8s2                                    1/1     Running     0          11d
airbyte-db-0                                                     1/1     Running     0          11d
airbyte-minio-0                                                  1/1     Running     0          11d
airbyte-server-7dbb45cf5d-6szts                                  1/1     Running     0          11d
airbyte-temporal-574df4f985-8fnxr                                1/1     Running     0          11d
airbyte-worker-58fbc95cf4-5krbn                                  1/1     Running     0          11d
airbyte-workload-api-server-84d48b5465-xzp6v                     1/1     Running     0          11d
airbyte-workload-launcher-6f7bbf4cd4-w9w72                       1/1     Running     0          11d
replication-job-30973-attempt-0                                  3/3     Running     0          17h
replication-job-31372-attempt-0                                  0/3     Completed   0          5m31s
replication-job-31373-attempt-0                                  3/3     Running     0          5m31s
replication-job-31374-attempt-0                                  0/3     Completed   0          5m31s
replication-job-31375-attempt-0                                  0/3     Completed   0          5m30s
source-file-check-44902c53-16ac-447e-9f28-73d38cd4ee56-0-wrepl   0/2     Error       0          18m
k
A message from kapa.ai
The failing pod is the key here:
Copy code
text source-file-check-44902c53-16ac-447e-9f28-73d38cd4ee56-0-wrepl   0/2  Error
This is exactly the pod that runs the “Test the source” for your File connector, so its failure explains the endless spinner and eventual 
HTTP 502
on:
Copy code
text POST /api/v1/scheduler/sources/check_connection
In very similar 1.8.x Kubernetes installs, users saw: - UI: 
Airbyte is temporarily unavailable. Please try again. (HTTP 502)
- A 
source-...-check-...
pod going to 
Error
- Little or nothing in 
airbyte-server
logs during the 502 The root causes in those cases were: 1. Worker ↔️ Workload API auth / config issues (not connector config): - Workload API calls rejected with 
401 Unauthorized
because 
WORKLOAD_API_BEARER_TOKEN
wasn’t aligned between the worker and the workload-api-server. [[workload 401](https://github.com/airbytehq/airbyte/issues/65567)] - Fix: set the same header/token on both deployments and restart them. [[source-file 502](https://github.com/airbytehq/airbyte/issues/65574); [Problem 1.8.3](https://github.com/airbytehq/airbyte/issues/66692)] 2. The check pod itself failing (e.g., auth / storage issues) and the platform surfacing only a generic 502, as you’re seeing now. [[source-file 502](https://github.com/airbytehq/airbyte/issues/65574)] Given that, the next concrete steps are: --- ### 1. Inspect the failing check pod’s logs This will tell you why the File check is failing:
Copy code
bash kubectl logs -n airbyte source-file-check-44902c53-16ac-447e-9f28-73d38cd4ee56-0-wrepl --all-containers
Look for: - S3-related errors (auth, permissions, timeouts) - AWS SDK credential errors (
Unable to load AWS credentials
, 401s, etc.) - Any stack trace indicating why the pod exited with 
Error
In similar “can’t create File connector” reports, the pod logs were where the actual S3/auth problem was visible, while the UI only showed 
502
. [[source-file 502](https://github.com/airbytehq/airbyte/issues/65574)] --- ### 2. Check worker and workload-api-server logs Since 1.8.x issues often stem from the worker ↔️ workload API path, tail these while re-running “Test the source”:
Copy code
bash kubectl logs -n airbyte deploy/airbyte-worker -f kubectl logs -n airbyte deploy/airbyte-workload-api-server -f
If you see 
Unauthorized
or similar around workload API calls, that matches the known issue where mismatched bearer tokens break checks and cause 502s. [[workload 401](https://github.com/airbytehq/airbyte/issues/65567)] --- ### 3. (If you see auth/401 issues) Align WORKLOAD_API_BEARER_TOKEN and restart A documented fix for repeated 502s on source setup is to ensure the worker and workload-api-server share the same bearer token, then restart both: [[Problem 1.8.3](https://github.com/airbytehq/airbyte/issues/66692); [source-file 502](https://github.com/airbytehq/airbyte/issues/65574)]
Copy code
bash NAMESPACE=airbyte TOKEN=$(openssl rand -hex 24)  kubectl -n "$NAMESPACE" set env deployment/airbyte-workload-api-server \   WORKLOAD_API_AUTH_HEADER=Authorization \   WORKLOAD_API_BEARER_TOKEN="$TOKEN"  kubectl -n "$NAMESPACE" set env deployment/airbyte-worker \   WORKLOAD_API_AUTH_HEADER=Authorization \   WORKLOAD_API_BEARER_TOKEN="$TOKEN"  kubectl -n "$NAMESPACE" rollout restart deployment airbyte-worker kubectl -n "$NAMESPACE" rollout restart deployment airbyte-workload-api-server
Users with the same “502 on any connector / File connector” symptoms report that once these are aligned and restarted, File sources and other connectors start working again. [[Problem 1.8.3](https://github.com/airbytehq/airbyte/issues/66692); [source-file 502](https://github.com/airbytehq/airbyte/issues/65574)] --- Because your smaller CSV source on the same bucket works, the problem is unlikely to be simple S3 reachability; it’s either: - A failure specific to how the large JSON check job runs (visible in that 
source-file-check-...
pod’s logs), or - A platform/job-launch issue (worker ↔️ workload API) that’s surfaced as a 502. The knowledge sources don’t document a confirmed 1.8.1 bug tied specifically to 2 GB JSON files, so to go further you’d need to inspect and share the logs from: - 
source-file-check-44902c53-16ac-447e-9f28-73d38cd4ee56-0-wrepl
- 
airbyte-worker
- 
airbyte-workload-api-server
j
@kapa.ai
Copy code
[root@CDA-AIRBYTE-TESP3 ~]# kubectl logs -n airbyte source-file-check-44902c53-16ac-447e-9f28-73d38cd4ee56-0-wrepl --all-containers
Unsetting empty environment variable 'DATA_PLANE_SERVICE_ACCOUNT_CREDENTIALS_PATH'
Unsetting empty environment variable 'AIRBYTE_URL'
Unsetting empty environment variable 'KEYCLOAK_INTERNAL_REALM_ISSUER'
Unsetting empty environment variable 'DATA_PLANE_SERVICE_ACCOUNT_EMAIL'
Unsetting empty environment variable 'KEYCLOAK_CLIENT_ID'
Unsetting empty environment variable 'AIRBYTE_ROLE'
Unsetting empty environment variable 'CONTROL_PLANE_AUTH_ENDPOINT'
2025-12-10 21:17:01,895 [main]  INFO    i.a.c.ApplicationKt(main):14 - Sidecar start

    ___    _      __          __
   /   |  (_)____/ /_  __  __/ /____
  / /| | / / ___/ __ \/ / / / __/ _ \
 / ___ |/ / /  / /_/ / /_/ / /_/  __/
/_/  |_/_/_/  /_.___/\__, /\__/\___/
                    /____/
 : airbyte-connector-sidecar :

2025-12-10 21:17:02,550 [main]  INFO    i.m.c.e.DefaultEnvironment(<init>):170 - Established active environments: [worker-v2, control-plane, oss, local-secrets]
2025-12-10 21:17:03,038 [main]  INFO    i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):113 - Setting log level 'INFO' for logger: 'io.netty'
2025-12-10 21:17:03,039 [main]  INFO    i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):113 - Setting log level 'ERROR' for logger: 'com.zaxxer.hikari'
2025-12-10 21:17:03,039 [main]  INFO    i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):113 - Setting log level 'INFO' for logger: 'io.grpc'
2025-12-10 21:17:03,039 [main]  INFO    i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):113 - Setting log level 'INFO' for logger: 'io.temporal'
2025-12-10 21:17:03,040 [main]  INFO    i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):113 - Setting log level 'ERROR' for logger: 'com.zaxxer.hikari.pool'
2025-12-10 21:17:03,040 [main]  INFO    i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):113 - Setting log level 'INFO' for logger: 'io.fabric8.kubernetes.client'
2025-12-10 21:17:03,062 [main]  INFO    i.m.r.Micronaut(start):183 - No embedded container found. Running as CLI application
2025-12-10 21:17:04,982 [pool-3-thread-1]       INFO    i.a.c.i.LineGobbler$Companion(gobble$lambda$2):108 -
2025-12-10 21:17:05,006 [pool-3-thread-1]       INFO    i.a.c.i.LineGobbler$Companion(gobble$lambda$2):108 - ----- START CHECK -----
2025-12-10 21:17:05,006 [pool-3-thread-1]       INFO    i.a.c.i.LineGobbler$Companion(gobble$lambda$2):108 -
2025-12-10 21:26:05,044 [main]  WARN    i.a.c.ConnectorWatcher(waitForConnectorOutput):108 - Failed to find output files from connector within timeout of 9 minute(s). Is the connector still running?
2025-12-10 21:26:05,046 [main]  INFO    i.a.c.ConnectorWatcher(failWorkload):324 - Failing workload 778daa7c-feaf-4db6-96f3-70fd645acc77_44902c53-16ac-447e-9f28-73d38cd4ee56_0_check.
2025-12-10 21:26:05,077 [main]  INFO    i.a.c.ConnectorWatcher(exitFileNotFound):242 - Deliberately exiting process with code 2.
Using AIRBYTE_ENTRYPOINT: /entrypoint.sh
Terminated
Unsetting empty environment variable 'AWS_SECRET_MANAGER_REGION'
Unsetting empty environment variable 'DATA_PLANE_SERVICE_ACCOUNT_CREDENTIALS_PATH'
Unsetting empty environment variable 'FEATURE_FLAG_CLIENT'
Unsetting empty environment variable 'DD_AGENT_HOST'
Unsetting empty environment variable 'AIRBYTE_URL'
Unsetting empty environment variable 'KEYCLOAK_INTERNAL_REALM_ISSUER'
Unsetting empty environment variable 'DATA_PLANE_SERVICE_ACCOUNT_EMAIL'
Unsetting empty environment variable 'VAULT_PREFIX'
Unsetting empty environment variable 'AB_AZURE_KEY_VAULT_TENANT_ID'
Unsetting empty environment variable 'SECRET_STORE_GCP_PROJECT_ID'
Unsetting empty environment variable 'AB_AZURE_KEY_VAULT_VAULT_URL'
Unsetting empty environment variable 'AWS_SECRET_MANAGER_SECRET_TAGS'
Unsetting empty environment variable 'FEATURE_FLAG_BASEURL'
Unsetting empty environment variable 'VAULT_ADDRESS'
Unsetting empty environment variable 'AB_AZURE_KEY_VAULT_TAGS'
Unsetting empty environment variable 'KEYCLOAK_CLIENT_ID'
Unsetting empty environment variable 'AIRBYTE_ROLE'
Unsetting empty environment variable 'LAUNCHDARKLY_KEY'
Unsetting empty environment variable 'CONTROL_PLANE_AUTH_ENDPOINT'
Unsetting empty environment variable 'AWS_KMS_KEY_ARN'
Unsetting empty environment variable 'DD_DOGSTATSD_PORT'
2025-12-10 21:16:57,062 [main]  INFO    i.a.i.ApplicationKt(main):14 - Init start
2025-12-10 21:16:57,576 [main]  INFO    i.m.c.e.DefaultEnvironment(<init>):170 - Established active environments: [worker-v2, control-plane, oss, local-secrets]
2025-12-10 21:16:58,631 [main]  INFO    c.z.h.HikariDataSource(<init>):79 - config-pool - Starting...
2025-12-10 21:16:58,651 [main]  INFO    c.z.h.HikariDataSource(<init>):81 - config-pool - Start completed.
2025-12-10 21:16:59,190 [main]  INFO    i.a.i.InputFetcher(fetch):31 - Fetching workload...
2025-12-10 21:17:00,071 [main]  INFO    i.a.i.InputFetcher(fetch):41 - Workload 778daa7c-feaf-4db6-96f3-70fd645acc77_44902c53-16ac-447e-9f28-73d38cd4ee56_0_check fetched.
2025-12-10 21:17:00,072 [main]  INFO    i.a.i.InputFetcher(fetch):42 - Processing workload...
2025-12-10 21:17:00,342 [main]  WARN    o.j.t.JooqLogger(lambda$warn$7):461 - Version mismatch         : Database version is older than what dialect POSTGRES supports: 13.22. Consider <https://www.jooq.org/download/support-matrix> to see what jOOQ version and edition supports which RDBMS versions.
2025-12-10 21:17:00,613 [main]  INFO    i.a.i.InputFetcher(fetch):51 - Workload processed.
2025-12-10 21:17:00,626 [main]  INFO    c.z.h.HikariDataSource(close):349 - config-pool - Shutdown initiated...
2025-12-10 21:17:00,629 [main]  INFO    c.z.h.HikariDataSource(close):351 - config-pool - Shutdown completed.
2025-12-10 21:17:00,631 [main]  INFO    i.a.i.ApplicationKt(main):43 - Init end
k
A message from kapa.ai
The key line in your pod logs is:
Copy code
text 2025-12-10 21:26:05,044 [main]  WARN i.a.c.ConnectorWatcher(waitForConnectorOutput):108 - Failed to find output files from connector within timeout of 9 minute(s). Is the connector still running?
This means: - The job pod started correctly and the sidecar began the 
CHECK
(
----- START CHECK -----
). - For 9 minutes, the sidecar waited for the File source container to produce its output file (the jobOutput JSON with the check result). - No output file ever appeared, so the sidecar failed the workload and exited with code 2. This is the same failure pattern as the closed Zendesk issue where checks on Kubernetes ended with: > Failed to find output files from connector within timeout of 9 minute(s). Is the connector still running? [[Zendesk support issue](https://github.com/airbytehq/airbyte/issues/54684)] In that case, the platform also reported: > Workload failed, source: unknown > Internal message: Failed to find output files from connector within timeout of 9 minute(s). [[Zendesk support issue](https://github.com/airbytehq/airbyte/issues/54684)] So, for your File source: - The platform and workload path are functioning (pod is created, workload fetched and processed). - The File source container itself never emits a result for the check when pointed at your 2 GB JSON file, so the sidecar times out and fails the job. - The UI then surfaces this failure generically as 
Airbyte is temporarily unavailable. Please try again. (HTTP 502)
. Because you confirmed a small CSV in the same bucket works, this points to something specific about this large JSON check run (e.g., connector hanging or taking too long) rather than S3 connectivity or basic platform wiring. From the available knowledge sources: - There is no documented 1.8.x bug specifically about 2 GB JSON in the File source. - The only clearly matching pattern is the “no output file within timeout” behavior on Kubernetes, where the connector process effectively hangs during check. [[Zendesk support issue](https://github.com/airbytehq/airbyte/issues/54684)] Given that, what you can do next (based only on what’s documented): 1. Confirm connector container logs (if available) Your 
kubectl logs --all-containers
output only shows the init and sidecar. If there is a separate container for the File connector, check its logs explicitly to see if it reports an error or is stuck. The knowledge sources don’t show that command example for this exact pod, but generally connector failures appear there; in the Zendesk example, the failure was detected only by the sidecar timeout, with no detailed connector logs in the snippet. [[Zendesk support issue](https://github.com/airbytehq/airbyte/issues/54684)] 2. Retry with a reduced dataset Since smaller files work and this one hangs the check, try: - A smaller JSON subset (e.g., split the 2 GB file) and point the File source at that. - If that succeeds, you at least have a workaround while you investigate why the full file check never finishes. The docs/issues available don’t specify size limits, so this is a pragmatic test rather than a documented constraint. 3. Collect job details from the UI / API In the Zendesk case, the job details JSON explicitly showed 
succeeded: false
and the same timeout message in 
failureReason
. [[Zendesk support issue](https://github.com/airbytehq/airbyte/issues/54684)] Grabbing that for your failed check will confirm it’s the same class of error and can help if you open an upstream issue. The knowledge sources do not provide a configuration change or known fix that guarantees this specific File source timeout will be resolved (no mention of a patch, flag, or higher timeout setting for this scenario). They only document that, when the connector doesn’t write its output file within 9 minutes, the sidecar fails the workload with exactly the message you see. [[Zendesk support issue](https://github.com/airbytehq/airbyte/issues/54684)] So, based on what is documented: - The immediate cause is: the File connector’s 
check
never produces its output within the 9‑minute window, so the sidecar times out and fails the job. - The platform part (worker, workload API) appears to be working, since the workload is fetched and processed and only the connector output is missing. - Without more connector‑container logs, the knowledge sources don’t say why the connector hangs on that JSON file, only how the platform reacts.
j
@kapa.ai worker:
Copy code
2025-12-10 22:00:36,044 [Activity Executor taskQueue="SYNC", namespace="default": 13675]        INFO    i.a.w.t.s.InvokeOperationsActivityImpl(invokeOperationsV2):65 -
----- START POST REPLICATION OPERATIONS -----


2025-12-10 22:00:36,048 [Activity Executor taskQueue="SYNC", namespace="default": 13675]        INFO    i.a.w.t.s.InvokeOperationsActivityImpl(invokeOperationsV2):96 - No post-replication operation(s) to perform.
2025-12-10 22:00:36,048 [Activity Executor taskQueue="SYNC", namespace="default": 13675]        INFO    i.a.w.t.s.InvokeOperationsActivityImpl(invokeOperationsV2):99 -
----- END POST REPLICATION OPERATIONS -----


2025-12-10 22:01:09,690 [Activity Executor taskQueue="SYNC", namespace="default": 13676]        INFO    i.a.w.t.s.InvokeOperationsActivityImpl(invokeOperationsV2):65 -
----- START POST REPLICATION OPERATIONS -----


2025-12-10 22:01:09,695 [Activity Executor taskQueue="SYNC", namespace="default": 13676]        INFO    i.a.w.t.s.InvokeOperationsActivityImpl(invokeOperationsV2):96 - No post-replication operation(s) to perform.
2025-12-10 22:01:09,696 [Activity Executor taskQueue="SYNC", namespace="default": 13676]        INFO    i.a.w.t.s.InvokeOperationsActivityImpl(invokeOperationsV2):99 -
----- END POST REPLICATION OPERATIONS -----


2025-12-10 22:01:16,214 [Activity Executor taskQueue="CONNECTION_UPDATER", namespace="default": 38281]  WARN    i.t.i.a.ActivityTaskExecutors$BaseActivityTaskExecutor(execute):126 - Activity failure. ActivityId=1674be5f-4f0e-3aca-89a3-db
1a308229d3, activityType=AutoDisableFailingConnection, attempt=5
java.lang.NullPointerException: null
        at java.base/java.util.Objects.requireNonNull(Objects.java:233)
        at java.base/java.util.ImmutableCollections$Map1.<init>(ImmutableCollections.java:1112)
        at java.base/java.util.Map.of(Map.java:1365)
        at io.airbyte.workers.temporal.scheduling.activities.AutoDisableConnectionActivityImpl.autoDisableFailingConnection(AutoDisableConnectionActivityImpl.kt:39)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64)
        at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43)
        at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107)
        at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124)
        at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:290)
        at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:254)
        at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:217)
        at io.temporal.internal.worker.PollTaskExecutor.lambda$process$1(PollTaskExecutor.java:96)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)

2025-12-10 22:01:16,254 [workflow-method-connection_manager_1580ba39-5c58-473b-b1a4-e16f568...-c2caae3e-e63e-4ac6-a63e-731cbf328217]    ERROR   i.a.w.t.s.ConnectionManagerWorkflowImpl(runMandatoryActivityWithOutput):783 - [ACTIVITY-FAILU
RE] Connection null failed to run an activity.(AutoDisableConnectionActivityInput).  Connection manager workflow will be restarted after a delay of PT10M.
io.temporal.failure.ActivityFailure: Activity with activityType='AutoDisableFailingConnection' failed: 'Activity task failed'. scheduledEventId=42, startedEventId=43, activityId=1674be5f-4f0e-3aca-89a3-db1a308229d3, identity='1@airbyte-w
orker-58fbc95cf4-5krbn', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED
        at java.base/java.lang.Thread.getStackTrace(Thread.java:2451)
        at io.temporal.internal.sync.ActivityStubBase.execute(ActivityStubBase.java:49)
        at io.temporal.internal.sync.ActivityInvocationHandler.lambda$getActivityFunc$0(ActivityInvocationHandler.java:83)
        at io.temporal.internal.sync.ActivityInvocationHandlerBase.invoke(ActivityInvocationHandlerBase.java:68)
        at jdk.proxy2/jdk.proxy2.$Proxy83.autoDisableFailingConnection(Unknown Source)
        at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowImpl.failJob$lambda$7(ConnectionManagerWorkflowImpl.kt:533)
        at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowImpl.runMandatoryActivityWithOutput(ConnectionManagerWorkflowImpl.kt:780)
        at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowImpl.failJob(ConnectionManagerWorkflowImpl.kt:531)
        at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowImpl.reportFailure(ConnectionManagerWorkflowImpl.kt:485)
        at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowImpl.reportFailure$default(ConnectionManagerWorkflowImpl.kt:422)
        at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowImpl.run(ConnectionManagerWorkflowImpl.kt:233)
        at ConnectionManagerWorkflowImplProxy.run$accessor$kptiZqYh(Unknown Source)
        at ConnectionManagerWorkflowImplProxy$auxiliary$bBRonSwA.call(Unknown Source)
        at io.airbyte.micronaut.temporal.TemporalActivityStubInterceptor.execute(TemporalActivityStubInterceptor.kt:72)
        at ConnectionManagerWorkflowImplProxy.run(Unknown Source)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation$RootWorkflowInboundCallsInterceptor.execute(POJOWorkflowImplementationFactory.java:426)
        at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation.execute(POJOWorkflowImplementationFactory.java:374)
        at io.temporal.internal.sync.WorkflowExecutionHandler.runWorkflowMethod(WorkflowExecutionHandler.java:71)
        at io.temporal.internal.sync.SyncWorkflow.lambda$start$0(SyncWorkflow.java:144)
        at io.temporal.internal.sync.CancellationScopeImpl.run(CancellationScopeImpl.java:103)
        at io.temporal.internal.sync.WorkflowThreadImpl$RunnableWrapper.run(WorkflowThreadImpl.java:107)
        at io.temporal.worker.ActiveThreadReportingExecutor.lambda$submit$0(ActiveThreadReportingExecutor.java:54)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: io.temporal.failure.ApplicationFailure: type='java.lang.NullPointerException', nonRetryable=false
        at java.base/java.util.Objects.requireNonNull(Objects.java:233)
        at java.base/java.util.ImmutableCollections$Map1.<init>(ImmutableCollections.java:1112)
        at java.base/java.util.Map.of(Map.java:1365)
        at io.airbyte.workers.temporal.scheduling.activities.AutoDisableConnectionActivityImpl.autoDisableFailingConnection(AutoDisableConnectionActivityImpl.kt:39)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64)
        at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43)
        at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107)
        at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124)
        at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:290)
        at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:254)
        at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:217)
        at io.temporal.internal.worker.PollTaskExecutor.lambda$process$1(PollTaskExecutor.java:96)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)

2025-12-10 22:01:16,255 [workflow-method-connection_manager_1580ba39-5c58-473b-b1a4-e16f568...-c2caae3e-e63e-4ac6-a63e-731cbf328217]    INFO    i.a.w.t.s.ConnectionManagerWorkflowImpl(runMandatoryActivityWithOutput):797 - Waiting PT10M b
efore restarting the workflow for connection null, to prevent spamming temporal with restarts.
        2025-12-10 22:01:33,510 [Activity Executor taskQueue="CONNECTION_UPDATER", namespace="default": 38282]  WARN    i.t.i.a.ActivityTaskExecutors$BaseActivityTaskExecutor(execute):126 - Activity failure. ActivityId=a8bdd51e-8ab9-3962
-807e-8e6af7a815e3, activityType=CheckProgress, attempt=3
java.lang.NullPointerException: null
        at io.airbyte.workers.temporal.scheduling.activities.CheckRunProgressActivityImpl.checkProgress(CheckRunProgressActivityImpl.kt:21)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64)
        at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43)
        at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107)
        at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124)
        at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:290)
        at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:254)
        at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:217)
        at io.temporal.internal.worker.PollTaskExecutor.lambda$process$1(PollTaskExecutor.java:96)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
@kapa.ai
k
Sorry @Joao Pedro Ferreira Canutto, I'm not sure what you mean. Please try again. 👋
j
@kapa.ai worker:
Copy code
2025-12-10 22:00:36,044 [Activity Executor taskQueue="SYNC", namespace="default": 13675]        INFO    i.a.w.t.s.InvokeOperationsActivityImpl(invokeOperationsV2):65 -
----- START POST REPLICATION OPERATIONS -----


2025-12-10 22:00:36,048 [Activity Executor taskQueue="SYNC", namespace="default": 13675]        INFO    i.a.w.t.s.InvokeOperationsActivityImpl(invokeOperationsV2):96 - No post-replication operation(s) to perform.
2025-12-10 22:00:36,048 [Activity Executor taskQueue="SYNC", namespace="default": 13675]        INFO    i.a.w.t.s.InvokeOperationsActivityImpl(invokeOperationsV2):99 -
----- END POST REPLICATION OPERATIONS -----


2025-12-10 22:01:09,690 [Activity Executor taskQueue="SYNC", namespace="default": 13676]        INFO    i.a.w.t.s.InvokeOperationsActivityImpl(invokeOperationsV2):65 -
----- START POST REPLICATION OPERATIONS -----


2025-12-10 22:01:09,695 [Activity Executor taskQueue="SYNC", namespace="default": 13676]        INFO    i.a.w.t.s.InvokeOperationsActivityImpl(invokeOperationsV2):96 - No post-replication operation(s) to perform.
2025-12-10 22:01:09,696 [Activity Executor taskQueue="SYNC", namespace="default": 13676]        INFO    i.a.w.t.s.InvokeOperationsActivityImpl(invokeOperationsV2):99 -
----- END POST REPLICATION OPERATIONS -----


2025-12-10 22:01:16,214 [Activity Executor taskQueue="CONNECTION_UPDATER", namespace="default": 38281]  WARN    i.t.i.a.ActivityTaskExecutors$BaseActivityTaskExecutor(execute):126 - Activity failure. ActivityId=1674be5f-4f0e-3aca-89a3-db
1a308229d3, activityType=AutoDisableFailingConnection, attempt=5
java.lang.NullPointerException: null
        at java.base/java.util.Objects.requireNonNull(Objects.java:233)
        at java.base/java.util.ImmutableCollections$Map1.<init>(ImmutableCollections.java:1112)
        at java.base/java.util.Map.of(Map.java:1365)
        at io.airbyte.workers.temporal.scheduling.activities.AutoDisableConnectionActivityImpl.autoDisableFailingConnection(AutoDisableConnectionActivityImpl.kt:39)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64)
        at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43)
        at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107)
        at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124)
        at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:290)
        at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:254)
        at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:217)
        at io.temporal.internal.worker.PollTaskExecutor.lambda$process$1(PollTaskExecutor.java:96)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)

2025-12-10 22:01:16,254 [workflow-method-connection_manager_1580ba39-5c58-473b-b1a4-e16f568...-c2caae3e-e63e-4ac6-a63e-731cbf328217]    ERROR   i.a.w.t.s.ConnectionManagerWorkflowImpl(runMandatoryActivityWithOutput):783 - [ACTIVITY-FAILU
RE] Connection null failed to run an activity.(AutoDisableConnectionActivityInput).  Connection manager workflow will be restarted after a delay of PT10M.
io.temporal.failure.ActivityFailure: Activity with activityType='AutoDisableFailingConnection' failed: 'Activity task failed'. scheduledEventId=42, startedEventId=43, activityId=1674be5f-4f0e-3aca-89a3-db1a308229d3, identity='1@airbyte-w
orker-58fbc95cf4-5krbn', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED
        at java.base/java.lang.Thread.getStackTrace(Thread.java:2451)
        at io.temporal.internal.sync.ActivityStubBase.execute(ActivityStubBase.java:49)
        at io.temporal.internal.sync.ActivityInvocationHandler.lambda$getActivityFunc$0(ActivityInvocationHandler.java:83)
        at io.temporal.internal.sync.ActivityInvocationHandlerBase.invoke(ActivityInvocationHandlerBase.java:68)
        at jdk.proxy2/jdk.proxy2.$Proxy83.autoDisableFailingConnection(Unknown Source)
        at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowImpl.failJob$lambda$7(ConnectionManagerWorkflowImpl.kt:533)
        at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowImpl.runMandatoryActivityWithOutput(ConnectionManagerWorkflowImpl.kt:780)
        at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowImpl.failJob(ConnectionManagerWorkflowImpl.kt:531)
        at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowImpl.reportFailure(ConnectionManagerWorkflowImpl.kt:485)
        at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowImpl.reportFailure$default(ConnectionManagerWorkflowImpl.kt:422)
        at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowImpl.run(ConnectionManagerWorkflowImpl.kt:233)
        at ConnectionManagerWorkflowImplProxy.run$accessor$kptiZqYh(Unknown Source)
        at ConnectionManagerWorkflowImplProxy$auxiliary$bBRonSwA.call(Unknown Source)
        at io.airbyte.micronaut.temporal.TemporalActivityStubInterceptor.execute(TemporalActivityStubInterceptor.kt:72)
        at ConnectionManagerWorkflowImplProxy.run(Unknown Source)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation$RootWorkflowInboundCallsInterceptor.execute(POJOWorkflowImplementationFactory.java:426)
        at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation.execute(POJOWorkflowImplementationFactory.java:374)
        at io.temporal.internal.sync.WorkflowExecutionHandler.runWorkflowMethod(WorkflowExecutionHandler.java:71)
        at io.temporal.internal.sync.SyncWorkflow.lambda$start$0(SyncWorkflow.java:144)
        at io.temporal.internal.sync.CancellationScopeImpl.run(CancellationScopeImpl.java:103)
        at io.temporal.internal.sync.WorkflowThreadImpl$RunnableWrapper.run(WorkflowThreadImpl.java:107)
        at io.temporal.worker.ActiveThreadReportingExecutor.lambda$submit$0(ActiveThreadReportingExecutor.java:54)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: io.temporal.failure.ApplicationFailure: type='java.lang.NullPointerException', nonRetryable=false
        at java.base/java.util.Objects.requireNonNull(Objects.java:233)
        at java.base/java.util.ImmutableCollections$Map1.<init>(ImmutableCollections.java:1112)
        at java.base/java.util.Map.of(Map.java:1365)
        at io.airbyte.workers.temporal.scheduling.activities.AutoDisableConnectionActivityImpl.autoDisableFailingConnection(AutoDisableConnectionActivityImpl.kt:39)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64)
        at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43)
        at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107)
        at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124)
        at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:290)
        at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:254)
        at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:217)
        at io.temporal.internal.worker.PollTaskExecutor.lambda$process$1(PollTaskExecutor.java:96)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)

2025-12-10 22:01:16,255 [workflow-method-connection_manager_1580ba39-5c58-473b-b1a4-e16f568...-c2caae3e-e63e-4ac6-a63e-731cbf328217]    INFO    i.a.w.t.s.ConnectionManagerWorkflowImpl(runMandatoryActivityWithOutput):797 - Waiting PT10M b
efore restarting the workflow for connection null, to prevent spamming temporal with restarts.
        2025-12-10 22:01:33,510 [Activity Executor taskQueue="CONNECTION_UPDATER", namespace="default": 38282]  WARN    i.t.i.a.ActivityTaskExecutors$BaseActivityTaskExecutor(execute):126 - Activity failure. ActivityId=a8bdd51e-8ab9-3962
-807e-8e6af7a815e3, activityType=CheckProgress, attempt=3
java.lang.NullPointerException: null
        at io.airbyte.workers.temporal.scheduling.activities.CheckRunProgressActivityImpl.checkProgress(CheckRunProgressActivityImpl.kt:21)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64)
        at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43)
        at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107)
        at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124)
        at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:290)
        at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:254)
        at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:217)
        at io.temporal.internal.worker.PollTaskExecutor.lambda$process$1(PollTaskExecutor.java:96)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
k
A message from kapa.ai
The worker logs you shared show Temporal activities failing with 
NullPointerException
in the scheduler layer, not in the File connector itself:
Copy code
text ActivityType=AutoDisableFailingConnection ...  java.lang.NullPointerException   at io.airbyte.workers.temporal.scheduling.activities.AutoDisableConnectionActivityImpl.autoDisableFailingConnection(AutoDisableConnectionActivityImpl.kt:39) ... ActivityType=CheckProgress ... java.lang.NullPointerException   at io.airbyte.workers.temporal.scheduling.activities.CheckRunProgressActivityImpl.checkProgress(CheckRunProgressActivityImpl.kt:21)
This pattern is essentially the same as the “intermittent null pointer exception” / “job hangs until cancelled” issues reported for older Airbyte versions using Temporal: - Jobs or checks sometimes hang or error with 
WorkerException: Error while getting checking connection
and a 
NullPointerException
in the Temporal workflow/activities, not inside the connector. [[Intermittent NPE](https://discuss.airbyte.io/t/2145); [MSSQL NPE](https://discuss.airbyte.io/t/1492)] - In those cases, rerunning the job often succeeds, and the root cause is in the Temporal/scheduler code path rather than the source/destination config. In particular, the official resolution for a very similar NPE case was: > I have upgraded airbyte and temporal version. it got resolved. I suspect, issue with temporal engine. [[Intermittent NPE](https://discuss.airbyte.io/t/2145)] From the knowledge sources, there is no specific documented fix or configuration change for these NPEs in 
AutoDisableConnectionActivityImpl
or 
CheckRunProgressActivityImpl
on 1.8.x. What is documented: - These NPEs occur in Temporal activities and can cause jobs to hang or fail intermittently. [[Intermittent NPE](https://discuss.airbyte.io/t/2145); [MSSQL NPE](https://discuss.airbyte.io/t/1492)] - Upgrading Airbyte (and thus the bundled Temporal/worker code) resolved the issue for affected users. [[Intermittent NPE](https://discuss.airbyte.io/t/2145)] So, tying this back to your situation: - The File check pod for the 2 GB JSON file is already failing with “Failed to find output files from connector within timeout of 9 minute(s).” (sidecar timeout). [[Zendesk timeout](https://github.com/airbytehq/airbyte/issues/54684)] - Separately, your worker is also experiencing NPEs in Temporal scheduling activities (
AutoDisableFailingConnection
, 
CheckProgress
), which is consistent with known scheduler/Temporal bugs that cause intermittent failures and stuck jobs. [[Intermittent NPE](https://discuss.airbyte.io/t/2145); [MSSQL NPE](https://discuss.airbyte.io/t/1492)] Based on the available information, the grounded recommendations are: 1. Treat the NPEs as a scheduler/Temporal bug, not a misconfiguration. The knowledge sources do not document a config workaround; the only confirmed resolution is upgrading Airbyte (and bundled Temporal) to a newer version where these NPEs no longer occur. [[Intermittent NPE](https://discuss.airbyte.io/t/2145)] 2. As a short-term mitigation, rerun stuck jobs/checks. In the NPE reports, rerunning often succeeded, and users worked around it by canceling and re-triggering when jobs hung. [[MSSQL NPE](https://discuss.airbyte.io/t/1492)] 3. For the 2 GB JSON File check specifically, continue using smaller subsets (which you’ve seen work) while planning an upgrade. The platform clearly times out waiting for the connector’s output on that file; combined with the scheduler NPEs, this suggests your current 1.8.1/Temporal stack is unstable under this workload. [[Zendesk timeout](https://github.com/airbytehq/airbyte/issues/54684); [Intermittent NPE](https://discuss.airbyte.io/t/2145)] The knowledge sources do not provide more detailed internals or a targeted patch for 
AutoDisableConnectionActivityImpl
/ 
CheckRunProgressActivityImpl
on 1.8.1, so beyond upgrading and rerunning there isn’t a documented, precise fix I can point you to.
Open in Slack
PreviousNext
Powered by