@kapa.ai Please help me with this error after trying to change the bucket for a connection. I just removed it and tried with Batched standard inserts and I get the same error. 2026-03-27 182745.132Z connector-sidecar 2026-03-27 182745,132 [main] INFO i.a.c.ConnectorWatcher(saveConnectorOutput):186 - Writing output of 22f6c74f-5699-40ff-833c-4a879ea40133_9ab0d1e6-a199-4d48-a104-06101dcbf18e_0_check to the doc store 2026-03-27 182747.616Z connector-sidecar 2026-03-27 182747,615 [main] WARN i.a.a.c.c.ClientConfigurationSupport(generateDefaultRetryPolicy$lambda$5):79 - Retry attempt 1 of 5. Last response: null main Stream closed: EOF 2026-03-27 182749.551Z connector-sidecar 2026-03-27 182749,551 [main] WARN i.a.a.c.c.ClientConfigurationSupport(generateDefaultRetryPolicy$lambda$5):79 - Retry attempt 2 of 5. Last response: null 2026-03-27 182751.889Z connector-sidecar 2026-03-27 182751,888 [main] WARN i.a.a.c.c.ClientConfigurationSupport(generateDefaultRetryPolicy$lambda$5):79 - Retry attempt 3 of 5. Last response: null 2026-03-27 182754.051Z connector-sidecar 2026-03-27 182754,050 [main] WARN i.a.a.c.c.ClientConfigurationSupport(generateDefaultRetryPolicy$lambda$5):79 - Retry attempt 4 of 5. Last response: null 2026-03-27 182756.495Z connector-sidecar 2026-03-27 182756,495 [main] WARN i.a.a.c.c.ClientConfigurationSupport(generateDefaultRetryPolicy$lambda$5):79 - Retry attempt 5 of 5. Last response: null 2026-03-27 182756.535Z connector-sidecar 2026-03-27 182756,532 [main] ERROR i.a.a.c.c.ClientConfigurationSupport(generateDefaultRetryPolicy$lambda$7):95 - Retry attempts exceeded. 2026-03-27 182756.535Z connector-sidecar java.io.IOException: HTTP error: 500 Internal Server Error 2026-03-27 182756.535Z connector-sidecar at io.airbyte.api.client.interceptor.ThrowOn5xxInterceptor.intercept(ThrowOn5xxInterceptor.kt:16) 2026-03-27 182756.535Z connector-sidecar at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) 2026-03-27 182756.535Z connector-sidecar at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) 2026-03-27 182756.535Z connector-sidecar at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) 2026-03-27 182756.535Z connector-sidecar at dev.failsafe.okhttp.FailsafeCall.lambda$execute$0(FailsafeCall.java:117) 2026-03-27 182756.535Z connector-sidecar at dev.failsafe.Functions.lambda$get$0(Functions.java:46) 2026-03-27 182756.535Z connector-sidecar at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74) 2026-03-27 182756.535Z connector-sidecar at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187) 2026-03-27 182756.535Z connector-sidecar at dev.failsafe.CallImpl.execute(CallImpl.java:33) 2026-03-27 182756.535Z connector-sidecar at dev.failsafe.okhttp.FailsafeCall.execute(FailsafeCall.java:121) 2026-03-27 182756.535Z connector-sidecar at io.airbyte.api.client.generated.WorkloadOutputApi.writeWorkloadOutputWithHttpInfo(WorkloadOutputApi.kt:376) 2026-03-27 182756.535Z connector-sidecar at io.airbyte.api.client.generated.WorkloadOutputApi.writeWorkloadOutput(WorkloadOutputApi.kt:62) 2026-03-27 182756.535Z connector-sidecar at io.airbyte.workers.workload.WorkloadOutputWriter.writeOutputThroughServer(WorkloadOutputWriter.kt:82) 2026-03-27 182756.535Z connector-sidecar at io.airbyte.workers.workload.WorkloadOutputWriter.write(WorkloadOutputWriter.kt:65) 2026-03-27 182756.535Z connector-sidecar at io.airbyte.connectorSidecar.ConnectorWatcher.saveConnectorOutput(ConnectorWatcher.kt:187) 2026-03-27 182756.535Z connector-sidecar at io.airbyte.connectorSidecar.ConnectorWatcher.run(ConnectorWatcher.kt:80) 2026-03-27 182756.535Z connector-sidecar at io.airbyte.connectorSidecar.ApplicationKt.main(Application.kt:22) 2026-03-27 182756.535Z connector-sidecar at io.airbyte.connectorSidecar.ApplicationKt.main(Application.kt) 2026-03-27 182756.535Z connector-sidecar 2026-03-27 182756.536Z connector-sidecar 2026-03-27 182756,535 [main] ERROR i.a.a.c.c.ClientConfigurationSupport(generateDefaultRetryPolicy$lambda$3):63 - Failed to call unknown. Last response: null 2026-03-27 182756.536Z connector-sidecar java.io.IOException: HTTP error: 500 Internal Server Error 2026-03-27 182756.536Z connector-sidecar at io.airbyte.api.client.interceptor.ThrowOn5xxInterceptor.intercept(ThrowOn5xxInterceptor.kt:16) 2026-03-27 182756.536Z connector-sidecar at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) 2026-03-27 182756.536Z connector-sidecar at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201) 2026-03-27 182756.536Z connector-sidecar at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154) 2026-03-27 182756.536Z connector-sidecar at dev.failsafe.okhttp.FailsafeCall.lambda$execute$0(FailsafeCall.java:117) 2026-03-27 182756.536Z connector-sidecar at dev.failsafe.Functions.lambda$get$0(Functions.java:46) 2026-03-27 182756.536Z connector-sidecar at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74) 2026-03-27 182756.536Z connector-sidecar at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187) 2026-03-27 182756.536Z connector-sidecar at dev.failsafe.CallImpl.execute(CallImpl.java:33) 2026-03-27 182756.536Z connector-sidecar at dev.failsafe.okhttp.FailsafeCall.execute(FailsafeCall.java:121) 2026-03-27 182756.536Z connector-sidecar at io.airbyte.api.client.generated.WorkloadOutputApi.writeWorkloadOutputWithHttpInfo(WorkloadOutputApi.kt:376) 2026-03-27 182756.536Z connector-sidecar at io.airbyte.api.client.generated.WorkloadOutputApi.writeWorkloadOutput(WorkloadOutputApi.kt:62) 2026-03-27 182756.536Z connector-sidecar at io.airbyte.workers.workload.WorkloadOutputWriter.writeOutputThroughServer(WorkloadOutputWriter.kt:82) 2026-03-27 182756.536Z connector-sidecar at io.airbyte.workers.workload.WorkloadOutputWriter.write(WorkloadOutputWriter.kt:65) 2026-03-27 182756.536Z connector-sidecar at io.airbyte.connectorSidecar.ConnectorWatcher.saveConnectorOutput(ConnectorWatcher.kt:187) 2026-03-27 182756.536Z connector-sidecar at io.airbyte.connectorSidecar.ConnectorWatcher.run(ConnectorWatcher.kt:80) 2026-03-27 182756.536Z connector-sidecar at io.airbyte.connectorSidecar.ApplicationKt.main(Application.kt:22) 2026-03-27 182756.536Z connector-sidecar at io.airbyte.connectorSidecar.ApplicationKt.main(Application.kt) 2026-03-27 182756.536Z connector-sidecar 2026-03-27 182756.537Z connector-sidecar 2026-03-27 182756,536 [main] ERROR i.a.c.ConnectorWatcher(handleException):200 - Error performing operation: io.airbyte.workers.workload.exception.DocStoreAccessException 2026-03-27 182756.537Z connector-sidecar io.airbyte.workers.workload.exception.DocStoreAccessException: Unable to write output for 22f6c74f-5699-40ff-833c-4a879ea40133_9ab0d1e6-a199-4d48-a104-06101dcbf18e_0_check 2026-03-27 182756.537Z connector-sidecar at io.airbyte.workers.workload.WorkloadOutputWriter.writeOutputThroughServer(WorkloadOutputWriter.kt:86) 2026-03-27 182756.537Z connector-sidecar at io.airbyte.workers.workload.WorkloadOutputWriter.write(WorkloadOutputWriter.kt:65) 2026-03-27 182756.537Z connector-sidecar at io.airbyte.connectorSidecar.ConnectorWatcher.saveConnectorOutput(ConnectorWatcher.kt:187) 2026-03-27 182756.537Z connector-sidecar at io.airbyte.connectorSidecar.ConnectorWatcher.run(ConnectorWatcher.kt:80) 2026-03-27 182756.537Z connector-sidecar at io.airbyte.connectorSidecar.ApplicationKt.main(Application.kt:22) 2026-03-27 182756.537Z connector-sidecar at io.airbyte.connectorSidecar.ApplicationKt.main(Application.kt) 2026-03-27 182756.537Z connector-sidecar Caused by: java.io.IOException: HTTP error: 500 Internal Server Error 2026-03-27 182756.537Z connector-sidecar at io.airbyte.api.client.interceptor.ThrowOn5xxInterceptor.intercept(ThrowOn5xxInterceptor.kt:16) 2026-03-27 182756.537Z connector-sidecar at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109) 2026-03-27 182756.537Z connector-sidecar at

The error you're seeing is a platform-level infrastructure issue, not directly related to your BigQuery destination configuration (whether GCS staging or Batched Standard Inserts). The key error is:

DocStoreAccessException: Unable to write output for 22f6c74f-..._check
Caused by: java.io.IOException: HTTP error: 500 Internal Server Error

This occurs when the

connector-sidecar

tries to write its check output to the Airbyte workload API server via

WorkloadOutputApi.writeWorkloadOutput

, but receives an HTTP 500 error after exhausting all 5 retry attempts. [HTTP 500 issue] This is a known issue in Airbyte OSS, particularly on Kubernetes/Helm deployments. It is related to the connector-sidecar not being able to successfully communicate with the workload API server. [Connector-Sidecar Auth Issue] To help narrow down the root cause, could you provide: 1. Airbyte version you are running 2. Deployment method (Docker Compose, Kubernetes/Helm, etc.) 3. Did this work before? If so, did you recently upgrade or change any configuration? If you are on Kubernetes/Helm, this may be related to a known issue where

WORKLOAD_API_AUTH_HEADER

and

WORKLOAD_API_BEARER_TOKEN

environment variables are not properly propagated to connector-sidecar containers. There is currently no straightforward Helm value to inject these into dynamically created sidecar containers. [Connector-Sidecar Auth Issue] Note that switching from GCS staging to Batched Standard Inserts would not resolve this error, since the problem is at the platform communication layer, not the connector configuration level. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

airbyte version 2.0.1, deployed via argocd with helm, this worked fine until I tried to change the bucket for gcs staging and ran the test

Thanks for the additional context. You're running Airbyte v2.0.1 on Kubernetes via Helm (ArgoCD), and the issue started after you attempted to change the GCS staging bucket and ran a connection test. This is consistent with a known bug reported for Airbyte v2.0.1 on Helm. There are a couple of relevant issues that may apply to your situation: 1. GCS credentialsJsonPath Bug (Likely Culprit) There is a known bug in the v2 Helm chart (App Version 2.0.1, Chart 2.0.18) where the Workload Launcher ignores the

credentialsJsonPath

when building job pod volume mounts, hardcoding the secret mount path to

/secrets/gcp-creds/

instead. If your GCS credentials path doesn't match that hardcoded path, the connector-sidecar crashes immediately. [GCS credentialsJsonPath bug] Workaround: Remove the

credentialsJsonPath

from your

values.yaml

and rely solely on

credentialsJsonSecretKey

, which forces the system to use the hardcoded

/secrets/gcp-creds/

path:

storage:
  type: gcs
  secretName: airbyte-mounted-secrets
  gcs:
    projectId: my-gcp-project
    credentialsJsonSecretKey: airbyte-gcp-json
    # Remove credentialsJsonPath entirely

2. Connector-Sidecar Auth Token Propagation Issue Additionally, in v2.0.1, there is a known issue where

WORKLOAD_API_AUTH_HEADER

and

WORKLOAD_API_BEARER_TOKEN

are not propagated to connector-sidecar containers, causing HTTP 500 errors when writing output. [Connector-Sidecar Auth Issue] Given that your setup worked before the bucket change, the GCS

credentialsJsonPath

bug (#1) is the most likely cause of your current failure. I'd recommend trying the workaround above first. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai is there an upgrade that could fix this?

Based on the knowledge sources available, I can see that the GCS

credentialsJsonPath

bug (issue #73739) has been escalated to the internal platform team for investigation and a fix, but there is no mention of a specific chart version that resolves it. [GCS credentialsJsonPath bug] The knowledge sources do not explicitly confirm that any currently released chart version fully fixes these issues. The latest chart version listed is 2.0.18 (App Version 2.0.1), which is the same version where these bugs are reported. What I can confirm from the knowledge sources: • The GCS

credentialsJsonPath

bug is tracked internally but no fix release is mentioned. [GCS credentialsJsonPath bug] • The connector-sidecar auth issue is also tracked internally. [Connector-Sidecar Auth Issue] In the meantime, the recommended workaround is to remove

credentialsJsonPath

from your

values.yaml

and use only

credentialsJsonSecretKey

with lowercase

gcs

as the storage type:

global:
  storage:
    type: "gcs"
    secretName: your-secret-name
    gcs:
      projectId: your-gcp-project
      credentialsJsonSecretKey: "gcp.json"

I'd recommend checking the Airbyte ArtifactHub and release notes directly for the latest chart versions and whether a fix has been released since these issues were filed. [Upgrading Airbyte] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai I am not using this I am using HMAC in the UI to connect to the storage.

Thank you for clarifying. Since you're using HMAC key authentication in the UI (not GCS service account/credentialsJsonPath), the GCS

credentialsJsonPath

Helm bug is not your issue. Your error — the connector-sidecar failing with

DocStoreAccessException: Unable to write output

and HTTP 500 — is then more likely related to the connector-sidecar authentication issue in v2.0.1, where the sidecar containers don't receive the necessary tokens to communicate with the workload API server. [Connector-Sidecar Auth Issue] A few things to check: 1. Verify

API_AUTHORIZATION_ENABLED

: Ensure

API_AUTHORIZATION_ENABLED=true

is set in your

airbyte-airbyte-env

ConfigMap. Without this, the

/api/v1/workload/{id}/output

endpoint may not be reachable. [Connector-Sidecar Auth Issue] 2. Check if the issue is specific to the connection test: Since this started when you ran a test after changing the GCS bucket, it's worth verifying whether existing syncs (not tests) are also failing, or if it's isolated to the check operation. 3. HMAC key permissions: Ensure the service account linked to your HMAC key has the required GCS permissions: [GCS Destination Docs]storage.multipartUploads.abort storage.multipartUploads.create storage.objects.create storage.objects.delete storage.objects.get storage.objects.listOne community report found that granting Storage Object Admin to the service account resolved a similar "failed status but file present" issue. [GCS Failed Status] The knowledge sources don't provide a definitive fix for the HTTP 500 workload API issue in v2.0.1 beyond what's already been mentioned. I'd recommend opening or tracking issue #70451 for updates. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai do I need to do a rolling restart of all components in the deployment?

Based on the knowledge sources, a rolling restart of relevant components is indeed part of the recommended workaround for the workload API authentication issue in Kubernetes/Helm deployments. Specifically, the workaround involves restarting the workload API server and worker deployments after aligning the

WORKLOAD_API_BEARER_TOKEN

. For example: [source-postgres 502 issue]

kubectl -n airbyte-abctl rollout restart deployment airbyte-abctl-worker
kubectl -n airbyte-abctl rollout restart deployment airbyte-abctl-workload-api-server

However, the knowledge sources do not explicitly state that all components need to be restarted — only the worker and workload API server are mentioned in the workarounds. Some users also restarted the server deployment:

kubectl -n airbyte-abctl rollout restart deploy/airbyte-abctl-workload-api-server airbyte-abctl-worker airbyte-abctl-server

[source-postgres 502 issue] That said, since your setup was working before the bucket change and you haven't made token/auth changes, a restart alone may not resolve the underlying HTTP 500 issue. The knowledge sources don't confirm that a restart without any configuration change would fix the problem in your specific case. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.