hi @kapa.ai i get error : Unable to persist the job Output, check the document store credentials. when i try to retest existing connections. the existing ones work fine, but also logs are not accessible. Time almost doubled for the existing connections.

This error is not fully documented in the provided sources, but it matches a known class of issues where Airbyte cannot write job results to its “document store” (the storage used for logs and job outputs). In the logs this appears as a

DocStoreAccessException: Unable to write output for ..._check

. [Helm SA issue; HTTP 500 doc store] Because of that: • Connection tests fail (they can’t persist the check output). • Existing syncs may still run, but ◦ Logs are not accessible, and ◦ Runtime can increase because the system is retrying or timing out when trying to write outputs. [Connector-sidecar auth; HTTP 500 doc store] From the issues, the most common root causes are: 1. Storage / document store credentials or access are wrong or missing ◦ For OSS with S3/MinIO or similar, misconfigured credentials or buckets can cause “Unable to write output” errors. [Minio / S3 doc store; S3 config] ◦ In Kubernetes/Helm, missing or incorrect service account / auth tokens for the connector-sidecar or workload pods prevent them from authenticating to the workload API server / doc store. [Helm SA issue; Connector-sidecar auth] 2. Known OSS bug with connector-sidecar auth to the workload API ◦ In some OSS versions, connector-sidecar containers don’t receive the auth credentials they need, so they can run the connector but fail when writing the output. [Connector-sidecar auth; HTTP 500 doc store] Because I don’t have your exact deployment details, I can’t give a precise fix, but here are the next concrete steps that align with the issues above: 1. Share environment details (for further help) The maintainers in similar issues ask for: [HTTP 500 doc store; S3 config] ◦ Airbyte version (OSS vs Cloud, and exact version). ◦ Deployment method (Docker Compose vs Kubernetes/Helm). ◦ Where your document store/logs are configured (S3/MinIO/other). 2. If you’re on Kubernetes/Helm ◦ Check that workload / connector-sidecar pods have the correct service account and can authenticate to the workload API server. Missing SA caused exactly this

DocStoreAccessException

in one report. [Helm SA issue] ◦ If you’re on a version mentioned in the connector-sidecar auth issue, this may be a *known bug*; upgrading to a fixed version (or applying the recommended Helm chart fixes) is usually required. [Connector-sidecar auth; HTTP 500 doc store] 3. If you’re using S3/MinIO or external object storage ◦ Re‑verify the credentials, bucket names, and regions used for Airbyte’s internal storage (logs / job outputs), not just for S3 destination connectors. [Minio / S3 doc store; S3 config] ◦ Confirm that the Airbyte pods/containers can actually write to that bucket (IAM policy / access key / instance profile). Given the symptoms you describe (tests fail, existing connections run but logs inaccessible and slower), this is almost certainly a doc store / workload output write problem, not an issue with the individual connectors themselves. If you can share: • Airbyte version, • Deployment type, • How your storage (logs/job output) is configured, I can map your situation more directly to the closest known issue and suggest a more specific remediation path. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

but i did not made any update to the application. we did not touched it @kapa.ai

Not changing the Airbyte version or config doesn’t rule out this class of errors. In the issues and forum threads, similar “suddenly everything starts failing” situations were caused by environment changes around Airbyte, not Airbyte itself: • Storage / disk / object store issues: ◦ Running out of disk or losing access to the storage used for logs/job outputs can suddenly break tests and log access, even if you never touched Airbyte. [Syncs failing / disk full; Minio / S3 doc store] ◦ S3 / MinIO / GCS / Azure credentials expiring, IAM policies changing, or buckets being modified will cause “Unable to write output” without any Airbyte upgrade. [S3 config] • Kubernetes / Helm environments: ◦ Changes to service accounts, RBAC, or workload API auth (e.g., cluster hardening, token rotation) can cause connector-sidecar pods to lose permission to write job output, again without touching the Airbyte release. [Helm SA issue; Connector-sidecar auth; HTTP 500 doc store] The knowledge sources don’t give a single root cause for your exact message, but they consistently point to doc store / storage / auth problems rather than connector logic. To move forward, you’ll need to check the environment around Airbyte: 1. Confirm deployment + storage setup ◦ Are you on Docker or Kubernetes/Helm? ◦ What is configured for Airbyte’s internal storage (S3/GCS/Azure/local)? [Storage values] 2. If using S3/GCS/Azure for logs/state ◦ Verify the credentials (Kubernetes secret or env vars) are still valid and haven’t expired/rotated. [Storage values; S3 config] ◦ Check that the bucket/container still exists and IAM / access policies allow write. 3. If on Kubernetes/Helm ◦ Check that workload / connector pods still have the correct service account and can talk to the workload API; missing SA or auth misconfig leads to

DocStoreAccessException

when writing output. [Helm SA issue; Connector-sidecar auth] If you can share: • Airbyte version, • Deployment method (Docker vs K8s/Helm), • What you use for Airbyte’s log/state storage, I can narrow this down to the closest documented issue and suggest more concrete checks. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

i have airbyte 2.0.1 on k8s @kapa.ai

On Airbyte OSS 2.0.1 on Kubernetes, your symptoms match a known issue with the document store / workload API, not with the connectors themselves: • Connectors run and complete successfully. • But the connector-sidecar cannot write the job output, causing: ◦ Errors like

Unable to persist the job Output, check the document store credentials

/

DocStoreAccessException: Unable to write output for ..._check

. ◦ Tests / re-tests fail and logs are not accessible. ◦ Jobs take longer due to retries/timeouts when writing output. [HTTP 500 doc store; Connector-sidecar auth] For Airbyte 2.0.1 on K8s, the knowledge sources highlight two closely related problems: 1. Workload API authorization / document store write failures In 2.0.1, the connector-sidecar writes output via the workload API server. If auth is misconfigured, you get exactly these errors. [Connector-sidecar auth; HTTP 500 doc store] Key points from the issues: ◦ The endpoint

/api/v1/workload/{id}/output

may return: ▪︎

404

until

API_AUTHORIZATION_ENABLED=true

is set in the

airbyte-airbyte-env

ConfigMap. ▪︎ Then

401

/

500

if the connector-sidecar does not have the auth token. [Connector-sidecar auth] ◦ Users configured:

workloadApiServer:
         extraEnv:
           - name: WORKLOAD_API_AUTH_HEADER
             value: "Authorization"
           - name: WORKLOAD_API_BEARER_TOKEN
             value: "[token]"

       worker:
         extraEnv:
           - name: WORKLOAD_API_AUTH_HEADER
             value: "Authorization"
           - name: WORKLOAD_API_BEARER_TOKEN
             value: "[same-token]"

       workloadLauncher:
         extraEnv:
           - name: WORKLOAD_API_AUTH_HEADER
             value: "Authorization"
           - name: WORKLOAD_API_BEARER_TOKEN
             value: "[same-token]"

but the connector-sidecar pods still did not receive

WORKLOAD_API_AUTH_HEADER

/

WORKLOAD_API_BEARER_TOKEN

. [Connector-sidecar auth] ◦ This is described as a regression / bug in 2.0.1 OSS: the Helm chart has no value to inject these vars into connector-sidecar, and the platform doesn’t propagate them correctly. [Connector-sidecar auth] 2. Resulting error: DocStoreAccessException / “Unable to write output” When the sidecar can run the connector but cannot authenticate to the workload API / doc store, you see

DocStoreAccessException: Unable to write output for ..._check

and HTTP 500s, exactly like what you’re describing. [HTTP 500 doc store; Helm SA issue] ------------------------- What you can check / do on 2.0.1 K8s Based on the issues, here are the concrete checks that are directly supported by the knowledge sources: 1. Confirm workload API auth is enabled In the

airbyte-airbyte-env

ConfigMap, verify: ◦

API_AUTHORIZATION_ENABLED=true

◦

WORKLOAD_API_SERVER_ENABLED=true

◦

WORKLOAD_API_HOST

points to the workload API service. [Connector-sidecar auth] 2. Verify env vars on the different pods From the issues, the pattern is: ◦

workload-api-server

pod: has

WORKLOAD_API_AUTH_HEADER

and

WORKLOAD_API_BEARER_TOKEN

. ◦

airbyte-workload-launcher

pod: also has them. ◦ connector-sidecar pods: missing them, only have

WORKLOAD_API_HOST

and timeout/retry vars. [Connector-sidecar auth] You can check similarly:

# Example: check a connector-sidecar pod
   kubectl get pod <some-workload-pod> -n <namespace> -o json \
     | jq -r '.spec.containers[] 
       | select(.name=="connector-sidecar") 
       | .env[] 
       | select(.name | contains("WORKLOAD")) 
       | "\(.name): \(.value)"'

If

WORKLOAD_API_AUTH_HEADER

and

WORKLOAD_API_BEARER_TOKEN

are missing there, you are hitting the same bug. [Connector-sidecar auth] 3. Understand the limitation in 2.0.1 The knowledge sources explicitly state: ◦ There is no documented Helm value in chart v2 that injects

WORKLOAD_API_AUTH_HEADER

/

WORKLOAD_API_BEARER_TOKEN

into connector-sidecar. ◦ In working examples, the sidecar token comes from a secret wired into platform templates, not from user-configurable

extraEnv

. ◦ This is “very likely a bug or regression rather than a config oversight.” [Connector-sidecar auth] That means on 2.0.1 + Helm, you may not be able to fully fix this just with values.yaml; it likely requires a platform/Helm update or custom patching of the job pod templates. ------------------------- Practical next steps Given you’re on 2.0.1 on K8s, and assuming you confirm the missing env vars on connector-sidecar: 1. Check the GitHub issue and track fixes Your situation matches the open issue: [airbytehq/airbyte#70451 – Connector-Sidecar Cannot Authenticate to Workload API Server in v2.0.1 OSS][Connector-sidecar auth] and the related doc store error issue: [airbytehq/airbyte#72826 – HTTP 500 / DocStoreAccessException][HTTP 500 doc store] The knowledge sources do not yet document a definitive fix in 2.0.1; they treat it as a known bug. 2. Short-term options (based on the issues) The sources suggest, but do not fully detail, that workarounds involve: ◦ Ensuring

API_AUTHORIZATION_ENABLED=true

and workload API env vars are set. [Connector-sidecar auth; HTTP 500 doc store] ◦ Customizing the K8s job/sidecar templates (outside of standard Helm values) to inject

WORKLOAD_API_AUTH_HEADER

and

WORKLOAD_API_BEARER_TOKEN

into connector-sidecar. However, the exact patch is not described in the provided knowledge sources, so I cannot give a concrete manifest snippet without guessing. 3. Medium-term: plan for an upgrade / patch Since this is described as a regression in 2.0.1 OSS on K8s, the most robust path is: ◦ Monitor the linked GitHub issues for a fix or a Helm chart update that correctly propagates the auth env vars to connector-sidecar. [Connector-sidecar auth] ◦ Upgrade to a version where this bug is resolved once available. ------------------------- To summarize: on Airbyte 2.0.1 on Kubernetes, your error and symptoms align with a known bug where connector-sidecar cannot authenticate to the workload API / document store because it’s missing auth env vars, leading to

Unable to write output

and inaccessible logs. The knowledge sources do not yet provide a complete, supported workaround via Helm values alone; they point to this as a platform/Helm regression that likely requires an update or custom template patching. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

but how come it worked so far @kapa.ai

The knowledge sources don’t explain why this would start happening after a period of working fine on 2.0.1, so I can’t give a definitive root cause. I can only outline the likely classes of reasons, based on what’s documented. For Airbyte 2.0.1 on Kubernetes, we know: • There is a bug/regression where connector-sidecar often does not receive

WORKLOAD_API_AUTH_HEADER

/

WORKLOAD_API_BEARER_TOKEN

, so it can run the connector but fails when writing output to the workload API / document store. [Connector-sidecar bug; Doc store 500] • This manifests exactly as: ◦

DocStoreAccessException: Unable to write output for ..._check

◦ UI errors about being unable to persist job output / check document store credentials ◦ Logs not accessible and tests timing out. [Doc store 500; Connector-sidecar bug] Why it used to work and now doesn’t, even without an Airbyte upgrade, is not described in the issues. Plausible categories (all outside Airbyte itself) are: • Cluster / environment changes: ◦ Changes to Kubernetes (service accounts, RBAC, secrets, ConfigMaps, or token rotation) can break the sidecar’s ability to authenticate or reach the workload API, even if the Airbyte version and Helm values stayed the same. [Doc store 500] • Storage / document store changes: ◦ Changes to S3 / object storage credentials, IAM policies, or buckets used for workload output/logs can suddenly cause “Unable to write output” without touching Airbyte. [Doc store 500] But the provided sources do not document a specific “it worked for a while then broke” scenario, so any more detailed explanation would be speculation. Given your situation, the most concrete thing you can do (and that the issues explicitly recommend) is: 1. Inspect a failing connector-sidecar pod and confirm whether it has: ◦

WORKLOAD_API_HOST

(likely yes), but ◦ Missing

WORKLOAD_API_AUTH_HEADER

/

WORKLOAD_API_BEARER_TOKEN

. [Connector-sidecar bug] 2. Compare with your workload-api-server and workload-launcher pods, which likely do have those auth env vars. [Connector-sidecar bug] If you see that exact pattern, you’re hitting the same known regression: the platform isn’t propagating the auth vars into connector-sidecar, and the issues treat this as a bug in 2.0.1 rather than a misconfiguration. [Connector-sidecar bug] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

so upgrading will solve this @kapa.ai

I’m not fully sure that upgrading will solve it, because the provided sources do not state that a specific version after 2.0.1 definitively fixes this bug. What the sources do say: • Your symptoms on 2.0.1 OSS on K8s match a known regression where connector-sidecar does not receive

WORKLOAD_API_AUTH_HEADER

/

WORKLOAD_API_BEARER_TOKEN

, causing

DocStoreAccessException

/ “Unable to write output” and HTTP 500s. [Connector-sidecar bug; Doc store 500] • This is described as “very likely a bug or regression rather than a config oversight,” and there is no documented Helm value in v2 to inject those vars into connector-sidecar. [Connector-sidecar bug] However, the issues: • Do not explicitly say “upgrade to version X and it’s fixed.” • Do not document a confirmed workaround for 2.0.1 purely via Helm values. So: • Upgrading to a newer Airbyte + Helm chart version is the most likely long‑term fix, because this is treated as a platform/Helm regression. • But based on the current knowledge sources, I cannot guarantee that “just upgrading” will solve it; you should: ◦ Check the release notes / GitHub issues for the target version to confirm that the connector-sidecar/workload API auth bug is mentioned as fixed. ◦ Verify after upgrade that connector-sidecar pods now contain

WORKLOAD_API_AUTH_HEADER

and

WORKLOAD_API_BEARER_TOKEN

in their env. [Connector-sidecar bug] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.