< kapa ai> I have this issue when starting a job on a machin Airbyte #replication-ask-ai

<@U04S8PLJWMT> I have this issue when starting a j...

Pierre Leroux

03/04/2026, 12:49 PM

@kapa.ai I have this issue when starting a job on a machine of the type

m6a.2xlarge

on a kubernetes cluster but I don't have it on an instance of type

t3.2xlarge

The configuration stays the same. Here is the issue:

Copy code

Init container error encountered while processing workload for id: 69589781-7828-43c5-9f63-8925b1c1ccc2_15616_0_check. Encountered exception of type: class com.amazonaws.SdkClientException. Exception message: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)) at com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:66) at

kapa.ai

03/04/2026, 12:50 PM

The error you're seeing is a well-known issue in Airbyte Kubernetes deployments where the init container spawned for a connector check job does not have the AWS credentials (

AWS_ACCESS_KEY_ID

AWS_SECRET_ACCESS_KEY

) in its environment, even though other containers (like the connector-sidecar) do. The difference between

m6a.2xlarge

and

t3.2xlarge

instances is likely related to how the pods are scheduled and which environment variables are propagated — but the root cause is a configuration gap in your

values.yaml

for the

workload-launcher

. [Airbyte 1.2.0 EKS issue] Recommended Fix Add the following to your

values.yaml

for the

workload-launcher

component:

Copy code

workload-launcher:
  env_vars:
    # Controls the presence of the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the spawned pods' environment
    AWS_ASSUME_ROLE_SECRET_NAME: airbyte-config-secrets

    AWS_SECRET_MANAGER_ACCESS_KEY_ID_REF_NAME: airbyte-config-secrets
    AWS_SECRET_MANAGER_ACCESS_KEY_ID_REF_KEY: aws-secret-manager-access-key-id

    AWS_SECRET_MANAGER_SECRET_ACCESS_KEY_REF_NAME: airbyte-config-secrets
    AWS_SECRET_MANAGER_SECRET_ACCESS_KEY_REF_KEY: aws-secret-manager-secret-access-key

This ensures that the spawned connector pods (including init containers) have the necessary AWS credentials. [Airbyte 1.2.0 EKS issue] Additional Context There is a known limitation where AWS credential environment variables are present in the

connector-sidecar

container but not in the init container, causing the init container to fail when it tries to load credentials from AWS Secret Manager. This is a documented gap in Airbyte's Helm chart configuration. [Airbyte 1.2.0 EKS issue] If you are using AWS Secret Manager as your secrets backend, make sure your

global.secretsManager

section in

values.yaml

is also correctly configured with the appropriate

authenticationType

and credentials. [AWS secret manager error] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

Pierre Leroux

03/04/2026, 1:06 PM

@kapa.ai the solution doesn't work What would be the appropriate

authenticationType

for

global.secretsManager

kapa.ai

03/04/2026, 1:06 PM

Based on the documentation, there are two valid options for

authenticationType

under `global.secretsManager.awsSecretManager`: • credentials
– Use this when you want to authenticate using explicit AWS access key ID and secret access key stored in a Kubernetes secret (by default, in the

airbyte-config-secrets

secret, under the keys

aws-secret-manager-access-key-id

and

aws-secret-manager-secret-access-key

). • instanceProfile
– Use this when the compute infrastructure running Airbyte (e.g., the EC2 instances backing your EKS nodes) already has an IAM role with pre-existing permissions to read and write from AWS Secrets Manager. In this case, no explicit credentials need to be stored. [Secret Management docs] Given that you're running on EKS with

m6a.2xlarge

instances, if your node group or pod has an IAM role attached with the appropriate AWS Secrets Manager permissions, instanceProfile
is the recommended approach and avoids the credential-loading issue entirely:

Copy code

global:
  secretsManager:
    type: AWS_SECRET_MANAGER
    secretName: "airbyte-config-secrets"
    awsSecretManager:
      region: <aws-region>
      authenticationType: instanceProfile

If you're using

credentials

instead, make sure the Kubernetes secret

airbyte-config-secrets

actually contains the keys

aws-secret-manager-access-key-id

and

aws-secret-manager-secret-access-key

with valid values. [Secret Management docs] The fact that the error occurs on

m6a.2xlarge

but not

t3.2xlarge

with the same config could suggest an IAM role/instance profile is attached to one node group but not the other — in which case switching to

instanceProfile

on the correct node group (or ensuring the IAM role is attached to all node groups) would be the fix. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

Pierre Leroux

03/04/2026, 1:27 PM

@kapa.ai Other issue nothing to do with the previous one 3 connexions failed on monday, since then everything is back up to normal, but I still get slack alerts about the same sync failing over and over even though they are succesfull. How to stop that ?

kapa.ai

03/04/2026, 1:27 PM

This sounds like a notification configuration issue. Based on the knowledge sources, here's what you can check: Airbyte has a "Warning - Repeated Failures" notification that triggers when a connection has failed 20 times consecutively with only failed jobs in the past 4 days, and a "Sync Disabled - Repeated Failures" notification for 30 consecutive failures over 7 days. [Manage notifications] However, the knowledge sources don't explicitly explain a scenario where successful syncs continue to trigger failure alerts. What you can do to manage your Slack notifications: 1. Go to Settings > Notifications 2. Review and toggle off the notification types that are being noisy (e.g., Failed Syncs, Warning - Repeated Failures) 3. Paste your Slack webhook URL in the

Webhook URL

field if you need to reconfigure it 4. Click Save changes [Slack notification setup] Beyond that, the knowledge sources don't contain enough information to explain why you'd keep receiving failure alerts for connections that are now succeeding. I'd recommend: • Checking if there are other connections in the same workspace that are still failing (since notifications are workspace-wide and can be noisy if many syncs run) • Reaching out to Airbyte support or the Airbyte Slack community for further investigation 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

Pierre Leroux

03/04/2026, 1:49 PM

@kapa.ai lastly another issue On some connexion with s3 and clickhouse, the sync stopped working suddenly because the jobs could not find a sync file that hadn't been created (

output/848f70f4-4a49-485e-9825-15d8faf14522_1277_0_sync

for instance). Why did that happen and how to resolve it ?

kapa.ai

03/04/2026, 1:49 PM

The error pattern you're describing — where a job fails because it cannot find an output file like

output/848f70f4-4a49-485e-9825-15d8faf14522_1277_0_sync

— is consistent with a known platform-level issue in Airbyte where the connector sidecar times out waiting for the connector to produce its output file. From the knowledge sources, this manifests as: > "Failed to find output files from connector within timeout of 9 minute(s). Is the connector still running?" with

failureOrigin: airbyte_platform

. [Zendesk timeout issue] Additionally, for the ClickHouse destination specifically, there is an open known issue where syncs fail with: >

java.lang.IllegalStateException: Sync completed, but unflushed states were detected.

This suggests the ClickHouse v2 destination connector has a bug where states are not properly flushed before shutdown, which can cause sync jobs to fail or not complete their output correctly. [ClickHouse v2 issue] It's also worth noting that the legacy ClickHouse connector is archived and explicitly flagged as potentially having performance issues with large datasets. [ClickHouse deprecated] What you can do 1. Check which ClickHouse connector version you're using — if you're on the deprecated/archived version, that may be contributing to instability. 2. For the
IllegalStateException
on ClickHouse v2, this is an open bug — monitor the GitHub issue for a fix. 3. For the timeout/missing output file issue, the knowledge sources don't provide a definitive resolution beyond verifying network connectivity and that the connector pod is healthy and able to write its output. I'm sorry, but the knowledge sources don't contain enough information to provide a more specific root cause or resolution for the missing sync output file in your S3 + ClickHouse setup. I'd recommend filing a GitHub issue or reaching out on the Airbyte Slack community with your connector versions and logs. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

Open in Slack

Previous Next