@kapa.ai My replication jobs are failing when scheduled on AWS fargate profile for eks with the following error:

Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The provided token is malformed or otherwise invalid.

I am using IRSA for authentication. My airbyte version is 1.5.7 and deployed it using helm chart on EKS.

That S3 error indicates that the AWS credentials the job container is using are invalid/garbled, not that S3 itself is misconfigured. Given your setup (EKS + Helm + IRSA + S3), there are a few known, related issues in the docs and GitHub/Forum threads: 1. Confirm IRSA is actually being used by all Airbyte pods (including workers on Fargate) With Helm, the service account and its IRSA annotation must be at the correct level in

values.yaml

, and all workload pods must mount that service account. Several users hit S3 auth failures when pods were either: - using the node role instead of the IRSA role, or - not getting any service account at all. [[IRSA issue](https://github.com/airbytehq/airbyte/issues/53652); [workload SA issue](https://github.com/airbytehq/airbyte/issues/50979)] Make sure in your Helm values:

yaml    global:      serviceAccountName: airbyte-admin      storage:        type: S3        bucket:          log: <bucket>          state: <bucket>          workloadOutput: <bucket>        s3:          region: <region>          authenticationType: instanceProfile   # critical for IRSA    serviceAccount:      annotations:        <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: "arn:aws:iam::<account_id>:role/<Airbyte-role>"

And verify at runtime:

bash    kubectl get pod <job-pod> -o jsonpath='{.spec.serviceAccountName}'    kubectl get pod <job-pod> -o jsonpath='{.spec.serviceAccount.annotations.eks\.amazonaws\.com/role-arn}'

If the job pod on Fargate is not using the IRSA-annotated service account, S3 auth will fail. [[IRSA values example](https://github.com/airbytehq/airbyte/issues/53652)] 2. Remove conflicting static AWS credentials from Airbyte pods In one reported case, Helm-based deployments injected

AWS_ACCESS_KEY_ID

/

AWS_SECRET_ACCESS_KEY

(for MinIO/S3 logging) into the server pod, which interfered with S3 access and produced 4xx auth errors. Removing those env vars fixed the issue. [[Invalid Access Key thread](https://discuss.airbyte.io/t/1669)] On EKS/Fargate with IRSA, you generally should not set static AWS credentials on the pods. Check:

bash    kubectl exec -it <server-or-job-pod> -- env | egrep 'AWS_ACCESS_KEY_ID|AWS_SECRET_ACCESS_KEY|AWS_SESSION_TOKEN'

If present from your chart overrides or environment, remove them and rely solely on IRSA. 3. Validate the IAM role and trust for your IRSA service account Users who saw S3 permission/auth problems with IRSA resolved them by: - Fixing the IAM role trust policy to allow the specific Kubernetes service account (

airbyte-admin

or your equivalent) to assume the role. - Ensuring the role actually has S3 permissions matching Airbyte’s S3 policy examples. [[IRSA fix comment](https://github.com/airbytehq/airbyte/issues/53652); [AWS S3 policy](https://docs.airbyte.com/platform/deploying-airbyte/infrastructure/aws)] Double‑check: - Trust policy: references the correct cluster OIDC provider and Kubernetes service account. - Permissions: allow the bucket actions Airbyte needs (

s3:ListBucket

,

s3:GetObject

,

s3:PutObject

, etc.) on your Airbyte buckets. [[AWS S3 policy](https://docs.airbyte.com/platform/deploying-airbyte/infrastructure/aws)] 4. Fargate specifics (possible limitation in our sources) The knowledge sources do not explicitly document IRSA + Fargate interaction or “malformed token” on Fargate, only IRSA on EKS nodes. They also don’t cover Airbyte 1.5.7 specifically for this error. I cannot confirm from the provided docs whether there is a known Fargate‑specific bug. Because the error is about a malformed/invalid token, the most concrete, source‑backed steps you can take from above are: - Ensure

authenticationType: instanceProfile

and a correctly annotated service account in Helm values. [[IRSA values example](https://github.com/airbytehq/airbyte/issues/53652)] - Ensure all job pods (including those running on Fargate) actually use that service account and its annotation. - Remove any static AWS credential env vars from Airbyte pods that could override IRSA. [[Invalid Access Key thread](https://discuss.airbyte.io/t/1669)] - Confirm the IAM role trust and S3 policies match the documented patterns. [[AWS S3 policy](https://docs.airbyte.com/platform/deploying-airbyte/infrastructure/aws)] If, after these checks, the error persists specifically only on Fargate, the knowledge sources do not contain further Fargate‑specific guidance; in that case you may want to open or update a GitHub issue with your pod spec, service account YAML, and full S3 stack trace so the team can investigate.

@kapa.ai Even if the job pods use the IRSA, I can verify the service account attached to the job pods. But I have checked the envs for the AWS static credential

AWS_ACCESS_KEY_ID

/

AWS_SECRET_ACCESS_KEY

exist on the job pods without any values. E though I haven't specified any of these as I tried to use IRSA.

A message from kapa.ai

@kapa.ai here are my helm value configs for s3:

storage:
    type: S3 # default storage used
    storageSecretName: "s3-secret"

    # S3
    bucket:
      ## S3 bucket names that you've created. We recommend storing the following all in one bucket.
      log: log-bucket
      state: state-bucket
      workloadOutput: wo-bucket
    s3:
      region: "us-east-1" ## e.g. us-east-1
      authenticationType: instanceProfile

How can I remove the static AWS credentials from replication job pods? The secret in storage doesn't contain anything should I also remove it?

A message from kapa.ai