https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> I have deployed Airbyte 1.6.0 using...
# replication-ask-ai
m
@kapa.ai I have deployed Airbyte 1.6.0 using Helm on an EKS cluster. Previously, we were scheduling the jobs on managed node groups. To provide access to the AWS resources for the jobs that we were using IRSA. We tried to schedule the replication jobs to Fargate and were getting the following error: """ Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The provided token is malformed or otherwise invalid """ To resolve this, we decided to use the access id and secret key. But we are not sure how will get the other permissions?
k
You don’t need to “discover” extra permissions from Airbyte itself – you must explicitly grant the needed AWS permissions to whichever IAM principal you’re now using (IAM user or role). The docs only specify the minimum policies Airbyte needs; they don’t auto‑provision anything. From the Airbyte platform docs, when running on AWS you should attach an IAM policy like this to the IAM user (access key/secret) or role that Airbyte uses so it can talk to S3 and (optionally) Secrets Manager: S3 policy (attach to your IAM user/role) [[AWS infra](https://docs.airbyte.com/platform/deploying-airbyte/infrastructure/aws); [AWS policies appendix](https://docs.airbyte.com/platform/enterprise-setup/implementation-guide#aws-policies-appendix)]
Copy code
json {   "Version": "2012-10-17",   "Statement": [     { "Effect": "Allow", "Action": "s3:ListAllMyBuckets", "Resource": "*" },     {       "Effect": "Allow",       "Action": ["s3:ListBucket", "s3:GetBucketLocation"],       "Resource": "arn:aws:s3:::YOUR-S3-BUCKET-NAME"     },     {       "Effect": "Allow",       "Action": [         "s3:PutObject",         "s3:PutObjectAcl",         "s3:GetObject",         "s3:GetObjectAcl",         "s3:DeleteObject"       ],       "Resource": "arn:aws:s3:::YOUR-S3-BUCKET-NAME/*"     }   ] }
If you also use AWS Secrets Manager with Airbyte, add this policy as well (or merge into one policy): [[AWS infra](https://docs.airbyte.com/platform/deploying-airbyte/infrastructure/aws); [AWS policies appendix](https://docs.airbyte.com/platform/enterprise-setup/implementation-guide#aws-policies-appendix)]
Copy code
json {   "Version": "2012-10-17",   "Statement": [     {       "Effect": "Allow",       "Action": [         "secretsmanager:GetSecretValue",         "secretsmanager:CreateSecret",         "secretsmanager:ListSecrets",         "secretsmanager:DescribeSecret",         "secretsmanager:TagResource",         "secretsmanager:UpdateSecret"       ],       "Resource": ["*"],       "Condition": {         "ForAllValues:StringEquals": {           "secretsmanager:ResourceTag/AirbyteManaged": "true"         }       }     }   ] }
Then: 1. Create/attach this policy to the IAM user whose Access Key ID / Secret Access Key you’re now putting into Airbyte (for S3 source/destination or S3 Data Lake). [[S3 source setup](https://docs.airbyte.com/integrations/sources/s3#step-1-set-up-s3); [S3 destination setup](https://docs.airbyte.com/integrations/destinations/s3#authentication-option-1-using-an-iam-role-most-secure); [S3 Data Lake](https://docs.airbyte.com/integrations/destinations/s3-data-lake#setup-guide)] 2. Use those credentials in the connector configuration (or in Kubernetes Secrets referenced by 
values.yaml
for Self‑Managed Enterprise). [[Configure K8s secrets](https://docs.airbyte.com/platform/1.6/enterprise-setup/implementation-guide#configure-kubernetes-secrets)] The “malformed token” error you saw on Fargate is an AWS auth issue (IRSA / STS token), not something Airbyte can fix; switching to access key/secret is valid, but you must ensure the IAM user has at least the S3 (and optional Secrets Manager) permissions shown above.
m
@kapa.ai when we run the jobs on managed node group with IRSA the pods for jobs run successfully without any permission error but when we run the job pod on fargate it started throwing the the above given error. So it looks like it has correct permissions.
k
A message from kapa.ai
You’re right that if it works on managed node groups with IRSA, your IAM role and S3 permissions are likely fine. The difference is how credentials are obtained on Fargate vs. your node group, not what the role is allowed to do. However, the knowledge sources you provided do not contain any Fargate‑specific guidance or a documented fix for the 
The provided token is malformed or otherwise invalid
error. They only show: - How to configure IAM policies for S3 and Secrets Manager for Airbyte on AWS generally. [[AWS infra](https://docs.airbyte.com/platform/1.6/deploying-airbyte/infrastructure/aws)] - An open issue where Airbyte pods intermittently lose IRSA permissions and fall back to the node IAM role, causing S3 access errors, but this is on EKS node groups, not Fargate. [[IRSA issue](https://github.com/airbytehq/airbyte/issues/53652)] Because of that, I cannot reliably tell you from these docs exactly why Fargate is producing a malformed token or how to fix it in your environment. What the docs do make clear is: - Airbyte on AWS expects either: - IAM Roles for Service Accounts (IRSA) / instance profile style auth (
authenticationType: instanceProfile
), or - Static credentials (access key / secret) with the S3 and optional Secrets Manager policies attached. [[AWS infra](https://docs.airbyte.com/platform/1.6/deploying-airbyte/infrastructure/aws)] So if you switch to access key / secret key for Fargate: 1. Attach the S3 policy from the docs to the IAM user whose access key/secret you’re using. [[AWS infra](https://docs.airbyte.com/platform/1.6/deploying-airbyte/infrastructure/aws)] 2. If you use AWS Secrets Manager with Airbyte, also attach the Secrets Manager policy from the docs. [[AWS infra](https://docs.airbyte.com/platform/1.6/deploying-airbyte/infrastructure/aws)] Beyond that, the knowledge sources don’t describe: - How to correctly wire IRSA to EKS Fargate for Airbyte, or - Why Fargate specifically would yield a malformed STS token while node groups work. Given these limits, the safest next step is to: - Verify your Fargate profile and pod’s service account are actually using the same IAM role that works on node groups (and that the token projected into the pod is valid), and/or - Continue with static credentials but ensure the IAM user has the S3/Secrets Manager policies shown above. If you can share your Helm 
values.yaml
(with secrets redacted), especially 
global.storage.*
, 
serviceAccount
, and how you’re targeting Fargate, I can try to map it against the documented patterns—but I still won’t be able to go beyond what’s in the linked docs and issues.
m
@kapa.ai this time I had attached the same policies to the user and used its access id and secret access key for access. Now where should I provide these credentials to the replication jobs?
k
A message from kapa.ai
You have two different places where AWS credentials can be used, depending on what you’re trying to authenticate: 1. Airbyte platform state/log storage (S3) 2. Connectors themselves (e.g., S3 source/destination) From the docs, static access key/secret are provided via Kubernetes Secrets and then referenced in 
values.yaml
. --- ### 1. For platform state/log storage (global S3 storage) Create/update the 
airbyte-config-secrets
secret with your IAM user’s keys (note the key names): [[1.6 K8s secret](https://docs.airbyte.com/platform/1.6/enterprise-setup/implementation-guide#creating-a-kubernetes-secret)]
Copy code
yaml apiVersion: v1 kind: Secret metadata:   name: airbyte-config-secrets type: Opaque stringData:   # AWS S3 Secrets   s3-access-key-id: YOUR_ACCESS_KEY_ID   s3-secret-access-key: YOUR_SECRET_ACCESS_KEY
Then in your Helm 
values.yaml
, configure S3 storage to use that secret: [[1.6 storage values](https://docs.airbyte.com/platform/1.6/deploying-airbyte/integrations/storage#values)]
Copy code
yaml global:   storage:     type: "S3"     secretName: airbyte-config-secrets  # Kubernetes secret name     bucket:       log: your-bucket       state: your-bucket       workloadOutput: your-bucket     s3:       region: "us-east-1"       authenticationType: credentials
Airbyte will read 
s3-access-key-id
and 
s3-secret-access-key
from that secret for all platform‑level S3 access. --- ### 2. For connector secret management (if using AWS Secrets Manager) If you also want Airbyte to store connector secrets in AWS Secrets Manager using static credentials, put these keys in the same secret: [[1.6 K8s secret](https://docs.airbyte.com/platform/1.6/enterprise-setup/implementation-guide#creating-a-kubernetes-secret); [1.6 secrets values](https://docs.airbyte.com/platform/1.6/deploying-airbyte/integrations/secrets#values)]
Copy code
yaml stringData:   aws-secret-manager-access-key-id: YOUR_ACCESS_KEY_ID   aws-secret-manager-secret-access-key: YOUR_SECRET_ACCESS_KEY
And configure in `values.yaml`:
Copy code
yaml global:   secretsManager:     type: awsSecretManager     secretName: "airbyte-config-secrets"     awsSecretManager:       region: <aws-region>       authenticationType: credentials
--- ### 3. For individual connectors (e.g., S3 source/destination) The knowledge sources you provided do not include connector‑specific UI fields for S3 credentials in 1.6, so I can’t point to an exact screen. They only describe the platform‑level storage and secret manager configuration above. If your replication jobs are failing on S3 access that is not covered by 
global.storage
or 
global.secretsManager
, the docs here don’t specify where else to inject credentials. So, based on the docs: - Put the IAM user’s access key/secret into the 
airbyte-config-secrets
Kubernetes secret under the documented keys. - Reference that secret in 
global.storage
(and optionally 
global.secretsManager
) in 
values.yaml
. - Redeploy the Helm release so all job pods pick up the new configuration. If you can share whether the failing access is for platform logs/state or for a specific S3 connector, I can map it more precisely to the documented settings above.
m
@kapa.ai lets take this as example where source and destination are both postgres.
k
A message from kapa.ai
For a Postgres → Postgres sync, the replication job itself does not need any AWS credentials. The only AWS credentials it may use are: 1. Platform state/log storage (S3) 2. External connector secret management (AWS Secrets Manager) Both are configured at the platform level, not per‑connection, and are read from Kubernetes Secrets referenced in 
values.yaml
. --- ### 1. If you use S3 for state/log storage Put your IAM user’s access key and secret into the 
airbyte-config-secrets
secret under the S3 keys: [[1.6 storage values](https://docs.airbyte.com/platform/1.6/deploying-airbyte/integrations/storage#values)]
Copy code
yaml apiVersion: v1 kind: Secret metadata:   name: airbyte-config-secrets type: Opaque stringData:   s3-access-key-id: YOUR_ACCESS_KEY_ID   s3-secret-access-key: YOUR_SECRET_ACCESS_KEY
Then configure 
global.storage
in `values.yaml`:
Copy code
yaml global:   storage:     type: "S3"     secretName: airbyte-config-secrets     bucket:       log: your-bucket       state: your-bucket       workloadOutput: your-bucket     s3:       region: "us-east-1"       authenticationType: credentials
All replication job pods (including Postgres→Postgres) will use these credentials only for writing logs/state to S3. [[1.6 storage values](https://docs.airbyte.com/platform/1.6/deploying-airbyte/integrations/storage#values)] --- ### 2. If you use AWS Secrets Manager for connector secrets (optional) Add the same IAM user’s keys under the Secrets Manager keys: [[1.6 K8s secret](https://docs.airbyte.com/platform/1.6/enterprise-setup/implementation-guide#creating-a-kubernetes-secret); [1.6 secrets values](https://docs.airbyte.com/platform/1.6/deploying-airbyte/integrations/secrets#values)]
Copy code
yaml stringData:   aws-secret-manager-access-key-id: YOUR_ACCESS_KEY_ID   aws-secret-manager-secret-access-key: YOUR_SECRET_ACCESS_KEY
And in `values.yaml`:
Copy code
yaml global:   secretsManager:     type: awsSecretManager     secretName: "airbyte-config-secrets"     awsSecretManager:       region: <aws-region>       authenticationType: credentials
This controls where the Postgres connector passwords are stored, but the replication job still doesn’t talk to S3 directly for data. --- So for a Postgres→Postgres example: - DB access: configured in the connection (host, user, password) – no AWS creds needed. - AWS access: only for S3 logs/state and/or Secrets Manager, injected via 
airbyte-config-secrets
and 
values.yaml
as shown above.
m
@kapa.ai How to resolve this when we use IRSA: """ Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The provided token is malformed or otherwise invalid """
k
A message from kapa.ai
The knowledge sources you provided do not explain why IRSA specifically produces: > 
S3Exception: The provided token is malformed or otherwise invalid
nor do they give a documented, step‑by‑step fix for this error on EKS Fargate. So I can’t tell you exactly what is wrong in your cluster; I can only point to what is documented and what has been discussed in related issues. From the docs and GitHub issues, these are the IRSA‑related points that are clear: 1. Airbyte supports using IRSA via 
instanceProfile
auth for S3 storage In 
values.yaml
you configure S3 like this, so Airbyte uses the pod’s IAM role (IRSA) instead of static keys: [[1.6 storage values](https://docs.airbyte.com/platform/1.6/deploying-airbyte/integrations/storage#values)]
Copy code
yaml    global:      storage:        type: "S3"        bucket:          log: airbyte-bucket          state: airbyte-bucket          workloadOutput: airbyte-bucket        s3:          region: "us-east-1"          authenticationType: instanceProfile  # <- IRSA / instance profile
2. The IAM role used by IRSA must have the S3 policy from the docs The role associated with your service account (via IRSA) needs at least the documented S3 permissions (ListBucket, GetBucketLocation, Put/Get/DeleteObject, etc.). [[AWS infra](https://docs.airbyte.com/platform/1.6/deploying-airbyte/infrastructure/aws)] 3. If IRSA is mis‑wired, pods fall back to the node role and S3 fails In one issue, job pods were missing the service account entirely, so the AWS SDK fell back to the node’s IAM role and S3 calls failed. [[Workload pods SA issue](https://github.com/airbytehq/airbyte/issues/50979)] In another, pods intermittently “lost” IRSA and again fell back to the node role, causing S3 403 errors. [[IRSA intermittent issue](https://github.com/airbytehq/airbyte/issues/53652)] 4. Common misconfigurations called out in the issues (but not full root cause): - The IRSA role was missing the assume‑role trust for the 
airbyte-admin
service account. Once that was fixed and 
authenticationType: instanceProfile
was set, S3 worked. [[IRSA intermittent issue](https://github.com/airbytehq/airbyte/issues/53652)] - The 
<http://serviceAccount.annotations.eks.amazonaws.com/role-arn|serviceAccount.annotations.eks.amazonaws.com/role-arn>
must be at the root of 
values.yaml
, not under 
global
. [[IRSA intermittent issue](https://github.com/airbytehq/airbyte/issues/53652)] - Some job pods were using the default service account instead of the IRSA‑annotated one, so they never got the correct role. [[Workload pods SA issue](https://github.com/airbytehq/airbyte/issues/50979)] Because the docs and issues don’t mention the specific “token is malformed” error, I can’t say more than: - Ensure your Helm values follow the documented patterns for IRSA:
Copy code
yaml   global:     serviceAccountName: airbyte-admin     storage:       type: "S3"       bucket:         log: <s3_bucket>         state: <s3_bucket>         workloadOutput: <s3_bucket>       s3:         region: "us-east-1"         authenticationType: instanceProfile   serviceAccount:     annotations:       <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: "arn:aws:iam::<account_id>:role/Airbyte-role"
[[IRSA intermittent issue](https://github.com/airbytehq/airbyte/issues/53652); [1.6 storage values](https://docs.airbyte.com/platform/1.6/deploying-airbyte/integrations/storage#values)] - Confirm that: - The same service account (
airbyte-admin
above) is actually used by the server and job pods (not 
default
). [[Workload pods SA issue](https://github.com/airbytehq/airbyte/issues/50979)] - The IAM role’s trust policy allows that service account to assume it. [[IRSA intermittent issue](https://github.com/airbytehq/airbyte/issues/53652)] - The role has the S3 policy from the AWS infra docs. [[AWS infra](https://docs.airbyte.com/platform/1.6/deploying-airbyte/infrastructure/aws)] If, after verifying those, you still see 
The provided token is malformed or otherwise invalid
, that specific symptom and its resolution are not described in the available Airbyte docs or issues, so I can’t provide a definitive fix based on these sources alone.
m
@kapa.ai I have tried everything, so please tell me another solution that'll work, actually other than IRSA.
k
A message from kapa.ai
You can avoid IRSA entirely by switching Airbyte to use static AWS credentials (access key ID + secret access key) for: - S3 state/log storage, and - (optionally) AWS Secrets Manager for connector secrets. This is documented and supported. --- ### 1. Put AWS credentials in a Kubernetes Secret Create or update the 
airbyte-config-secrets
secret with your IAM user’s keys. [[K8s secret 1.6](https://docs.airbyte.com/platform/1.6/enterprise-setup/implementation-guide#creating-a-kubernetes-secret)]
Copy code
yaml apiVersion: v1 kind: Secret metadata:   name: airbyte-config-secrets type: Opaque stringData:   # S3 credentials   s3-access-key-id: YOUR_ACCESS_KEY_ID   s3-secret-access-key: YOUR_SECRET_ACCESS_KEY    # (optional) AWS Secrets Manager credentials   aws-secret-manager-access-key-id: YOUR_ACCESS_KEY_ID   aws-secret-manager-secret-access-key: YOUR_SECRET_ACCESS_KEY
Apply it:
Copy code
bash kubectl apply -f airbyte-config-secrets.yaml -n <your-namespace>
Make sure the IAM user behind these keys has at least the S3 and (if used) Secrets Manager policies from the docs. [[AWS infra](https://docs.airbyte.com/platform/1.6/deploying-airbyte/infrastructure/aws)] --- ### 2. Configure S3 storage to use credentials (not instanceProfile) In your Helm 
values.yaml
, configure global storage with 
authenticationType: credentials
. [[Storage values 1.6](https://docs.airbyte.com/platform/1.6/deploying-airbyte/integrations/storage#values)]
Copy code
yaml global:   storage:     type: "S3"     secretName: airbyte-config-secrets     bucket:       log: your-bucket       state: your-bucket       workloadOutput: your-bucket     s3:       region: "your-region"        # e.g. us-east-1       authenticationType: credentials
This tells Airbyte to read 
s3-access-key-id
and 
s3-secret-access-key
from the secret instead of using IRSA/instance profile. --- ### 3. (Optional) Use AWS Secrets Manager with credentials If you also want Airbyte to store connector secrets in AWS Secrets Manager using the same IAM user, configure: [[Secrets values 1.6](https://docs.airbyte.com/platform/1.6/deploying-airbyte/integrations/secrets#values)]
Copy code
yaml global:   secretsManager:     type: awsSecretManager     secretName: "airbyte-config-secrets"     awsSecretManager:       region: your-region       authenticationType: credentials
Airbyte will then use 
aws-secret-manager-access-key-id
and 
aws-secret-manager-secret-access-key
from the same secret. --- After updating the secret and 
values.yaml
, upgrade your Helm release so all pods (including job pods on Fargate) use these static credentials instead of IRSA. This path is fully described in the docs and avoids the malformed IRSA token issue by not using IRSA at all.
2 Views
Open in Slack
PreviousNext
Powered by