https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> Since upgrading to v2.2 we are rece...
# replication-ask-ai
c
@kapa.ai Since upgrading to v2.2 we are receiving below error on deploying terraform sources and destinations. What permissions are missing? Deployment is EKS based with Terraform provider version 1.3 │ unknown status code returned: Status 500 │ {"status":500,"type":"reference.airbyte.com/reference/errors","title":"unexpected-problem","detail":"An unexpected problem has occurred. If this is │ an error that needs to be addressed, please submit a pull request or github issue.","documentationUrl":null,"data":{"message":"Failed to read secret │ airbyte_workspace_2fc21f58-9ce6-448e-a465-6807de524c49_secret_a30d113c-5f0e-47a9-a857-ddd569e71513_v10: The security token included in the request is │ invalid."}}
k
Thinking completed Based on the error message (
Failed to read secret ... The security token included in the request is invalid.
), this error comes directly from AWS Secrets Manager when Airbyte attempts to read a stored connector secret. The underlying cause typically relates to the IAM authentication mechanism or specific Secrets Manager permissions assigned to Airbyte: 1. IRSA / Service Account IAM Role Authentication Issue The error 
"The security token included in the request is invalid"
commonly indicates that temporary AWS STS credentials have expired, cannot be assumed, or the Kubernetes Service Account using IAM Roles for Service Accounts (IRSA) has lost or failed to provide its AWS credentials [Airbyte Github Issue #53652; Airbyte Github Issue #48809]. • Verify Service Account Annotations: Ensure your Kubernetes Service Account used by Airbyte has the correct IRSA role ARN annotationeks.amazonaws.com/role arn "arnawsiam::account_id:role/Airbyte-role" • Verify Trust Relationships: Ensure the AWS IAM role's trust relationship allows your EKS cluster's OIDC identity provider and the specific Kubernetes service account (
airbyte
or 
airbyte-admin
) to assume the role [AWS Secrets Troubleshooting; Airbyte Github Issue #49938]. • Pod Identity vs IRSA: Note that Java-based components in Airbyte do not support EKS Pod Identity file-based tokens (
AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE
), requiring IRSA (OIDC) instead [Airbyte Github Issue #78116]. 2. Required AWS Secrets Manager IAM Permissions To manage and read secrets in AWS Secrets Manager, the IAM Role associated with your Airbyte deployment requires the following permissions [AWS Secrets Policy; External Secrets Policy]:
Copy code
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:CreateSecret",
        "secretsmanager:ListSecrets",
        "secretsmanager:DescribeSecret",
        "secretsmanager:TagResource",
        "secretsmanager:UpdateSecret"
      ],
      "Resource": [
        "*"
      ],
      "Condition": {
        "ForAllValues:StringEquals": {
          "secretsmanager:ResourceTag/AirbyteManaged": "true"
        }
      }
    }
  ]
}
Note on conditions and tags: If your IAM policy restricts actions by tags (such as 
secretsmanager:ResourceTag/AirbyteManaged
), make sure your secrets actually match the required tags. Restricting actions strictly by tags can sometimes lead to unexpected access failures during secret lookups and version transitions [Airbyte Github Issue #55670]. 3. KMS Decryption Permissions (If Applicable) If your AWS Secrets Manager secrets are encrypted using a customer-managed AWS KMS key, your IAM role must also have permission to decrypt using that KMS key (
kms:Decrypt
) [Secrets Manager Values]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
c
@kapa.ai Here is the full error from server pod
Copy code
2026-09-22 07:26:59,920 [public-api-executor-thread-2]   ERROR   i.a.c.s.e.p.AbstractThrowableProblemHandler(handle):31 - Throwable Problem Handler caught exception:
[pod/airbyte-server-5f95795886-6mjqc/airbyte-server-container] io.airbyte.api.problems.throwable.generated.UnexpectedProblem: Detail: class UnexpectedProblemResponse {
[pod/airbyte-server-5f95795886-6mjqc/airbyte-server-container]     status: 500
[pod/airbyte-server-5f95795886-6mjqc/airbyte-server-container]     type: <https://reference.airbyte.com/reference/errors>
[pod/airbyte-server-5f95795886-6mjqc/airbyte-server-container]     title: unexpected-problem
[pod/airbyte-server-5f95795886-6mjqc/airbyte-server-container]     detail: An unexpected problem has occurred. If this is an error that needs to be addressed, please submit a pull request or github issue.
[pod/airbyte-server-5f95795886-6mjqc/airbyte-server-container]     documentationUrl: null
[pod/airbyte-server-5f95795886-6mjqc/airbyte-server-container]     data: class ProblemMessageData {
[pod/airbyte-server-5f95795886-6mjqc/airbyte-server-container]         message: Failed to read secret airbyte_workspace_2fc21f58-9ce6-448e-a465-6807de524c49_secret_a30d113c-5f0e-47a9-a857-ddd569e71513_v10: The security token included in the request is invalid.
[pod/airbyte-server-5f95795886-6mjqc/airbyte-server-container]     }
[pod/airbyte-server-5f95795886-6mjqc/airbyte-server-container] }.detail, Data: class ProblemMessageData {
[pod/airbyte-server-5f95795886-6mjqc/airbyte-server-container]     message: Failed to read secret airbyte_workspace_2fc21f58-9ce6-448e-a465-6807de524c49_secret_a30d113c-5f0e-47a9-a857-ddd569e71513_v10: The security token included in the request is invalid.
[pod/airbyte-server-5f95795886-6mjqc/airbyte-server-container] }
k
Thinking completed The error 
The security token included in the request is invalid
is an AWS STS / AWS SDK error indicating that when the 
airbyte-server
pod attempts to call AWS Secrets Manager to read the secret, the AWS credentials or session token it is presenting are invalid, missing, or improperly assumed [Airbyte Github Issue #48809; Airbyte Github Issue #67588]. Based on the knowledge sources, check the following configuration points to resolve the issue: 1. Required IAM Policy Permissions for AWS Secrets Manager Airbyte requires permissions to manage and read secrets in AWS Secrets Manager [AWS Infrastructure; External Secrets]:
Copy code
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:CreateSecret",
        "secretsmanager:ListSecrets",
        "secretsmanager:DescribeSecret",
        "secretsmanager:TagResource",
        "secretsmanager:UpdateSecret"
      ],
      "Resource": [
        "*"
      ],
      "Condition": {
        "ForAllValues:StringEquals": {
          "secretsmanager:ResourceTag/AirbyteManaged": "true"
        }
      }
    }
  ]
}
Note on tag conditions: If the IAM policy enforces 
secretsmanager:ResourceTag/AirbyteManaged: true
, verify whether the secret being accessed actually has that tag [Troubleshooting Secrets]. 2. IRSA / Service Account IAM Role Authentication Because the deployment is running on EKS, this error typically occurs when the pod fails to assume its IAM role via IRSA (IAM Roles for Service Accounts): • Service Account Annotation: Confirm that the Kubernetes Service Account used by 
airbyte-server
has the IRSA annotation attached at the root level [Airbyte Github Issue #53652]serviceAccount annotations: eks.amazonaws.com/role-arn: "arnawsiam::account_id:role/Airbyte-role" • Trust Relationship: Ensure the IAM role trust policy is configured with your EKS cluster's OIDC provider URL and matches the exact Service Account name and namespace used by the server [Troubleshooting Secrets]. • Pod Session Expiration: In some Helm/EKS deployments, server pods can intermittently lose IRSA authorization or fail to refresh the injected token; restarting/deleting the server pod can temporarily restore the session if the token environment variables were out of sync [Airbyte Github Issue #53652]. 3. Static Credentials (If Not Using IRSA) If your deployment uses static credentials (Access Key ID and Secret Access Key) rather than IRSA, an invalid token error means a temporary session token is missing, expired, or the credentials in the Kubernetes secret are invalid or improperly mapped [Airbyte Github Issue #67588]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
c
Resolution Approach 1: Remove 
secretName
from 
secretsManager
(Recommended) If you are authenticating using IRSA / 
instanceProfile
, you do not need to specify a custom 
secretName
for secret manager credentials. In your `values.yaml`: global: secretsManager: enabled: true type: AWS_SECRET_MANAGER # ❌ Remove: secretName: "airbyte-config-secrets" awsSecretManager: region: <your-aws-region> authenticationType: instanceProfile tags: - key: AirbyteManaged value: "true" • Why this works: When 
secretName
is omitted under 
secretsManager
, the Helm chart defaults to referencing the chart's internal secret (
<release>-airbyte-secrets
), which passes empty strings (
""
) for the access keys. Airbyte sees empty credentials and falls back to the IRSA role on the ServiceAccount. Approach 2: Ensure dummy keys are empty strings 
""
 If your deployment pipeline requires specifying 
global.secretsManager.secretName
pointing to a custom Kubernetes secret: Ensure the keys inside that Kubernetes Secret are set to empty strings, not placeholder strings like `"dummy"`: apiVersion: v1 kind: Secret metadata: name: airbyte-config-secrets stringData: # Must be empty strings so Airbyte falls back to IRSA AWS_SECRET_MANAGER_ACCESS_KEY_ID: "" AWS_SECRET_MANAGER_SECRET_ACCESS_KEY: "" • Verification Checklist for IRSA 1. Pod has IAM token mounted: Check that 
AWS_ROLE_ARN
and 
AWS_WEB_IDENTITY_TOKEN_FILE
exist in the pod environment. 2. Access key env vars are empty: Ensure 
AWS_SECRET_MANAGER_ACCESS_KEY_ID
and 
AWS_SECRET_MANAGER_SECRET_ACCESS_KEY
are empty in the container. 3. IAM Trust Policy: Confirm the IAM Role trust policy includes 
sts:AssumeRoleWithWebIdentity
matching the Airbyte ServiceAccount (
system:serviceaccount:<namespace>:<service-account-name>
). 4. IAM Permissions: Ensure the IAM Role has 
secretsmanager:GetSecretValue
, 
secretsmanager:DescribeSecret
, and 
secretsmanager:CreateSecret
permissions (including KMS decryption if using a CMK).
Open in Slack
PreviousNext
Powered by