Title
m
Manav Sikka
04/13/2023, 4:01 PMHi, I am in process of migrating Atlantis to the new K8s version 1.24, my state file resides on S3, and I am using an IAM role to access that. But with the new deployment on K8s 1.24, I am getting:
Initializing the backend...
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
Error refreshing state: AccessDenied: Access Denied
status code: 403
I have checked the IAM role, within the same pod, I am able to access the state file from S3, using AWS CLI commands, I can perform write operations on the S3 bucket. So the IAM role is perfectly fine, so is the role binding with the service account, not sure why TF isn't able to access it.
d
Dylan Page
04/13/2023, 4:15 PMHow are you specifying the IAM role, is it configured at the instance level or within your TF?
j
Jason Reslock
04/13/2023, 4:31 PMDoes the backend configuration use kms? Does the role have permission to use the kms key for crypto operations on the state file(s)?
💯 1
m
Manav Sikka
04/13/2023, 10:29 PMThe IAM role is configured with IRSA on the pod
the bucket is encrypted using Amazon S3 managed keys, and the same IAM role is working on the old atlantis deployment i have
d
Dylan Page
04/14/2023, 2:06 AMI would recommend checking your terraform to make sure there isn't anything override the AWS provider configuration
Sine you mentioned exec'ing into the pod and being able to access the state file from S3, thus confirming the IAM role from IRSA service account is working on the pod
m
Manav Sikka
04/19/2023, 1:32 PMafter trying a lot of things, upgrading the TF version made it work, we were running an old version 0.12.27, and somehow it wasn't picking the right IAM role, running TF plan within the pod from version 0.12 and 0.13, produced different results, 0.13 picked the right IAM role