https://cloudposse.com logo
Join Slack
Powered by
We make heavy use of `remote-state`, generally v1....
# terraform
t
We make heavy use of 
remote-state
, generally v1.8.0. Recently we've encountered an issue with our 
eks
component. We believe there is a piece of this on Spacelift’s end that is causing the issue but was curious if anyone else using remote-state might've observed this behavior. All Spacelift UI outputs for each of our eks stacks (>20) recently started to display 
sandbox
values. We don't have any corresponding runs that show these being set. The stacks are failing to apply due to a lineage mismatch. Throughout the planning phase we have multiple remote-state calls reading cross account values and we observe the workspace changing by checking 
terraform workspace show
. I assume this might be normal for the remote-state calls to actually lookup the state values. It seems like the last remote-state call might be to 
sandbox
, and Spacelift isn't reading 
.terraform/enviroment
a final time after it switches back to workspace of the actual stack we are planning. This causes Spacelift to pull the sandbox state and attempt an apply which fails. While Spacelift UI outputs are incorrect, the state file for each of our eks components are intact and don't have erroneous sandbox values. All that said our local atmos applies succeed. So I guess the question to the group is has anyone seen an issue with 
remote-state
ever targeting the incorrect workspace at the end of a plan/apply?
e
@Andriy Knysh (Cloud Posse) anything related to the recent utils provider update?
a
remote-state v1.8.0 pinned the utils provider to 
< 2
, the new 
utils
provider major version v2 should not affect it
@Tyler Rankin could you please check from the Spacelift run which versions of the 
utils
provider is it downloading? Also, when you run locally, what version of the 
utils
provider it's using?
since it's ok locally, I don't know yet what the issue might be in Spacelift, i'll ak you more questions to try to figure it out
t
Thanks Andriy! both the success and failed runs are using 
v1.31.0
of 
utils
. We believe we narrowed in on the culprit. What version of remote-state does 
atmos.Component
use? We found the commit where these runs start to fail and we are using the atmos.Component function to get our eks cluster arn into an iam-delegated-roles configuration:
Copy code
eks_cluster_arn: '{{ (atmos.Component "eks" (printf "core-uw2-prod") ).outputs.eks_cluster_arn }}'
When we hard code the cluster arn above the issue resolves. And we noticed during the failed runs that the workspace switches between all those that we have used with the atmos.Compoent lookup
a
the current version of 
remote-state
module and 
utils
provider should support 
atmos.Component
template function, but it looks like there are some issues with that - we'll review
instead of 
atmos.Component
template function, we recommend using the 
!terraform.state
YAML function - it loads the state directly from the bucket and is much faster then 
atmos.Component
(which calls the terraform binary) https://atmos.tools/functions/yaml/terraform.state
please try it
We found the commit where these runs start to fail
what commit is that?
t
I meant to say that we recently introduced 
atmos.Component
to our iam-delegated-roles component, and it wasn’t apparent that was causing this issue in 
eks
. So we found the commit on our end 🙂 We are going to look at trying 
!terraform.state
🙌
👍 1
🙌 1
e
Yes, this should avoid the kind of issue you ran into.
2 Views
Open in Slack
PreviousNext
Powered by