G’day - we are experiencing a

fatal error: stack overflow

when using atmos > v1.200.0. Are we up against a stack count limit? import limit? we do make use of go templates and functions (!terraform.state). Seemingly all atmos commands (describe component, validate stacks, validate schema, etc.) complete successfully, with the exception of

atmos describe stacks -s <stack>

. Recent change to cloudposse/utils v1.32.0 bumped atmos above v1.200.0, and now all of our remote-state stacks are failing with a generic terraform error

failed to find import

. Pinning utils to

v1.31.0

resolves this, which i believe uses an older atmos version < 1.200.0. Any tips for the best path to resolution? We’d like to use the latest version of atmos. TIA!

@Tyler Rankin there should be no stack limits, and we have optimized Atmos to be 10x faster processing YAML files. What old Atmos version are you on now?

regarding the first issue, please execute

atmos describe stacks -s <stack> --process-templates=false --process-functions=false

This produces the same fatal stack overflow error, seemingly at the same spot looking at the Trace. We are now running

v1.208.0

. The above describe stacks commands fails on anything above 1.200.0

ok (there are some isues in the stacks that affects the Atmos CLI and the utils provider. We'd like to get to the root of it and fix (which we can reproduce using your stacks 🙂 )

atmos describe stacks -s <stack>

- does it throw the error for all stacks, or for a specific stack only?

Thanks for the tips. 1. this happens to all stacks 2. disabling

templates.settings.enabled

still fails the same way 3. we have a lot of

{{ }}

but did a bulk find/replace, hopefully removing all. still fails in the same 4. we are not using Atmos Auth Sorry not much help 🙂

this is exactly the situation why we are working on Atmos AI 🙂 • https://atmos.tools/ai/ • https://atmos.tools/examples/ai • https://github.com/cloudposse/atmos/pull/2171 But it will not work for you (it's in the latest Atmos version, and the

--ai

flag is not relased yet)

interesting - i just make a pretty empty stack file without any imports and it still errored on 1.209.0. Again, switching to 1.200.0 was successful, it returned

{}

. Here’s the file:

settings:
  spacelift:
    autodeploy: true
    space_id: space-dev-id
    labels:
      - tenant:customer
      - environment:gbl
      - stage:dev
    local_preview_enabled: true

vars:
  stage: dev

terraform:
  vars: {}

components: {}

We are seeing this on our spaceift CI runs as well, so i don’t think it is just my machine. Would there be any clues in the atmos config file?

please review this doc https://atmos.tools/changelog/base-path-behavior-change please set

base_path: "."

in

atmos.yaml

and try again let me know if it's working for you. If not, we'll continue debugging

No change when setting

base_path

or not

i just make a pretty empty stack file without any imports and it still errored on 1.209.0. Again, switching to 1.200.0 was successful, it returned

{}

. Here’s the file:

do you have just this one file, or all other stacks are still in the repo when you tested it?

Just the one, but I had the catalog files present. I think there is some issue with templating / imports as you mentioned. I will try to narrow it down by removing files that use these and see how it goes adding them back. We also make use of yaml anchors and abstract component definitions. Lots of import processing to get us to our final values 🥲

if you find the file(s) that is causing the issue, please share with me (yes, the best way to debug is to remove and then add all YAML manifests. To speed it up exponentially, you can use

The half-split troubleshooting technique isolatess faults by systematically dividing a system, process, or code in half to locate the issue. By testing at the midpoint, you determine if the issue is in the first or second half, eliminating 50% of possibilities with each check. This methodical approach continues, cutting the remaining faulty segment in half until the problem is identified.

Might be on to something. We have a number of abstract components where we set defaults that get inherited elsewhere. It seems like we are setting the component within

metadata

on the abstract definitions - is that necessary? For instance we have:

components:
  terraform:
    iam-delegated-roles-defaults:
      metadata:
        component: iam-delegated-roles
        type: abstract

    iam-delegated-roles:
      metadata:
        component: iam-delegated-roles
        type: real
        inherits:
          - iam-delegated-roles-defaults

When i remove

metadata.component

from

iam-delegated-roles-defaults

the

atmos describe stacks

command completes!!

remove

metadata.component

from

iam-delegated-roles-defaults

the

atmos describe stacks

command completes

is it working with the latest atmos version? if it is, then we'll review the code - it should not panic when you have

metadata.component

on

type: abstract

thanks for reporting

The stack overflow when running describe stacks seems to be resolved. Not sure if it contributed to the issue but some of the inherited components use !terraform.state in their definitions. However, terraform plan still fails on latest utils. It just gives that generic import error. I can see the cloned monorepo in the path we expect, and if I do

terraform console

I am even able to see the correct values for module.account_map. This error is still an atmos query not working correctly?

The stack overflow when running describe stacks seems to be resolved. Not sure if it contributed to the issue but some of the inherited components use !terraform.state in their definitions.

what did you do to solve this issue? did you just remove

component: iam-delegated-roles

from the abstract components, e.g.

iam-delegated-roles-defaults:
      metadata:
        component: iam-delegated-roles
        type: abstract

or someting else?

That’s correct. i just removed

metadata.component

from the abstract components. I had to do this on 2

iam-delegated-roles

, and

eks

. I think there are some dependencies between the two, but once they are both removed the atmos commands complete. I can probably share these files in DM

we can review and fix that, thanks for finding.

metadata.component

on abstract components does not do anything (since the abstract components can't be applied), but it should not fail as well I'll let you know if we need your files

Yes - confirming. Atmos v1.209.0, utils 1.33.0

@Andriy Knysh (Cloud Posse)

metadata.component

on abstract components does not do anything (since the abstract components can't be applied), b

But now with metadata inheritance it should "do something": populate the default component

Here is a snippet from utils v1.31.0, which plans successfully. I need to compare this on a failed run but i am making note of the EOF right before the utils provider exits

thanks @Tyler Rankin

Do you expect that will correct the

failed to find import

error here as well? I’m not sure i’m headed in the right path, and feel this might be devolving from

atmos

but i do notice there are a couple differences on the terraform packages being installed, some with recent breaking changes: •

cloudposse/utils v1.31.0

◦ terraform-plugin-go@v0.27.0 ◦ terraform-plugin-sdk@2v2.37.0 •

cloudposse/utils v1.33.0

◦ terraform-plugin-go@v0.31.0 ◦ terraform-plugin-sdk@2v2.40.0 https://github.com/hashicorp/terraform-plugin-go/releases https://github.com/hashicorp/terraform-plugin-sdk/releases

failed to find import

- I'll get to it next (it's prob not related to the fix for

fatal error: stack overflow

at all)

Let me know if there’s anything else I can provide. We are either pinning a lot of versions.tf files or finding a better way to manage these dependencies so that we aren’t affected by changes like this in the future. And I want to be sure it isn’t a config we are missing on our end.

@Tyler Rankin the fix for

fatal error: stack overflow

is in the new release - please try it out https://github.com/cloudposse/atmos/releases/tag/v1.210.0 https://github.com/cloudposse/atmos/pull/2214

I continue to get a stack overflow on 1.210.0, but we can move past that i will just remove all

metadata.component

references from our abstract definitions

@Andriy Knysh (Cloud Posse) we should still try to get to the bottom of this

@Tyler Rankin w/o your stacks, I'm not sure we can find the issue - we have a lot of tests that exercise all of that including

metadata.component

in abstract components help us to understand the issue

Can we focus on the failed import error from the remote-state / utils lookup?

do you see the issue on CI, or locally as well?

CI shows the same. I’m trying to pass in skip-functions functionality (we forked https://github.com/cloudposse/github-action-atmos-affected-trigger-spacelift/tree/main)

we'll release Atmos, then a new utils provider (it uses the Atmos code)

@Andriy Knysh (Cloud Posse) Something wrong with the

cloudposse/utils

v2.2.0 checksums I'm getting this error during

terraform init

Error while installing cloudposse/utils v2.2.0: checksum list has unexpected SHA-256 hash 1f75418c5a89d8f6fd95b9dd58903574cf8b2c58fdd9dddf21a019d2d7da34d (expected c4f8282b0664d27f4c475a800c244a87fd844e73b67aeb737d8479f84fe8b155)

@Igor Rodionov can you please take a look at this?

let me cut new release

please cut a new v2 release, and 1.210.1 as well

@Andriy Knysh (Cloud Posse) may I have an approval https://github.com/cloudposse/terraform-provider-utils/pull/534 ?

approved

I upgraded

remote-state

to v2, but i still receive failed to find import on terraform plan (with utils using v2.3.0), when setting

ATMOS_BASE_PATH

to relative path from CWD. It succeeds with absolute path

We're still working on publishing the updated provider.

@Tyler Rankin apologies for all the issues, let's get to the bottom of it and fix it Can I ask you to run it again using the latest Atmos 1.210.1 and the latest utils provider 2.3.0, and report the issues, show the error

I want to be sure i’m understanding this correctly - my version of atmos almost doesn’t’ matter right? We run atmos in spacelift to generate the tf vars file, and then spacelift just run

terraform plan <vars-file>

. The remote-state module calls the utils provider which ultimately defines the version of atmos. Assuming I have that correct - my terraform plan still fails.

module.yaml_config.module.maps_deepmerge.data.utils_deep_merge_yaml.all_map_configs: Reading...
module.yaml_config.module.maps_deepmerge.data.utils_deep_merge_yaml.all_map_configs: Read complete after 0s [id=02338e6bebdc5adb05d28a654095b7397a2cad9e]

Planning failed. Terraform encountered an error while generating this plan.
╷
│ Error: failed to find import
│ 
│   with module.account_map.data.utils_component_config.config[0],
│   on .terraform/modules/account_map/modules/remote-state/main.tf line 1, in data "utils_component_config" "config":
│    1: data "utils_component_config" "config" {
│ 
╵
╷
│ Error: failed to find import
│ 
│   with module.iam_roles.module.account_map.data.utils_component_config.config[0],
│   on .terraform/modules/iam_roles.account_map/modules/remote-state/main.tf line 1, in data "utils_component_config" "config":
│    1: data "utils_component_config" "config" {
│ 
╵
Releasing state lock. This may take a few moments...
 ⧉  BrightAI 
 √ . [bai-mgmt-gbl-identity-admin] (HOST) terraform ⨠ echo $ATMOS_BASE_PATH
.terraform/modules/monorepo

utils provider 2.3.0?

yes

√ . [bai-mgmt-gbl-identity-admin] (HOST) terraform ⨠ terraform version
Terraform v1.5.7
on linux_arm64
+ provider <http://registry.terraform.io/cloudposse/template|registry.terraform.io/cloudposse/template> v2.2.0
+ provider <http://registry.terraform.io/cloudposse/utils|registry.terraform.io/cloudposse/utils> v2.3.0
+ provider <http://registry.terraform.io/hashicorp/aws|registry.terraform.io/hashicorp/aws> v6.37.0
+ provider <http://registry.terraform.io/hashicorp/external|registry.terraform.io/hashicorp/external> v2.3.5
+ provider <http://registry.terraform.io/hashicorp/http|registry.terraform.io/hashicorp/http> v3.5.0
+ provider <http://registry.terraform.io/hashicorp/local|registry.terraform.io/hashicorp/local> v2.7.0
+ provider <http://registry.terraform.io/hashicorp/time|registry.terraform.io/hashicorp/time> v0.13.1

Your version of Terraform is out of date! The latest version
is 1.14.7. You can update by downloading from <https://www.terraform.io/downloads.html>

ok, thanks again, let me get to it

@Andriy Knysh (Cloud Posse) are you able to repro or should we consider switching to absolute paths in our environment?

we fixed it in a PR and merged to main. we'll cut a new Atmos release next week, then update the provider to use it

excellent! Thank you!

Here is the postmortem about the issue

Error while installing cloudposse/utils v2.2.0: checksum list has unexpected SHA-256 hash 1f75418c5a89d8f6fd95b9dd58903574cf8b2c58fdd9dddf21a019d2d7da34d (expected c4f8282b0664d27f4c475a800c244a87fd844e73b67aeb737d8479f84fe8b155)

https://github.com/cloudposse/terraform-provider-utils/issues/536

Nice. When I get my paycheck?

Appreciate everyone helping surface issues here—it really helps us move quickly 🙏

@Andriy Knysh (Cloud Posse) we still get failed to find import on the new utils v1.34.0. i think at this point, if there are no ill effects, we will migrate to absolutely path

@Tyler Rankin we have not released a new provider version yet, just a new Atmos release https://github.com/cloudposse/atmos/releases/tag/v1.211.0 (30 mins ago) we'll release new provider tomorrow morning

ah, got it. apologies! I saw the 1.34 release an hour ago and thought that might’ve held the key.

@Tyler Rankin one more attempt 🙂 We have released: • https://github.com/cloudposse/terraform-provider-utils/releases/tag/v2.5.0 • https://github.com/cloudposse/atmos/releases/tag/v1.211.0 Can I ask you to test it (before we release a new version v1 of the utils provider)? You mentioned before that you were able to test utils provider v2 Thank you

v2.5.0 looks good!

- Installing cloudposse/utils v2.5.0...
- Installed cloudposse/utils v2.5.0 (self-signed, key ID 7B22D099488F3D11)

I kept my relative path set in

ATMOS_BASE_PATH

during the spacelift run and the plan completed successfully

finally, we did it 🙂

@Andriy Knysh (Cloud Posse) @Tyler Rankin we cut 1.35.0 release for utils provider

Awesome, thank you all! I just ran a successful plan on v1 🎉

Excellent! 😌