really enjoying DAG, seems to be working decently ...
# atmos
z
really enjoying DAG, seems to be working decently with a PoC for targeted stack precedence
πŸŽ‰ 1
really just a ton of bash logic specifically fit for this new repo layout I'm putting together. lot of yq filtering
a
what can we add to atmos to eliminate bash and yq?
z
more terraform command options that aren't specifically a stack or every single stack
thread above has some more context
e
more terraform command options that aren't specifically a stack or every single stack
Tags & labels, no?
z
Didn't we determine that was only for components..?
my problem is I need to target a set of stacks, and I can't render my entire atmos monorepo at once
atmos terraform apply --affected --tags production
hmmmmm
e
So I think there are 2 things: We're introducing an
--error-mode
which handles graceful degradation. For example, it would be able to run
atmos terraform apply --affected --tags production
and it will only error if one of the targeted components cannot resolve the values. But if say some value cannot be resolved in dev, it doesn't care.
the mode can be strict, and error, or lenient and only warn.
z
^ will this handle the
!terraform.state
not provisioned thing also?
e
yes
To some degree
z
because this really did a number on me
Copy code
'{{ atmos.Resolve (printf `!terraform.state cloudposse-kms-key hello-world-%s ".key_arn // ""default"""` (regexReplaceAll "[0-9]+$" .vars.deployment_requirements.stage "x")) }}'
e
It will still be a problem if you truly depend on a value; there's no magic there.
But if that value is not needed for the operation, then it will not not block it.
Copy code
'{{ atmos.Resolve (printf `!terraform.state cloudposse-kms-key hello-world-%s ".key_arn // ""default"""` (regexReplaceAll "[0-9]+$" .vars.deployment_requirements.stage "x")) }}'
Yea, I'm an expert at atmos and I can't read that
Generally we have no sich complicated expressions
z
that's one of the worse ones. But yeah. Introduced when implementing DAG ( I think..? Also greenfield deployment...)
e
I would recommend an alternative but I can't understand (intuit the intent) that expression ; )
z
but yeah, basically introduced to the team the concept of layers of stacks
account layers, logical dev layer, blue/green dev layer
anyway, I need to try out the tags thing after the error-mode thing is introduced, that might make life easier
a
I asked Claude to read that expression, it took about 30 seconds πŸ™‚
Copy code
Here's what that expression does, step by step (I verified the semantics against internal/exec/template_funcs.go:61, internal/exec/yaml_func_terraform_state.go, and the new argument parser in pkg/function/parser/parser.go):

  1. It's a Go template expression, evaluated while Atmos renders templates during stack processing β€” i.e., before YAML functions are normally processed, with the current component/stack as context.
  2. Innermost step β€” regexReplaceAll "[0-9]+$" .vars.deployment_requirements.stage "x": takes the stage name from vars.deployment_requirements.stage and replaces a trailing run of digits with the literal letter x. For example, prod-3 β†’ prod-x, dev12 β†’ devx; a stage with no trailing digits (e.g. dev) passes through unchanged. So all numbered instances of a stage collapse to one -x name.
  3. printf builds a YAML-function call as a string, injecting that transformed stage into the format hello-world-%s. The result is:
  !terraform.state cloudposse-kms-key hello-world-prod-x ".key_arn // ""default""" (using the prod-3 example).
  4. The doubled quotes are the escaping convention for quoted YAML-function arguments (a "" inside a "…"-quoted argument is a literal "). After unquoting, the third argument is the YQ expression .key_arn // "default". The final """ is an escaped quote plus the closing quote.
  5. atmos.Resolve evaluates that string as the !terraform.state YAML function at template-render time. This is the reason for the whole construction: a bare !terraform.state YAML tag owns the entire scalar and can't have a templated stack name spliced into it, whereas atmos.Resolve lets you compose the function call from template data.
  6. !terraform.state reads the Terraform state directly from the configured backend (no terraform init/terraform output subprocess) for component cloudposse-kms-key in stack hello-world-<stage-with-x>, then applies the YQ expression to the state outputs.
  7. The YQ // operator provides a fallback: it returns the key_arn output, or the string "default" if key_arn is null or missing.
  8. The fallback also covers the never-provisioned case: because the expression contains //, Atmos treats "state not provisioned" and "output not found" as recoverable β€” it evaluates the YQ default against an empty document and returns "default" instead of failing the whole stack processing.

  Net effect: "Give me the KMS key ARN from the cloudposse-kms-key component in the hello-world-<stage> stack β€” where any numeric stage suffix is normalized to x (so e.g. prod-1, prod-2, prod-3 all share hello-world-prod-x's key) β€” and if that component isn't provisioned or has no key_arn output, use the string default instead."

  One caveat worth knowing: the digit-run replacement maps prod-123 to prod-x (the entire run of digits is replaced by a single x), and the literal string "default" is a placeholder value, not a real ARN β€” so downstream consumers presumably treat default as a sentinel (e.g., "use the AWS-managed key").
z
Yeah just build that into atmos now 😁
e
What would also be helpful is if you could share the other kinds of expressions you use that are difficult to comprehend.
z
Tried to simplify it with codex, couldn't. Explanation:
Copy code
distinction between direct Terraform state references and state references
  used inside generated JSON.

  A direct value like this works as expected:

  kms_key_id: !terraform.state cloudposse-kms-key ...

  That value is consumed directly as a Terraform input, so Atmos resolves it during stack
  processing.

  The external-secrets KMS ARN follows a different path:

  Terraform variable
  β†’ IAM policy statement
  β†’ JSON serialization with toJson
  β†’ aws_iam_policy

  Because the value is embedded inside a templated policy structure, the !terraform.state
  expression can survive into the later template/JSON-rendering phase instead of being fully
  evaluated first. Terraform then receives malformed JSON and reports errors such as:

  invalid character '.' after array element

  Wrapping the expression with atmos.Resolve forces Atmos to resolve the remote state value before
  the policy is serialized:

  external_secrets_kms_key_arn: '{{ atmos.Resolve (printf `!terraform.state ...`) }}'

  That produces a concrete ARN before toJson builds the IAM policy.

  General rule: direct !terraform.state references are appropriate for ordinary Terraform
  variables. When a state-derived value is interpolated into a generated JSON document, policy, or
  another nested template, resolve it explicitly first with atmos.Resolve.
e
What if we supported:
Copy code
foo: !terraform.state {
  component: cloudposse-kms-key,
  stack: 'hello-world-{{ regexReplaceAll "[0-9]+$" .vars.deployment_requirements.stage "x" }}',
  output: key_arn,
  default: default,
}
Today, all !functions are scalars (strings)
This would make it work with named parameters
z
If that would work when feeding it into a
| toJson
or similar. I don't totally hate the current implementation, it's probably just how I'm feeding and templating everything
e
why feed into
| toJson
?
(the idea behind using
!anythinng
is it's not templating; it's actually YAML. so it doesn't need manipulation)
z
Think we're appending yaml lists and feeding them into inline iam policies into a module that take json
Can double check tomorrow for the exact usecase
@Erik Osterman (Cloud Posse) confirmed that
--labels
won't work for our use case, had the AI try to replicate what I did recently with our (new) layered stacks pattern
Copy code
atmos terraform plan --all \
    --labels=poc=atmos-lifecycle,account=$ACCOUNT,app=hello-world,environment=dev,layer=slot,slot=1 \
    --dry-run --max-concurrency=1

   Error

   Error: failed to execute describe stacks: failed to describe component  uds-eks  in stack  $STACK-stg  in YAML function:
   !terraform.state uds-eks $STACK-stg ".eks_properties.cluster_name // ""default"""  invalid component: Could not find the component  uds-
   eks  in the stack  $STACK-stg . Check that all the context variables are correctly defined in the stack manifests. Are the component
   and stack names correct? Did you forget an import?
this is it bombing out, trying to hit our staging environment. Tries to do some kind of evaluation before filtering through the stacks that it would be applicable to.
Not sure if there should be an earlier stage of filtering or what. Also kind of rough I need to attach these labels to every single component's metadata, there's not a top level
terraform.metadata
thing I can apply to all components in the stack 🀷. I ended up creating an abstract component and importing it and inheriting to each component to add the metadata, dry-ish
I guess the only other option would be to completely flatten our stacks by account, and chunk everything up by labels. that would be a huge undertaking
also probably make debugging a lot harder
ai answer:
Copy code
In a monorepo containing multiple accounts and independently valid stack families, --all --labels=… should filter candidate
component instances before resolving all stack templates/functions/state. Today it eagerly evaluates unrelated stacks, making label-based cross-stack execution unusable for account-scoped operations.
I think overall if there was like a way to solve for that, and also have like some kind of stack level metadata - like there's a
name
now, it might come closer to what I'm picturing for labels/tags to work. Also if there was more logic to be able to do AND/OR with both labels/tags
e
Could we maybe rewind a bit? πŸ˜ƒ
Best case, there's an elegant/different way of modeling your infrastructure to accomplish this
You definitely have pushed the current approach to the limit
Or maybe there are missing primitives in atmos.
Copy code
atmos terraform plan --all \
    --labels=poc=atmos-lifecycle,account=$ACCOUNT,app=hello-world,environment=dev,layer=slot,slot=1 \
    --dry-run --max-concurrency=1
I feel like we've traded complex yaml expressoins for complex command line fu
Both to me are a red flag
😁 1
@Zack when you get a chance, can you try it with this version?
--use-version=2807
z
Will this allow me to do an apply --all to target a specific stack and apply all stacks that are a dependency of that stack πŸ‘€
I think the labels could work for us and replace a lot of bash logic. Not a huge leap from where we are now directory structure wise
Will there be a way to list stacks and components matching the filters? Probably list stacks?
List stacks was another one that gave us that error, had to skip templating and the other thing
e
This fixes the problem where it eagerly evaluates functions that are associated with components not matching the filters; only evaluate functions for selected components
Will there be a way to list stacks and components matching the filters? Probably list stacks?
Adding that. Good point.
Anywhere you can filter by
--stack
you should also be able to filter by
--labels
and
--tags
So for
atmos describe affected
, we have:
Copy code
--include-dependents                 Include the dependent components and stacks`
and apply all stacks that are a dependency of that stack
If we added that flag to
atmos terraform apply
would that meet your needs?
z
Meaning the things that the stack depends on?
Wording kind of odd. To me it sounds like that would meant you'd be applying the parent specifically, and then it would fan out and do all of the dependants
I'm thinking you're targeting a thing, and it would apply all dependencies, and then the thing you're targeting
That's what we're trying to accomplish with bash currently
TBH both directions sound pretty awesome, two separate flags
e
Include dependents, combined with labels, would mean anything anything that the labels select, and anything that depends on (dependents) those things selected
It sounds like you need all dependencies (think pre reqs) of those selected components.
z
Yeah I think so πŸ€” again putting thing in the scope of different layers of stacks, dev and test rely on the same parent stack, dev1 and dev2 rely on dev, test1 test2 rely on test etc
--include-dependencies?
This is sounding good and won't make my labeling strategy insane
And if it could have another flag for level of dependencies, by N levels that would be sick
describe component provenance has a similar functionality, levels of imports, i mean
Honestly either way would work. Include dependents would allow us to target a stack from our deployment pipeline, and filtering by tags would give us the specific dependents
Sorry rambling but yeah something atmos native would 100% be better than the Metadata and bash wrapper I built
e
Yes, if you ever need to wrap atmos with scripts then there's something missing
z
hey - logging in - wanted to kick the tires on this
e
Will add the
--include-dependents=depth
(default unlimited) and
--include-dependencies=depth
for all commands that support labels and tags. Hit rate limits though, so won't get back to it until this afternoon.
πŸ‘€ 1
If there's any nasty shell-fu you have to use, please share it (or DM), and it can help influence our roadmap.
πŸ‘ 1
❀️ 1
z
sweet. lmk if I can help in any way. This + the global metadata (stack level) will be a great pattern for us
e
Cool, we'll get those knocked out this week probably; maybe not for release but at least in main.
This should work now in #2807 (note it's still building)
Copy code
atmos terraform apply --all --labels=env:dev --include-dependencies --use-version=2807
πŸ‘€ 1
Also,
atmos list *
commands were updated to support the labels & tags
πŸ™Œ 1
z
Trying this out an am hitting the auth error to my other account
Copy code
atmos terraform apply --all --labels=layer:$app-dev1 --include-dependencies=3 --use-version=2807

βœ— Failed to resolve Terraform component instances

   Terraform preflight failed

   Error: failed to execute describe stacks: failed to read Terraform state for component  uds-vpc  in stack $stack...in YAML function:
   !terraform.state uds-vpc...
...
failed to get object from S3:....$bucket" because no resource-based policy allows the s3:ListBucket
  action

    Terraform component instances could not be resolved before Terraform started.
Copy code
atmos list components --use-version=2807
βœ“ Switching to Atmos version 2807

   Error

   Error: failed to execute describe stacks: template: describe-stacks-all-sections:165:93: executing "describe-stacks-all-sections" at
   <atmos.Resolve>: error calling Resolve: failed to describe component  uds-eks  in stack  $stack  in YAML function:  !terraform.state uds-eks
   $stack ".eks_properties.oidc_provider_arn // ""arn:aws-us-gov:iam::000000000000:oidc-provider/default"""  invalid component: Could not find the
   component  uds-eks  in the stack  $stack . Check that all the context variables are correctly defined in the stack manifests. Are the
  component
   and stack names correct? Did you forget an import?
same error when using
--include-dependencies
and a label
e
And do be clear, the expection is that it shouldn't have even tried to auth to that other account because it was out side of the
--labels=layer:$app-dev1 --include-dependencies
scope?
z
manifests from those other stacks are not referenced or imported from the stack or resources I'm trying to actually deploy. FWIW it does show the other account's bucket name, so atmos is resolving those yaml functions properly , getting an actual IAM error from s3 saying I don't have access
e
My guess is it's related to
atmos.Resolve
z
Just for extra context; What my current bash wrapper does at a high level is I can deploy a stack, or a set of stacks, based on metadata in the stack(s) to extrapolate that. And I have all the actual proper DAG stuff on the components, including cross stack references
Copy code
$app-dev1

vars:
  workflow:
    app: $app
    layer: slot
    environment: dev
    slot: 1
    is_active: true
    parents:
      - uds-dev1
      - $app-devx

---
$app-devx

vars:
  workflow:
    app: $app
    layer: environment
    environment: dev
    parents:
      - $app-$account

---
$app-$account

vars:
  workflow:
    app: $app
    layer: account
    account: $account
    parents: []
Hmm, what should atmos be ignoring
e
Using functions like
atmos.Resovle
means atmos literally doesn't know before it tries to resolve it what to do πŸ˜ƒ
πŸ‘€ 1
Will see what we can do
z
meaning, globally? I would have to rip out all atmos.Resolve
e
Let's see what we can do... Fable taking a look πŸ˜ƒ
z
πŸ™
my original thing with combining
--all
with
--stack
🀨.
--reference-stack
to select starting point for DAG-ing or something.
e
Copy code
Pushed as cbddda4e2e β€” Zack's report exposed two real scope leaks, both now fixed on PR #2807. Here's the full picture to relay:

Zack's expectation is correct β€” and he found two genuine holes

Hole 1 β€” stack-granularity evaluation (his "off by one" instinct was right). The closure decided which stacks to evaluate, then evaluated every component in those stack files β€” so a component that merely shares a stack file with an in-closure prerequisite got its !terraform.state resolved, and its cross-account S3 read failed auth, even though it was never selected. That matches his evidence exactly: atmos showed the other account's bucket name because it was faithfully evaluating a component it should never have touched. Fixed: every resolved describe pass is now narrowed to the closure's own components within each stack, with a regression test proving a "poison" component sharing a stack with closure members is never evaluated.

Hole 2 β€” templated selector values. If metadata.labels values are templates (likely, given his labeling strategy), the lightweight pass couldn't decide them, so it conservatively treated every such component as a closure seed β€” quietly inflating the closure toward the whole repo, which explains functions resolving in stacks "not referenced or imported" by anything he deployed. Fixed: since the purity contract guarantees selectors are at most simple templates over the stack context, the lightweight matcher now renders them directly (sprig functions, custom delimiters honored); a value like layer: "{{ .<http://vars.app|vars.app> }}-dev1" now matches exactly. Only genuinely unresolvable values still widen scope β€” conservatively, never excluding.
z
metadata.labels is a string. no templating
e
if you checkout
cbddda4e2e
and run it - see if it works
z
I built off of that branch using
atmos build
and am getting the same error running with
--all --labels
Copy code
[atmos] git log --oneline | head -n 1                                                                                                                                                                                                          cbddda4e2β”‚15:57:55
cbddda4e2 fix: evaluate only the closure's own components, and resolve templated selectors in the lightweight phase
Copy code
Failed to read Terraform state after all retries exhausted
...
βœ— Failed to resolve Terraform component instances

   Terraform preflight failed

   Error: failed to execute describe stacks: failed to describe component  uds-eks  in stack  $stack-stg1  in YAML function:  !terraform.state uds-
  eks   ....

   is not authorized to perform:
  s3:ListBucket
   on resource: "$bucket" because no resource-based policy allows the s3:ListBucket action

    Terraform component instances could not be resolved before Terraform started.
Sometimes it's an
atmos.Resolve
error, sometimes it's not
Copy code
βœ— Failed to resolve Terraform component instances

   Terraform preflight failed

   Error: failed to execute describe stacks: template: describe-stacks-all-sections:221:24: executing "describe-stacks-all-sections" at
   <atmos.Resolve>: error calling Resolve: failed to read Terraform state for component  cloudposse-kms-key  in stack  $app-stgx  in YAML
   function:  !terraform.state cloudposse-kms-key $app-stgx ".key_arn // ""default"""  failed to get object from S3:
every time I run it I kinda get a different reference of something failing, but it's always state backend related
e
Sounds like this is going to take a few more attempts
Alternatively would you be willing to create a branch off mine and have a go?
z
yeah I can try that
would you be opposed to some kind of stack targeted cli switch, so it works similar to how
--stack
somehow functions?
I'll throw some rounds at getting this current thing to work though
e
Can you elaborate on why such a switch would make a difference?
z
No I can't πŸ˜‚ For some reason
--stack
works for me, right? I was thinking if we could point it at reference stack, maybe atmos would be able to look at all the components from the reference stack and build the dependency tree from the reference stack, cross stacks, assuming all of the components were configured correctly
Copy code
dependencies:
        components:
          - name: cloudposse-kms-key
            stack: "$app-{{ .vars.environment_stage }}"
          - name: uds-vpc
            stack: '{{ printf "$app-%s" .vars.deployment_requirements.stage }}'
          - name: uds-eks
            stack: '{{ printf "$app-%s" .vars.deployment_requirements.stage }}'
          - name: uds-bastion
            stack: '{{ printf "$app-%s" .vars.deployment_requirements.stage }}'
Still try to solve for not using any extra switches initially
e
So we also changed labels and tags to not eagerly resolve functions. That's the same behavior as stacks.
There's still something, somewhere we are not seeing.
z
would my templating settings in my atmos.yaml or anything else override that?
I don't have eager_evaluations or whatever the new setting is set.
e
Probably, related; templates are evaluated at different times in the lifecycle
Will pass this along later tonight to see what could be the matter
z
πŸ‘ will let you know if my robot figures something out
e
Random idea, what if you don't specify β€”all
z
I just got it working in my branch, seems to just do the same thing with both
Trying to get the dependencies/dependant flags working now
Copy code
/Volumes/robot-garden/git-repos/oss/atmos/build/atmos \
    terraform plan --all --labels=layer:$app-dev1 \
    --include-dependencies=3 --dry-run

βœ“ Resolved Terraform stacks and dependencies
β–Ά Processing components in dependency order
βœ“ Would plan bastion-admin-policy in $platform-devx (dry run)
βœ“ Would plan cloudposse-kms-key in $app-devx (dry run)
βœ“ Would plan data-$platform-resources in $platform-dev1 (dry run)
βœ“ Would plan key-pair in $platform-dev1 (dry run)
βœ“ Would plan uds-vpc in $platform-dev1 (dry run)
βœ“ Would plan uds-bastion in $platform-dev1 (dry run)
βœ“ Would plan uds-eks in $platform-dev1 (dry run)
βœ“ Would plan uds-rds in $app-dev1 (dry run)
βœ“ Would plan cloudposse-irsa-external-secrets in $app-dev1 (dry run)
cross stack πŸ•
still iterating on this, but progress
πŸ™Œ 1
e
Great progress!
z
Probably wrap this up later tonight
Opened a MR from my fork to your branch github.com/cloudposse/atmos/pull/2816
πŸ™ 1
Any chance you'd have time to peek at this one ^? Going to work on ripping out all my custom workflow stuff and instead using tags/labels going forward assuming this logic makes it in
Scratch that, may be running into some bugginess..
Going to run it through some more scenarios, sorry.
e
are the profiles now working out well for you?
z
yeah. Wish we thought of this sooner
a
i merged your pr into the Erik’s branch
can you test it and confirm everything is working?
πŸ‘ 1
e
I am also running a local code review with fable
z
I actually am having problems with profiles 🫠
e
Heh, good having QA testers like you...
z
Running
atmos describe config
looks like it's working - but it's not actually seeming to respect the
stacks.base_path.included_paths
everything was working fine, then I was like OH I SHOULD DRY THINGS OUT heheh!!!
e
Damp is not bad πŸ˜‰
z
I'm gonna be bald soon from tearing all my hair out
Is there a command in atmos that tells you what stack files it found ?
yeah the only problem is my whole "account split thing" is supposed to be elegantly solved by having separate atmos
stacks.included_paths
configs πŸ˜‚
e
see
atmos profile
commands
z
looks like
--use-version=
doesn't work with
atmos --profile
Copy code
β–Ά New commit on PR #2807 (sha: 654af54 β†’ 82965f1)
β–Ά Installing Atmos from PR #2807...

   Error

   Error: tool installation failed: tool not found

    Build artifact not found for PR #2807

    Failed to install Atmos from PR #2807

   πŸ’‘ Artifacts may have expired (90-day retention)

   πŸ’‘ Build job may have been skipped

   πŸ’‘ Check PR status: <https://github.com/cloudposse/atmos/pull/2807>
oh, nvm it looks like the binary disappeared
e
CleanShot 2026-07-29 at 13.29.08@2x.png
build is in progress
The problem is that it takes the latest commit SHA if you don't specify it and the latest commit SHA doesn't have a binary yet
πŸ‘ 1
Binary is being built
You can specify a previous commit too
z
Oh ok I see
I guess I'll put in another issue/PR for the profile thing
e
Do you have a lead on what the issue is?
z
Having codex dig around for it right now in the background
You don't think it has anything to do with git worktrees does it? Codex is having issues reproducing πŸ€”
e
But I don't know what β€œit” is?
What is the issue with profiles you run into
z
in my main
atmos.yaml
, if I don't have
stacks.included_paths
defined and instead have that merged in via a profile, atmos acts as if it can't find any stacks
even though, if I type atmos describe profile, I see
stacks.included_paths
defined properly, ingesting from the profile I selected
this is when just doing a normal
atmos terraform apply --stack $stack
command, none of the new stuff
ok, found the problem
my fault
name_template
was also getting lopped off..... πŸ₯²
back to this:
can you test it and confirm everything is working?
I think I'm running into a new issue with
atmos.Resolve
where I was not previously. Going to play around with it to make sure I didn't break something else unintentionally..
πŸ‘ 1