Zack
07/22/2026, 11:50 PMZack
07/22/2026, 11:52 PMAndriy Knysh (Cloud Posse)
07/22/2026, 11:55 PMZack
07/22/2026, 11:56 PMZack
07/22/2026, 11:56 PMErik Osterman (Cloud Posse)
07/23/2026, 12:11 AMmore terraform command options that aren't specifically a stack or every single stackTags & labels, no?
Zack
07/23/2026, 12:11 AMZack
07/23/2026, 12:12 AMZack
07/23/2026, 12:14 AMatmos terraform apply --affected --tags production hmmmmmErik Osterman (Cloud Posse)
07/23/2026, 12:16 AM--error-mode which handles graceful degradation. For example, it would be able to run atmos terraform apply --affected --tags production and it will only error if one of the targeted components cannot resolve the values. But if say some value cannot be resolved in dev, it doesn't care.Erik Osterman (Cloud Posse)
07/23/2026, 12:16 AMZack
07/23/2026, 12:16 AM!terraform.state not provisioned thing also?Erik Osterman (Cloud Posse)
07/23/2026, 12:17 AMErik Osterman (Cloud Posse)
07/23/2026, 12:17 AMZack
07/23/2026, 12:17 AM'{{ atmos.Resolve (printf `!terraform.state cloudposse-kms-key hello-world-%s ".key_arn // ""default"""` (regexReplaceAll "[0-9]+$" .vars.deployment_requirements.stage "x")) }}'Erik Osterman (Cloud Posse)
07/23/2026, 12:17 AMErik Osterman (Cloud Posse)
07/23/2026, 12:17 AMErik Osterman (Cloud Posse)
07/23/2026, 12:18 AMYea, I'm an expert at atmos and I can't read thatCopy code'{{ atmos.Resolve (printf `!terraform.state cloudposse-kms-key hello-world-%s ".key_arn // ""default"""` (regexReplaceAll "[0-9]+$" .vars.deployment_requirements.stage "x")) }}'
Erik Osterman (Cloud Posse)
07/23/2026, 12:18 AMZack
07/23/2026, 12:18 AMErik Osterman (Cloud Posse)
07/23/2026, 12:18 AMZack
07/23/2026, 12:20 AMZack
07/23/2026, 12:21 AMZack
07/23/2026, 12:22 AMAndriy Knysh (Cloud Posse)
07/23/2026, 12:33 AMHere's what that expression does, step by step (I verified the semantics against internal/exec/template_funcs.go:61, internal/exec/yaml_func_terraform_state.go, and the new argument parser in pkg/function/parser/parser.go):
1. It's a Go template expression, evaluated while Atmos renders templates during stack processing β i.e., before YAML functions are normally processed, with the current component/stack as context.
2. Innermost step β regexReplaceAll "[0-9]+$" .vars.deployment_requirements.stage "x": takes the stage name from vars.deployment_requirements.stage and replaces a trailing run of digits with the literal letter x. For example, prod-3 β prod-x, dev12 β devx; a stage with no trailing digits (e.g. dev) passes through unchanged. So all numbered instances of a stage collapse to one -x name.
3. printf builds a YAML-function call as a string, injecting that transformed stage into the format hello-world-%s. The result is:
!terraform.state cloudposse-kms-key hello-world-prod-x ".key_arn // ""default""" (using the prod-3 example).
4. The doubled quotes are the escaping convention for quoted YAML-function arguments (a "" inside a "β¦"-quoted argument is a literal "). After unquoting, the third argument is the YQ expression .key_arn // "default". The final """ is an escaped quote plus the closing quote.
5. atmos.Resolve evaluates that string as the !terraform.state YAML function at template-render time. This is the reason for the whole construction: a bare !terraform.state YAML tag owns the entire scalar and can't have a templated stack name spliced into it, whereas atmos.Resolve lets you compose the function call from template data.
6. !terraform.state reads the Terraform state directly from the configured backend (no terraform init/terraform output subprocess) for component cloudposse-kms-key in stack hello-world-<stage-with-x>, then applies the YQ expression to the state outputs.
7. The YQ // operator provides a fallback: it returns the key_arn output, or the string "default" if key_arn is null or missing.
8. The fallback also covers the never-provisioned case: because the expression contains //, Atmos treats "state not provisioned" and "output not found" as recoverable β it evaluates the YQ default against an empty document and returns "default" instead of failing the whole stack processing.
Net effect: "Give me the KMS key ARN from the cloudposse-kms-key component in the hello-world-<stage> stack β where any numeric stage suffix is normalized to x (so e.g. prod-1, prod-2, prod-3 all share hello-world-prod-x's key) β and if that component isn't provisioned or has no key_arn output, use the string default instead."
One caveat worth knowing: the digit-run replacement maps prod-123 to prod-x (the entire run of digits is replaced by a single x), and the literal string "default" is a placeholder value, not a real ARN β so downstream consumers presumably treat default as a sentinel (e.g., "use the AWS-managed key").Zack
07/23/2026, 12:36 AMErik Osterman (Cloud Posse)
07/23/2026, 12:57 PMZack
07/23/2026, 4:23 PMdistinction between direct Terraform state references and state references
used inside generated JSON.
A direct value like this works as expected:
kms_key_id: !terraform.state cloudposse-kms-key ...
That value is consumed directly as a Terraform input, so Atmos resolves it during stack
processing.
The external-secrets KMS ARN follows a different path:
Terraform variable
β IAM policy statement
β JSON serialization with toJson
β aws_iam_policy
Because the value is embedded inside a templated policy structure, the !terraform.state
expression can survive into the later template/JSON-rendering phase instead of being fully
evaluated first. Terraform then receives malformed JSON and reports errors such as:
invalid character '.' after array element
Wrapping the expression with atmos.Resolve forces Atmos to resolve the remote state value before
the policy is serialized:
external_secrets_kms_key_arn: '{{ atmos.Resolve (printf `!terraform.state ...`) }}'
That produces a concrete ARN before toJson builds the IAM policy.
General rule: direct !terraform.state references are appropriate for ordinary Terraform
variables. When a state-derived value is interpolated into a generated JSON document, policy, or
another nested template, resolve it explicitly first with atmos.Resolve.Erik Osterman (Cloud Posse)
07/24/2026, 2:20 AMfoo: !terraform.state {
component: cloudposse-kms-key,
stack: 'hello-world-{{ regexReplaceAll "[0-9]+$" .vars.deployment_requirements.stage "x" }}',
output: key_arn,
default: default,
}Erik Osterman (Cloud Posse)
07/24/2026, 2:20 AMErik Osterman (Cloud Posse)
07/24/2026, 2:21 AMZack
07/24/2026, 2:52 AM| toJson or similar. I don't totally hate the current implementation, it's probably just how I'm feeding and templating everythingErik Osterman (Cloud Posse)
07/24/2026, 3:13 AM| toJson?Erik Osterman (Cloud Posse)
07/24/2026, 3:14 AM!anythinng is it's not templating; it's actually YAML. so it doesn't need manipulation)Zack
07/24/2026, 3:16 AMZack
07/24/2026, 3:20 AMZack
07/24/2026, 5:40 AM--labels won't work for our use case, had the AI try to replicate what I did recently with our (new) layered stacks pattern
atmos terraform plan --all \
--labels=poc=atmos-lifecycle,account=$ACCOUNT,app=hello-world,environment=dev,layer=slot,slot=1 \
--dry-run --max-concurrency=1
Error
Error: failed to execute describe stacks: failed to describe component uds-eks in stack $STACK-stg in YAML function:
!terraform.state uds-eks $STACK-stg ".eks_properties.cluster_name // ""default""" invalid component: Could not find the component uds-
eks in the stack $STACK-stg . Check that all the context variables are correctly defined in the stack manifests. Are the component
and stack names correct? Did you forget an import?
this is it bombing out, trying to hit our staging environment. Tries to do some kind of evaluation before filtering through the stacks that it would be applicable to.Zack
07/24/2026, 5:42 AMterraform.metadata thing I can apply to all components in the stack π€·. I ended up creating an abstract component and importing it and inheriting to each component to add the metadata, dry-ishZack
07/24/2026, 5:44 AMZack
07/24/2026, 5:44 AMZack
07/24/2026, 5:49 AMIn a monorepo containing multiple accounts and independently valid stack families, --all --labels=β¦ should filter candidate
component instances before resolving all stack templates/functions/state. Today it eagerly evaluates unrelated stacks, making label-based cross-stack execution unusable for account-scoped operations.
I think overall if there was like a way to solve for that, and also have like some kind of stack level metadata - like there's a name now, it might come closer to what I'm picturing for labels/tags to work. Also if there was more logic to be able to do AND/OR with both labels/tagsErik Osterman (Cloud Posse)
07/24/2026, 5:56 AMErik Osterman (Cloud Posse)
07/24/2026, 5:56 AMErik Osterman (Cloud Posse)
07/24/2026, 5:56 AMErik Osterman (Cloud Posse)
07/24/2026, 5:57 AMErik Osterman (Cloud Posse)
07/24/2026, 5:59 AMErik Osterman (Cloud Posse)
07/24/2026, 5:59 AMatmos terraform plan --all \
--labels=poc=atmos-lifecycle,account=$ACCOUNT,app=hello-world,environment=dev,layer=slot,slot=1 \
--dry-run --max-concurrency=1Erik Osterman (Cloud Posse)
07/24/2026, 6:00 AMErik Osterman (Cloud Posse)
07/24/2026, 6:00 AMErik Osterman (Cloud Posse)
07/25/2026, 11:19 PMErik Osterman (Cloud Posse)
07/25/2026, 11:19 PMErik Osterman (Cloud Posse)
07/25/2026, 11:19 PM--use-version=2807Zack
07/26/2026, 1:54 AMZack
07/26/2026, 2:01 AMZack
07/26/2026, 2:02 AMZack
07/26/2026, 2:04 AMErik Osterman (Cloud Posse)
07/26/2026, 2:46 AMErik Osterman (Cloud Posse)
07/26/2026, 2:48 AMWill there be a way to list stacks and components matching the filters? Probably list stacks?Adding that. Good point.
Erik Osterman (Cloud Posse)
07/26/2026, 2:49 AM--stack you should also be able to filter by --labels and --tagsErik Osterman (Cloud Posse)
07/26/2026, 2:54 AMatmos describe affected, we have:
--include-dependents Include the dependent components and stacks`Erik Osterman (Cloud Posse)
07/26/2026, 2:54 AMand apply all stacks that are a dependency of that stack
Erik Osterman (Cloud Posse)
07/26/2026, 2:55 AMatmos terraform apply would that meet your needs?Zack
07/26/2026, 3:05 AMZack
07/26/2026, 3:07 AMZack
07/26/2026, 3:07 AMZack
07/26/2026, 3:08 AMZack
07/26/2026, 3:11 AMErik Osterman (Cloud Posse)
07/26/2026, 3:19 AMErik Osterman (Cloud Posse)
07/26/2026, 3:21 AMZack
07/26/2026, 3:26 AMZack
07/26/2026, 3:29 AMZack
07/26/2026, 3:29 AMZack
07/26/2026, 3:31 AMZack
07/26/2026, 4:24 AMZack
07/26/2026, 3:21 PMZack
07/26/2026, 3:22 PMErik Osterman (Cloud Posse)
07/26/2026, 5:39 PMZack
07/27/2026, 4:17 PMErik Osterman (Cloud Posse)
07/27/2026, 4:20 PM--include-dependents=depth (default unlimited) and --include-dependencies=depth for all commands that support labels and tags. Hit rate limits though, so won't get back to it until this afternoon.Erik Osterman (Cloud Posse)
07/27/2026, 4:21 PMZack
07/27/2026, 4:32 PMErik Osterman (Cloud Posse)
07/27/2026, 6:36 PMErik Osterman (Cloud Posse)
07/27/2026, 9:03 PMatmos terraform apply --all --labels=env:dev --include-dependencies --use-version=2807Erik Osterman (Cloud Posse)
07/27/2026, 9:04 PMatmos list * commands were updated to support the labels & tagsZack
07/27/2026, 9:44 PMatmos terraform apply --all --labels=layer:$app-dev1 --include-dependencies=3 --use-version=2807
β Failed to resolve Terraform component instances
Terraform preflight failed
Error: failed to execute describe stacks: failed to read Terraform state for component uds-vpc in stack $stack...in YAML function:
!terraform.state uds-vpc...
...
failed to get object from S3:....$bucket" because no resource-based policy allows the s3:ListBucket
action
Terraform component instances could not be resolved before Terraform started.Zack
07/27/2026, 9:54 PMatmos list components --use-version=2807
β Switching to Atmos version 2807
Error
Error: failed to execute describe stacks: template: describe-stacks-all-sections:165:93: executing "describe-stacks-all-sections" at
<atmos.Resolve>: error calling Resolve: failed to describe component uds-eks in stack $stack in YAML function: !terraform.state uds-eks
$stack ".eks_properties.oidc_provider_arn // ""arn:aws-us-gov:iam::000000000000:oidc-provider/default""" invalid component: Could not find the
component uds-eks in the stack $stack . Check that all the context variables are correctly defined in the stack manifests. Are the
component
and stack names correct? Did you forget an import?
same error when using --include-dependencies and a labelErik Osterman (Cloud Posse)
07/27/2026, 10:05 PM--labels=layer:$app-dev1 --include-dependencies scope?Zack
07/27/2026, 10:10 PMErik Osterman (Cloud Posse)
07/27/2026, 10:15 PMatmos.ResolveZack
07/27/2026, 10:17 PM$app-dev1
vars:
workflow:
app: $app
layer: slot
environment: dev
slot: 1
is_active: true
parents:
- uds-dev1
- $app-devx
---
$app-devx
vars:
workflow:
app: $app
layer: environment
environment: dev
parents:
- $app-$account
---
$app-$account
vars:
workflow:
app: $app
layer: account
account: $account
parents: []Zack
07/27/2026, 10:18 PMErik Osterman (Cloud Posse)
07/27/2026, 10:28 PMatmos.Resovle means atmos literally doesn't know before it tries to resolve it what to do πErik Osterman (Cloud Posse)
07/27/2026, 10:29 PMZack
07/27/2026, 10:29 PMErik Osterman (Cloud Posse)
07/27/2026, 10:29 PMZack
07/27/2026, 10:30 PMZack
07/27/2026, 10:34 PM--all with --stack π€¨. --reference-stack to select starting point for DAG-ing or something.Erik Osterman (Cloud Posse)
07/27/2026, 10:39 PMPushed as cbddda4e2e β Zack's report exposed two real scope leaks, both now fixed on PR #2807. Here's the full picture to relay:
Zack's expectation is correct β and he found two genuine holes
Hole 1 β stack-granularity evaluation (his "off by one" instinct was right). The closure decided which stacks to evaluate, then evaluated every component in those stack files β so a component that merely shares a stack file with an in-closure prerequisite got its !terraform.state resolved, and its cross-account S3 read failed auth, even though it was never selected. That matches his evidence exactly: atmos showed the other account's bucket name because it was faithfully evaluating a component it should never have touched. Fixed: every resolved describe pass is now narrowed to the closure's own components within each stack, with a regression test proving a "poison" component sharing a stack with closure members is never evaluated.
Hole 2 β templated selector values. If metadata.labels values are templates (likely, given his labeling strategy), the lightweight pass couldn't decide them, so it conservatively treated every such component as a closure seed β quietly inflating the closure toward the whole repo, which explains functions resolving in stacks "not referenced or imported" by anything he deployed. Fixed: since the purity contract guarantees selectors are at most simple templates over the stack context, the lightweight matcher now renders them directly (sprig functions, custom delimiters honored); a value like layer: "{{ .<http://vars.app|vars.app> }}-dev1" now matches exactly. Only genuinely unresolvable values still widen scope β conservatively, never excluding.Zack
07/27/2026, 10:41 PMErik Osterman (Cloud Posse)
07/27/2026, 10:48 PMcbddda4e2e and run it - see if it worksZack
07/27/2026, 11:03 PMatmos build and am getting the same error running with --all --labels
[atmos] git log --oneline | head -n 1 cbddda4e2β15:57:55
cbddda4e2 fix: evaluate only the closure's own components, and resolve templated selectors in the lightweight phaseZack
07/27/2026, 11:06 PMFailed to read Terraform state after all retries exhausted
...
β Failed to resolve Terraform component instances
Terraform preflight failed
Error: failed to execute describe stacks: failed to describe component uds-eks in stack $stack-stg1 in YAML function: !terraform.state uds-
eks ....
is not authorized to perform:
s3:ListBucket
on resource: "$bucket" because no resource-based policy allows the s3:ListBucket action
Terraform component instances could not be resolved before Terraform started.
Sometimes it's an atmos.Resolve error, sometimes it's not
β Failed to resolve Terraform component instances
Terraform preflight failed
Error: failed to execute describe stacks: template: describe-stacks-all-sections:221:24: executing "describe-stacks-all-sections" at
<atmos.Resolve>: error calling Resolve: failed to read Terraform state for component cloudposse-kms-key in stack $app-stgx in YAML
function: !terraform.state cloudposse-kms-key $app-stgx ".key_arn // ""default""" failed to get object from S3:Zack
07/27/2026, 11:07 PMErik Osterman (Cloud Posse)
07/27/2026, 11:09 PMErik Osterman (Cloud Posse)
07/27/2026, 11:09 PMZack
07/27/2026, 11:10 PMZack
07/27/2026, 11:11 PM--stack somehow functions?Zack
07/27/2026, 11:11 PMErik Osterman (Cloud Posse)
07/27/2026, 11:17 PMZack
07/27/2026, 11:20 PM--stack works for me, right? I was thinking if we could point it at reference stack, maybe atmos would be able to look at all the components from the reference stack and build the dependency tree from the reference stack, cross stacks, assuming all of the components were configured correctly
dependencies:
components:
- name: cloudposse-kms-key
stack: "$app-{{ .vars.environment_stage }}"
- name: uds-vpc
stack: '{{ printf "$app-%s" .vars.deployment_requirements.stage }}'
- name: uds-eks
stack: '{{ printf "$app-%s" .vars.deployment_requirements.stage }}'
- name: uds-bastion
stack: '{{ printf "$app-%s" .vars.deployment_requirements.stage }}'Zack
07/27/2026, 11:22 PMErik Osterman (Cloud Posse)
07/27/2026, 11:25 PMErik Osterman (Cloud Posse)
07/27/2026, 11:25 PMZack
07/27/2026, 11:26 PMZack
07/27/2026, 11:27 PMErik Osterman (Cloud Posse)
07/27/2026, 11:27 PMErik Osterman (Cloud Posse)
07/27/2026, 11:28 PMZack
07/27/2026, 11:30 PMErik Osterman (Cloud Posse)
07/27/2026, 11:56 PMZack
07/28/2026, 12:00 AMZack
07/28/2026, 12:00 AMZack
07/28/2026, 12:16 AM/Volumes/robot-garden/git-repos/oss/atmos/build/atmos \
terraform plan --all --labels=layer:$app-dev1 \
--include-dependencies=3 --dry-run
β Resolved Terraform stacks and dependencies
βΆ Processing components in dependency order
β Would plan bastion-admin-policy in $platform-devx (dry run)
β Would plan cloudposse-kms-key in $app-devx (dry run)
β Would plan data-$platform-resources in $platform-dev1 (dry run)
β Would plan key-pair in $platform-dev1 (dry run)
β Would plan uds-vpc in $platform-dev1 (dry run)
β Would plan uds-bastion in $platform-dev1 (dry run)
β Would plan uds-eks in $platform-dev1 (dry run)
β Would plan uds-rds in $app-dev1 (dry run)
β Would plan cloudposse-irsa-external-secrets in $app-dev1 (dry run)Zack
07/28/2026, 12:17 AMZack
07/28/2026, 12:20 AMErik Osterman (Cloud Posse)
07/28/2026, 12:21 AMZack
07/28/2026, 1:10 AMZack
07/28/2026, 5:07 AMZack
07/28/2026, 6:28 PMZack
07/28/2026, 7:00 PMZack
07/28/2026, 7:02 PMErik Osterman (Cloud Posse)
07/29/2026, 5:01 PMZack
07/29/2026, 5:01 PMAndriy Knysh (Cloud Posse)
07/29/2026, 5:11 PMAndriy Knysh (Cloud Posse)
07/29/2026, 5:11 PMErik Osterman (Cloud Posse)
07/29/2026, 5:13 PMZack
07/29/2026, 6:14 PMErik Osterman (Cloud Posse)
07/29/2026, 6:15 PMZack
07/29/2026, 6:16 PMatmos describe config looks like it's working - but it's not actually seeming to respect the stacks.base_path.included_pathsZack
07/29/2026, 6:16 PMErik Osterman (Cloud Posse)
07/29/2026, 6:17 PMZack
07/29/2026, 6:17 PMZack
07/29/2026, 6:18 PMZack
07/29/2026, 6:19 PMstacks.included_paths configs πErik Osterman (Cloud Posse)
07/29/2026, 6:19 PMatmos profile commandsZack
07/29/2026, 6:21 PM--use-version= doesn't work with atmos --profileZack
07/29/2026, 6:21 PMβΆ New commit on PR #2807 (sha: 654af54 β 82965f1)
βΆ Installing Atmos from PR #2807...
Error
Error: tool installation failed: tool not found
Build artifact not found for PR #2807
Failed to install Atmos from PR #2807
π‘ Artifacts may have expired (90-day retention)
π‘ Build job may have been skipped
π‘ Check PR status: <https://github.com/cloudposse/atmos/pull/2807>Zack
07/29/2026, 6:22 PMErik Osterman (Cloud Posse)
07/29/2026, 6:29 PMErik Osterman (Cloud Posse)
07/29/2026, 6:29 PMErik Osterman (Cloud Posse)
07/29/2026, 6:29 PMErik Osterman (Cloud Posse)
07/29/2026, 6:29 PMErik Osterman (Cloud Posse)
07/29/2026, 6:30 PMZack
07/29/2026, 6:31 PMZack
07/29/2026, 6:32 PMErik Osterman (Cloud Posse)
07/29/2026, 7:39 PMZack
07/29/2026, 7:39 PMZack
07/29/2026, 8:06 PMErik Osterman (Cloud Posse)
07/29/2026, 8:08 PMErik Osterman (Cloud Posse)
07/29/2026, 8:08 PMZack
07/29/2026, 8:09 PMatmos.yaml, if I don't have stacks.included_paths defined and instead have that merged in via a profile, atmos acts as if it can't find any stacksZack
07/29/2026, 8:09 PMstacks.included_paths defined properly, ingesting from the profile I selectedZack
07/29/2026, 8:11 PMatmos terraform apply --stack $stack command, none of the new stuffZack
07/29/2026, 8:18 PMZack
07/29/2026, 8:18 PMZack
07/29/2026, 8:19 PMname_template was also getting lopped off..... π₯²Zack
07/30/2026, 12:39 AMcan you test it and confirm everything is working?I think I'm running into a new issue with
atmos.Resolve where I was not previously. Going to play around with it to make sure I didn't break something else unintentionally..