https://cloudposse.com logo
Join Slack
Powered by
For those that vendor their service catalog with a...
# atmos
j
For those that vendor their service catalog with atmos, does the following workflow align with what you do?
Copy code
atmos vendor pull          # download IAC service catalog
cd atmos                   # this is where service catalog is pulled
atmos vendor pull          # download dependencies of the service catalog
atmos toolchain install    # install binaries
atmos terraform plan --all
I am starting to abstract docker from the service catalog so IAC developers can use atmos locally
e
Why do you want to vendor twice?
Technically, toolchain dependencies get installed automatically, so you don't need to do it explicitly unless you want to.
@RB and @PePe Amengual might have opinions here
p
mmm we do this differently
we vendor pull 
components
, 
stacks
and 
catalog
all in one go
if you want to pull some components and not others then you need to create a vendor file with that list
we vendor everything on the runner ( never on the service repo) and work from the 
/atmos
dir within the service repo
the 
/atmos
dir of the service only contains the same dir structure that the monorepo IAC where we vendor from and a service.yaml per stack for every environment
so vendoring happens before we run any atmos command to make the service a valid stack
that way the service does not have TF files, or any core infra stack files and the user can just modify the service.yaml with what they need and they have no clue we run atmos or TF ( some of them know )
j
I keep my service catalog in a separate repo from the stacks. So the first pull brings in the service catalog and the second, inside of 
./atmos
, pulls in dependencies for the service catalog (TF modules)
Not saying it's the right way, it's actually why I am asking. Not sure if there's a more efficient method
p
I do not see the reason to have it separated, the whole point of the catalog is to create 
"safe defaults"
for your components , so it should leave as close as the components as possible
j
Not sure I'm understanding
Could you please elaborate @PePe Amengual
p
the catalog is just inputs vars for your components that then you import in your stacks
your components are in A repo, you change the component code, decide that a input needs to be added as default to the catalog, you change the catalog and then everyone can get the new value automatically, why would you have the catalog in another repo?
the same process I described will ave to be done in 2 different repos base of what I understand of your setup
j
ah, because we have different atmos projects for different scopes of work, but what brings them together is the use of the service catalog, which contains the bulk of the configurations, schema validation, etc.
p
ahhh ok, we do not do that
j
@Erik Osterman (Cloud Posse) i haven't seen evidence that atmos automatically installs from toolchain - maybe something with my config?
p
what you call the service catalog for us is the service.yaml that is on the service repo, but that is the only file from atmos that lives there
j
interesting - ty for the clarification - with that being said - is there anything inherently "wrong" with what i am suggesting?
p
not really, more a comment but anything, but you need to think how you version the catalog and in a separated repo that could be a nightmare
compomnent repo changes-> new version is created--> renovate pics the change and created pr in catalog repo?-> then your service changes automatically or manually? etc
vendoring 
main
is not a good idea
j
we don't vendor 
main
- only tagged releases
p
ok, but you vendor versioned components from one repo and catalog from another?
j
nope - components and catalog in one repo - stacks (and custom stuff) in one or more
p
I see ok
so is basically what we do but we have all in one repo and we "delete" stuff that the service do not need
also we use 
locked:true
in the stacks repo (in your case) so that the service can't modify core infra
j
Good to know. Mind showing me docs for that?
p
is a metadata value
✅ 1
j
@Erik Osterman (Cloud Posse) safe to assume this is something that would help? Just-in-Time Component Vendoring with source | atmos
If I'm read the above, and underlying documentation, I should be able to set component-level vendoring, which means the only reason I would need to run 
atmos vendor pull
should be when I need to vendor the service catalog as a whole
e
Absolutely! That would make things easier
j
Good to know. You also said running 
atmos toolchain install
explicitly was unnecessary because tools should be installed automatically. What configs are needed for that? Can you point me to the example documentation?
e
j
It has to be at the component level? How would I add to a mixin that could be added to any component?
I updated my AWS mixin, but 
terraform
still doesn't install automatically
Copy code
terraform:
  dependencies:
    tools:
      terraform: "1.11.4"
  providers:
    aws:
      region: "us-east-1"
      version: "5.100.0" #TODO: <https://github.com/cloudposse/atmos/issues/1194>
Same error when running 
atmos terraform plan --all
Copy code
Error 

   Error: exec: "terraform": executable file not found in $PATH
e
that LGTM; can you try with out 
--all
and do a specific component instead?
j
same error if i specify a specific component/stack
e
Note, we have another fix which is realted to the 
PATH
issue (atmos was using only relative paths)
Can you confirm it's being installed, just not found
the PATH fix issue is resovled but not released
j
Not seeing an install being attempted
Copy code
INFO  Executing command="atmos terraform plan al2023-dnf-upgrade --stack cfsb-aws-compute-dev"
 DEBU  ExecuteTerraform entry SubCommand=plan ComponentFromArg=al2023-dnf-upgrade FinalComponent="" Stack=cfsb-aws-compute-dev StackFromArg=""
 DEBU  readParentDirConfig checking CWD for local config cwd=/home/jrose/bb/iac-service-catalog/tests/atmos
 DEBU  Skipping parent directory search (local config exists) path=/home/jrose/bb/iac-service-catalog/tests/atmos
 DEBU  readGitRootConfig checking CWD for local config cwd=/home/jrose/bb/iac-service-catalog/tests/atmos
 DEBU  Skipping git root search (local config exists) path=/home/jrose/bb/iac-service-catalog/tests/atmos
 DEBU  Git root base path discovery enabled
 DEBU  Checking if should call ExecuteDescribeComponent Stack=cfsb-aws-compute-dev ComponentFromArg=al2023-dnf-upgrade SubCommand=plan
 DEBU  readParentDirConfig checking CWD for local config cwd=/home/jrose/bb/iac-service-catalog/tests/atmos
 DEBU  Skipping parent directory search (local config exists) path=/home/jrose/bb/iac-service-catalog/tests/atmos
 DEBU  readGitRootConfig checking CWD for local config cwd=/home/jrose/bb/iac-service-catalog/tests/atmos
 DEBU  Skipping git root search (local config exists) path=/home/jrose/bb/iac-service-catalog/tests/atmos
 DEBU  Git root base path discovery enabled
 DEBU  Found component 'al2023-dnf-upgrade' in the stack 'cfsb-aws-compute-dev' in the stack manifest 'orgs/aws-compute/run'
 DEBU  Resolved component path type=terraform component=component resolved_path=/home/jrose/bb/iac-service-catalog/tests/atmos/components/terraform/terraform-aws-image-builder/modules/component base_path=/home/jrose/bb/iac-service-catalog/tests/atmos/components/terraform env_override=false
 DEBU  CreateAndAuthenticateManager called identityName="" hasAuthConfig=true
 DEBU  No identity resolved, returning nil
 DEBU  Found component 'al2023-dnf-upgrade' in the stack 'cfsb-aws-compute-dev' in the stack manifest 'orgs/aws-compute/run'
 DEBU  Resolved component path type=terraform component=component resolved_path=/home/jrose/bb/iac-service-catalog/tests/atmos/components/terraform/terraform-aws-image-builder/modules/component base_path=/home/jrose/bb/iac-service-catalog/tests/atmos/components/terraform env_override=false
 DEBU  Resolved component path type=terraform component=component resolved_path=/home/jrose/bb/iac-service-catalog/tests/atmos/components/terraform/terraform-aws-image-builder/modules/component base_path=/home/jrose/bb/iac-service-catalog/tests/atmos/components/terraform env_override=false
 DEBU  Variables for the component in the stack component=al2023-dnf-upgrade stack=cfsb-aws-compute-dev
 DEBU  PrintAsYAMLToFileDescriptor
  data=
  │ component_name: al2023-dnf-upgrade
  │ component_version: 2023.9.20251117
  │ create: true
  │ data: |
  │   phases:
  │     - name: 'build'
  │       steps:
  │         - name: 'al2023-dnf-upgrade'
  │           action: 'ExecuteBash'
  │           inputs:
  │             commands:
  │               - 'dnf upgrade --releasever=2023.9.20251117 -y' # REFS: <https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.9.20251027.html>
  │               - 'dnf update && dnf upgrade -y'
  │           onFailure: 'Abort'
  │   schemaVersion: '1.0'
  │ description: Upgrade AL2023 to latest package versions
  │ enabled: true
  │ environment: compute
  │ namespace: cfsb
  │ owner: self
  │ platform: Linux
  │ stage: dev
  │ tags:
  │   Terraform: cfsb/terraform-aws-image-builder/modules/component
  │ tenant: aws

 DEBU  Resolved component path type=terraform component=component resolved_path=/home/jrose/bb/iac-service-catalog/tests/atmos/components/terraform/terraform-aws-image-builder/modules/component base_path=/home/jrose/bb/iac-service-catalog/tests/atmos/components/terraform env_override=false
 DEBU  Writing the variables file=/home/jrose/bb/iac-service-catalog/tests/atmos/components/terraform/terraform-aws-image-builder/modules/component/cfsb-aws-compute-dev-terraform-aws-image-builder-modules-al2023-dnf-upgrade.terraform.tfvars.json
 DEBU  Validating component=al2023-dnf-upgrade schema=jsonschema file=image-builder/component.json
 DEBU  Validate 'image-builder' component variables using JSON Schema
 DEBU  Resolved component path type=terraform component=component resolved_path=/home/jrose/bb/iac-service-catalog/tests/atmos/components/terraform/terraform-aws-image-builder/modules/component base_path=/home/jrose/bb/iac-service-catalog/tests/atmos/components/terraform env_override=false
 DEBU  Writing the backend config to file. file=/home/jrose/bb/iac-service-catalog/tests/atmos/components/terraform/terraform-aws-image-builder/modules/component/backend.tf.json
 DEBU  Writing the provider overrides to file. file=/home/jrose/bb/iac-service-catalog/tests/atmos/components/terraform/terraform-aws-image-builder/modules/component/providers_override.tf.json
 DEBU  Using ENV vars:
 DEBU  ATMOS_CLI_CONFIG_PATH=/home/jrose/bb/iac-service-catalog/tests/atmos
 DEBU  ATMOS_BASE_PATH=/home/jrose/bb/iac-service-catalog/tests/atmos
 DEBU  TF_IN_AUTOMATION=true
 DEBU  TF_APPEND_USER_AGENT=Atmos/test (Cloud Posse; +<https://atmos.tools>)
 DEBU  TF_PLUGIN_CACHE_DIR=/home/jrose/.cache/atmos/terraform/plugins
 DEBU  TF_PLUGIN_CACHE_MAY_BREAK_DEPENDENCY_LOCK_FILE=true
 DEBU  Terraform environment file not found. No action needed. file=/home/jrose/bb/iac-service-catalog/tests/atmos/components/terraform/terraform-aws-image-builder/modules/component/.terraform/environment
 DEBU  Executing command="terraform init"
 DEBU  readParentDirConfig checking CWD for local config cwd=/home/jrose/bb/iac-service-catalog/tests/atmos
 DEBU  Skipping parent directory search (local config exists) path=/home/jrose/bb/iac-service-catalog/tests/atmos
 DEBU  readGitRootConfig checking CWD for local config cwd=/home/jrose/bb/iac-service-catalog/tests/atmos
 DEBU  Skipping git root search (local config exists) path=/home/jrose/bb/iac-service-catalog/tests/atmos
 DEBU  Git root base path discovery enabled
 DEBU  readParentDirConfig checking CWD for local config cwd=/home/jrose/bb/iac-service-catalog/tests/atmos
 DEBU  Skipping parent directory search (local config exists) path=/home/jrose/bb/iac-service-catalog/tests/atmos
 DEBU  readGitRootConfig checking CWD for local config cwd=/home/jrose/bb/iac-service-catalog/tests/atmos
 DEBU  Skipping git root search (local config exists) path=/home/jrose/bb/iac-service-catalog/tests/atmos
 DEBU  Git root base path discovery enabled
 DEBU  Telemetry is disabled, skipping capture

   Error 

   Error: exec: "terraform": executable file not found in $PATH
 DEBU  Exiting with exit code code=1
e
https://github.com/cloudposse/atmos/pull/2127 can you try 
--use-version 2127
j
close
Copy code
$ atmos --use-version 2127
ℹ Installing Atmos from PR #2127...
✓ Found workflow run #22608145490 (sha: f44ddca) from Mar 2, 2026 at 11:21 PM
✓ Installed to /home/jrose/.local/share/atmos/toolchain/bin/cloudposse/atmos/pr-2127/atmos
✓ Switching to Atmos version 2127
╭─────────────────────────────────────────╮                                                                                      
│                                         │                                                                                      
│     Commands                            │      Stacks                                     Components                           
│                                         │                                                                                      
│    14 items                             │     9 items                                    17 items                              
│                                         │                                                                                      
│    > terraform plan                     │     > cfsb-aws-compute-dev                     > al2023-dnf-upgrade                  
│      terraform apply                    │       cfsb-aws-containers-dev                    amazon-cloudwatch-agent-linux       
│                                         │                                                                                      
│    •••••••                              │     •••••                                      •••••••••                             
│                                         │                                                                                      
╰─────────────────────────────────────────╯                                                                                      
↑/k up       /   filter                    enter      execute                                                                    
↓/j down     esc clear filter              esc/ctrl+c quit                                                                       
←/h left     tab flip stacks/components                                                                                          

 INFO  Executing command="atmos terraform plan al2023-dnf-upgrade --stack cfsb-aws-compute-dev"
✓ Installed hashicorp/terraform@1.11.4 to /home/jrose/.local/share/atmos/toolchain/bin/hashicorp/terraform/1.11.4/terraform (86mb)

   Error 

   Error: exec: "terraform": executable file not found in $PATH
not the directory i was expecting though
i was expecting terraform to be installed into 
./tools
e
Did you override the tool chain installation path?
By default, it's using the XDG location.
What is your atmos.yaml configuration?
j
Copy code
# Toolchain configuration
toolchain:
  # Path to .tool-versions file (relative or absolute)
  file_path: ".tool-versions"
  # Directory where tools are installed
  tools_dir: ".tools"
  # Tool name aliases for convenient CLI usage
  aliases:
    # HashiCorp tools
    terraform: hashicorp/terraform
    tf: hashicorp/terraform
e
Hrmmm Alright, I don't know what to say. Maybe another issue. :(
j
maybe?
e
There was another PR I can't recall if we merged it or not.
There's a PR fixing the path issue.
We need that one to go out in the next release as well, and I think that's why you're seeing it installed correctly but not finding it.
@Andriy Knysh (Cloud Posse) We need to get this one merged next.
👍 1
Jonathan, this is the reason why it's installing it but not able to find it.
You’re not doing anything wrong. We just have to get this fixed by merging it. -
j
okie dokie
a
the PR is approved and merged. a new release will go out today
j
Sweet(Ops) 🙂
👍 1
@Erik Osterman (Cloud Posse) I think WSL does this, which would make sense as to why 
atmos toolchain
installed 
terraform
where it did
Copy code
XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
e
Alright, so that makes sense that it would install there, but if you had overridden the installation path, then it should not be installed there, which would be a separate issue.
And just to confirm, you did override the installation path, Is that correct?
j
yeah
9 Views
Open in Slack
PreviousNext
Powered by