Terragrunt converts: if you could have started usi...
# atmos
e
Terragrunt converts: if you could have started using atmos from day one, just by doing something this in your stack configs, would that have accelerated migration/adoption?
Copy code
import:
# import terragrunt units
- <terragrunt://my-terragrunt-dir/infra-live/vpc/terragrunt.hcl>
- <terragrunt://my-terragrunt-dir/infra-live/iam/terragrunt.hcl>
- <terragrunt://my-terragrunt-dir/infra-live/eks/terragrunt.hcl>

# or import a terragrunt stack
- <terragrunt://myterragrunt/infra-live/terragrunt.stack.hcl>

components:
  terraform:
    vpc:
      vars: ... anything you want to start managing in atmos
    iam:
      vars: ... anything you want to start managing in atmos
    eks:
      vars: ... anything you want to start managing in atmos
This would then get imported automatically into native atmos, and executed by atmos, not atmos calling terragrunbt. You could then systematically move things out of terragrunt, and into stack configs, or just leave it this way for legacy infra. If something doesn't work, you'd be able to continue using terragrunt because it's not changing the terragrunt configs.
@Jonathan Rose
@Cyrus Dukart
j
@Esteban Angee
Going to be honest, that is pretty freaking cool!
e
@Erik Osterman (Cloud Posse) looks pretty cool. There's one thing I've noticed since we use terraform module wrappers adapted for terragrunt. With atmos, I haven't found a way to reference one item in the items collection from another. i.e:
Copy code
security-groups:
  items:
    sg1:
      ...
     sg2:
       ingress_rules_with_source_security_group: 
         security_group_id: <sg1-id>
Not asking for any feature. Just something to note. We have had to split these type of components into multiple for easy
!terraform.outputs
reference
j
@Esteban Angee I think that's a "symptom" to how we use terragrunt-style wrappers as root modules
e
With atmos, I haven't found a way to reference one item in the items collection from another.
Yes, as @Jonathan Rose alludes to, we are quite opinionated against creating factories in terraform
If for no other reason, it creates this problem; the other problem is you state grows and grows and grows. And one change to any item in the factory, has a blast radius of every item in there.
e
@Erik Osterman (Cloud Posse) noted. We'll certainly have a discussions about this soon
j
@Erik Osterman (Cloud Posse) fun fact. @Esteban Angee and I discussed about refactoring some code we use for deploying EC2 Image Builder and removing terragrunt-style wrappers. I guess my question is how that impacts remote state configuration and how to ensure components/mixins continue to be reusable.
I read through the documentation above and maybe I'm not understanding because I'm a visual learner?
e
So moving the state out of the terraform factory will be semi pain. But the good news is you're doing it on your own terms, instead of being corned when it bites you for real. Unfortunately though, we don't provide any utilities for helping move state around at this time, so it would need to be largely "manual" using
import { ... }
blocks, loading it into a new workspace.
I guess my question is how that impacts remote state configuration and how to ensure components/mixins continue to be reusable.
We use this pattern almost exclusively, I wouldn't anticipate any problems; that said, if you can bring up a specific example of your concern, we can tell you how we would handle it.
j
I am making a root terraform module to do one of two things: 1. aws_imagebuilder_component | Resources | hashicorp/aws | Terraform | Terraform Registry to create a new private component 2. aws_imagebuilder_component | Data Sources | hashicorp/aws | Terraform | Terraform Registry to store an existing (e.g. aws managed) component into remote state Here is my component (POC):
Copy code
components:
  terraform:
    image-builder/component:
      backend_type: s3
      backend:
        s3:
          acl: "bucket-owner-full-control"
          encrypt: true
          key: "terraform.tfstate"
          region: "us-east-1"
          use_lockfile: true
          workspace_key_prefix: "image-builder/component"
      metadata:
        component: terraform-aws-image-builder/modules/component
      settings:
        validation:
          validate-image-builder-component-with-jsonschema:
            schema_type: jsonschema
            schema_path: "image-builder/component.json"
            description: Validate 'image-builder' component variables using JSON Schema
      vars:
        tags:
          Terraform: "cfsb/terraform-aws-modules/terraform-aws-image-builder"
Here is how I believe it should be implemented if I am understanding the use case correctly
Copy code
import:
  - catalog/image-builder/component

components:
  terraform:
    cis-benchmark-level-1-amazon-linux:
      metadata:
        component: terraform-aws-image-builder/modules/component
        inherits:
          - image-builder/component
      vars:
        create: false
        name: cis-benchmark-level-1-amazon-linux
        owner: third-party
        tags:
          Purpose: "CIS Benchmark Level 1 AMI"
          Service: "ImageBuilder CIS"
But how do I ensure each item is stored correctly into remote state?
I assume I would need
backend.s3.key
for each item?
nvm...I think I answered my own question
e
but how do I ensure each item is stored correctly into remote state?
If you are relying on atmos backend generation, and it's properly configured, you don't need to do anything
We always ensure each component instance has it's own workspace in the terraform state
j
OK, so I think I have a working example, but the trade-off of not using terragrunt-style wrapper means the atmos workflow would be much longer. Correct?
e
Not it you use
--all
or
--affected
j
ooooo
e
Here's what we use for the backend config.
Copy code
backend:
    s3:
      acl: bucket-owner-full-control
      bucket: acme-core-use1-root-tfstate
      encrypt: true
      key: terraform.tfstate
      region: us-east-1
      assume_role:
        role_arn: arn:aws:iam::__ROOT_ACCOUNT_NUMBER__:role/acme-core-gbl-root-tfstate
      use_lockfile: true
And now if you run
atmos list intances
you'll see all the component instances from your factory
and it's really easy to plan/apply changes to any instance in the factory
j
I think I got a working example:
Copy code
import:
  - catalog/image-builder/component

components:
  terraform:
    al2023-dnf-upgrade:
      metadata:
        component: terraform-aws-image-builder/modules/component
        inherits:
          - image-builder/component
      vars:
        create: true
        data: |
          phases:
            - name: 'build'
              steps:
                - name: 'al2023-dnf-upgrade'
                  action: 'ExecuteBash'
                  inputs:
                    commands:
                      - 'dnf upgrade --releasever=2023.9.20251117 -y' # REFS: <https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.9.20251027.html>
                      - 'dnf update && dnf upgrade -y'
                  onFailure: 'Abort'
          schemaVersion: '1.0'
        description: Upgrade AL2023 to latest package versions
        name: al2023-dnf-upgrade
        owner: self
        platform: Linux
        component_version: 2023.9.20251117
    amazon-cloudwatch-agent-linux:
      metadata:
        component: terraform-aws-image-builder/modules/component
        inherits:
          - image-builder/component
      vars:
        create: false
        name: amazon-cloudwatch-agent-linux
        owner: aws
        tags:
          Purpose: "Amazon CloudWatch Agent for Linux"
          Service: "ImageBuilder CloudWatch"
    cis-benchmark-level-1-amazon-linux:
      metadata:
        component: terraform-aws-image-builder/modules/component
        inherits:
          - image-builder/component
      vars:
        create: false
        name: cis-benchmark-level-1-amazon-linux
        owner: third-party
        tags:
          Purpose: "CIS Benchmark Level 1 AMI"
          Service: "ImageBuilder CIS"
Does this look right?
The above would deploy three components where one is privately created and two are used to get Arns for existing components
Copy code
INFO  Executing command="atmos terraform plan amazon-cloudwatch-agent-linux --stack cfsb-aws-compute-dev"
Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v5.100.0

╷
│ Warning: Version constraints inside provider configuration blocks are deprecated
│ 
│   on providers_override.tf.json line 16, in provider.aws:
│   16:       "version": "5.100.0"
│ 
│ Terraform 0.13 and earlier allowed provider version constraints inside the provider configuration block, but that is now deprecated and will be removed in a future version of Terraform. To silence
│ this warning, move the provider version constraint into the required_providers block.
│ 
│ (and one more similar warning elsewhere)
╵
Terraform has been successfully initialized!

Workspace "cfsb-aws-compute-dev-amazon-cloudwatch-agent-linux" doesn't exist.

You can create this workspace with the "new" subcommand 
or include the "-or-create" flag with the "select" subcommand.
Created and switched to workspace "cfsb-aws-compute-dev-amazon-cloudwatch-agent-linux"!

You're now on a new, empty workspace. Workspaces isolate their state,
so if you run "terraform plan" Terraform will not see any existing state
for this configuration.
data.aws_partition.current: Reading...
data.aws_region.current: Reading...
data.aws_partition.current: Read complete after 0s [id=aws]
data.aws_region.current: Read complete after 0s [id=us-east-1]
data.aws_imagebuilder_component.this[0]: Reading...
data.aws_imagebuilder_component.this[0]: Read complete after 0s [id=arn:aws:imagebuilder:us-east-1:aws:component/amazon-cloudwatch-agent-linux/1.0.1/1]

Changes to Outputs:
  + arn = "arn:aws:imagebuilder:us-east-1:aws:component/amazon-cloudwatch-agent-linux/1.0.1/1"

You can apply this plan to save these new output values to the Terraform state, without changing any real infrastructure.

Warnings:

- Value for undeclared variable
- Value for undeclared variable
- Values for undeclared variables
- Version constraints inside provider configuration blocks are deprecated
  on providers_override.tf.json line 16

To see the full warning notes, run Terraform without -compact-warnings.
Fun fact. One of main reasons my team choose atmos over terragrunt was because you guys made it much easier to leverage publicly accessible terraform modules.
e
ya, we should probably add first class component and module registries that you can configure and search
list
e.g.
Copy code
atmos registry list components
atmos registry search "vpc" --include-modules
atmos registry install "@cloudposse/vpc" --version 1.2.3
thoughts? ☝️
j
oh! that looks awesome!
@Esteban Angee ^^