https://cloudposse.com logo
Join Slack
Powered by
I am trying to understand how I can use <cloudposs...
# terraform
j
I am trying to understand how I can use cloudposse/terraform-github-repository at v1.1.0 to create sample "projects" (e.g. dotnet, python, java, HCL) with an expected directory structure
e
Yes, that should be possible, as you can use GitHub template repo together with this module to stamp out projects
Where are you stuck?
j
I guess the old "chicken or the egg" scenario?
Adding the TF as an Atmos component in our service catalog, which works, but trying to understanding how to make "cookie cutter" repositories
e
Do you mean as to how the individual customization within a vended repository would work vis a vie cookie cutter?
Or how the terraform would be able to create a vendor repository from a template?
j
I think both - in a nutshell we need to establish the foundation which means several objectives: 1. Create IAC to manage GitHub repositories 2. Create/Identify Template repositories for different projects including, but not limited to: a. Atmos Stacks b. Terraform c. Java d. .NET e. NodeJS 3. Via service catalog, deploy GitHub repositories that include language-based scaffolding (e.g. create a repository for Java based on template from item 2c)
In the event we need to create template repositories, would something like this work?
Copy code
import:
  - catalog/github/repository/defaults

components:
  terraform:
    github/repository/atmos:
      metadata:
        inherits:
          - github/repository
      vars:
        name: iac-atmos-github-repository
        description: Template repository for Atmos-based projects
        is_template: true
    github/repository/dotnet:
      metadata:
        inherits:
          - github/repository
      vars:
        name: dotnet-github-repository
        description: Template repository for dotnet-based projects
        is_template: true
    github/repository/java:
      metadata:
        inherits:
          - github/repository
      vars:
        name: java-github-repository
        description: Template repository for java-based projects
        is_template: true
    github/repository/nodejs:
      metadata:
        inherits:
          - github/repository
      vars:
        name: nodejs-github-repository
        description: Template repository for nodejs-based projects
        is_template: true
    github/repository/python:
      metadata:
        inherits:
          - github/repository
      vars:
        name: python-github-repository
        description: Template repository for python-based projects
        is_template: true
    github/repository/terraform:
      metadata:
        inherits:
          - github/repository
      vars:
        name: iac-terraform-github-repository
        description: Template repository for Terraform-based projects
        is_template: true
I am assuming yes?
e
Not quite.
You're going to want each component instance to be the name of the repository. That keeps it unique.
And then yes, you'll pass the name. The name can also be a Go template referencing the component instance.
When we've deployed this in the past, we created dedicated GitHub Infra Repo.
j
I was able to ask, based on what I am seeing, 
name
appears to be prefixed with 
{namespace}-{tenant}-environment}-{stage}
(same as 
.atmos_stack
) which isn't necessarily how I want to name my resources in GitHub. Any way around this? For example, if I set 
name
to 
iac-atmos-github-repository
, I want the repository name to be that as is
e
This goes back to the point of having a separate GitHub repository for managing GitHub infra repository names.
Of course, we wouldn't want your repositories named according to that convention.
But that's also why I don't think they belong in that same repo, or if they do, they belong in a separate subfolder of that repo with a different stack name convention.
But no, it's not going to use the canonical name of that component instance. It's going to use whatever is under repository. And nonetheless, each component instance would be disambiguated by its repository name in the stack.
If you did
Copy code
github/repository/terraform:
      metadata:
        inherits:
          - github/repository
      vars:
        name: iac-terraform-github-repository
        description: Template repository for Terraform-based projects
        is_template: true
Then you can only have one 
github/repository/terraform
imo, it should be:
Copy code
github/repository/iac-terraform-github-repository:
      metadata:
        inherits:
          - github/repository
      vars:
        name: iac-terraform-github-repository
        description: Template repository for Terraform-based projects
        is_template: true
or
Copy code
iac-terraform-github-repository:
      metadata:
        inherits:
          - github/repository
      vars:
        name: '{{ .atmos_component }}'
        description: Template repository for Terraform-based projects
        is_template: true
j
So I'm kind of hitting a brick wall here. Here is my stack...
Copy code
vars:
  namespace: cfsb
  tenant: gh
  environment: template
  stage: repos

import:
  - catalog/github/repository/defaults

components:
  terraform:
    github/repository:
      vars:
        is_template: true
        teams: {}
    template/atmos:
      metadata:
        inherits: [github/repository]
      vars:
        description: Template Repository for Atmos projects
        name: iac-atmos
    template/dotnet:
      metadata:
        inherits: [github/repository]
      vars:
        description: Template Repository for .NET projects
        name: dotnet
    template/java:
      metadata:
        inherits: [github/repository]
      vars:
        description: Template Repository for Java projects
        name: java
    template/python:
      metadata:
        inherits: [github/repository]
      vars:
        description: Template Repository for Python projects
        name: python
    template/terraform:
      metadata:
        inherits: [github/repository]
      vars:
        description: Template Repository for Terraform projects
        name: iac-terraform
if I run 
atmos tf plan template/atmos -s cfsb-gh-template-repos
, I see this:
Copy code
# github_repository.default[0] will be created
  + resource "github_repository" "default" {
      + allow_auto_merge                        = false
      + allow_forking                           = (known after apply)
      + allow_merge_commit                      = true
      + allow_rebase_merge                      = true
      + allow_squash_merge                      = true
      + allow_update_branch                     = false
      + archive_on_destroy                      = false
      + archived                                = false
      + auto_init                               = false
      + default_branch                          = (known after apply)
      + delete_branch_on_merge                  = true
      + description                             = "Template Repository for Atmos projects"
      + etag                                    = (known after apply)
      + fork                                    = "false"
      + full_name                               = (known after apply)
      + git_clone_url                           = (known after apply)
      + has_discussions                         = false
      + has_issues                              = false
      + has_projects                            = false
      + has_wiki                                = false
      + html_url                                = (known after apply)
      + http_clone_url                          = (known after apply)
      + id                                      = (known after apply)
      + ignore_vulnerability_alerts_during_read = false
      + is_template                             = true
      + merge_commit_message                    = "PR_BODY"
      + merge_commit_title                      = "PR_TITLE"
      + name                                    = "cfsb-template-repos-iac-atmos"
      + node_id                                 = (known after apply)
      + primary_language                        = (known after apply)
      + private                                 = (known after apply)
      + repo_id                                 = (known after apply)
      + source_owner                            = (known after apply)
      + source_repo                             = (known after apply)
      + squash_merge_commit_message             = "PR_BODY"
      + squash_merge_commit_title               = "PR_TITLE"
      + ssh_clone_url                           = (known after apply)
      + svn_url                                 = (known after apply)
      + topics                                  = (known after apply)
      + visibility                              = "internal"
      + vulnerability_alerts                    = true
      + web_commit_signoff_required             = false

      + security_and_analysis (known after apply)
    }
the core problem is that the code to manage the above is in the same code base used to deploy self-hosted runners. the expectation is that the name of the repo should be exactly what i have in the stack and not be prefixed with anything (repo names should be atmos agnostic)
we need to have absolute control over the name of the repos, but i'm not seeing how that's possible if i use cloudposse/terraform-github-repository at v1.1.0 as an atmos component
e
@Dan Miller (Cloud Posse) can maybe share details. I am 1000 sure it is not opinionated on repo names
cc @johncblandii
I suspect the reason it's getting all those is you're passing this:
Copy code
vars:
  namespace: cfsb
  tenant: gh
  environment: template
  stage: repos
Those are not required. If you don't set those globally they won't get passed
j
That implies I don't set name_pattern in atmos.yaml
e
I think maybe we didn't encounter it b/c it was always separate github-infra repo, with it's own name template.
Since the repo has a GitHub App with org-level admin, we decided it felt safer to keep it separate, that way there's a separation of concerns. Infra repo admins can't access secrets for GitHub App, vise-versa.
j
Got it - I'll try putting the GitHub IAC in an isolated area and test. If that works, then I'm going to assume that, while I can leverage the service catalog, I'll need to have a custom atmos config in an isolated repo
e
There is one more thing you can do that I’ve seen
Recommend not using name pattern as that is an antiquated parameter we support from back in the day when we didn’t support name template. Name pattern only works with the opinionated no label style variables
Instead, if you use a name template, you can make every field optional
Then the name template will work if you also undefined those other variables
Since name templates are just go templates you have full conditional
j
Interesting - I can try that too
e
Copy code
name_template: '{{ list .vars.namespace .vars.stage .vars.environment .vars.name | compact | join "-" }}'
Something like that
d
If I understand correctly, are you asking how you can set the repository name? You can use 
var.repository.name
Copy code
components:
  terraform:
    my-foo-app:
      metadata:
        component: github-repository
      vars:
        repository:
          name: my-foo-app
j
that configuration isn't compatible with cloudposse/terraform-github-repository at v1.1.0
d
checking
Ah I was looking at the component, not the root module. To set it for the module, just do this
Copy code
module "repository" {
  source  = "cloudposse/repository/github"
  version = "1.1.0"

  enabled = local.enabled

  name = var.repository.name

...
}
j
Copy code
atmos
╭─────────────────────────────────────────────╮                                                                                              
│                                             │                                                                                              
│     Commands                                │      Stacks                                         Components                               
│                                             │                                                                                              
│    14 items                                 │     1 item                                         5 items                                   
│                                             │                                                                                              
│    > terraform plan                         │     > deploy/templates                             > dotnet                                  
│      terraform apply                        │                                                      iac-atmos                               
│      terraform destroy                      │                                                      iac-terraform                           
│      terraform init                         │                                                      java                                    
│      terraform output                       │                                                      python                                  
│      terraform clean                        │                                                                                              
│      terraform workspace                    │                                                                                              
│      terraform refresh                      │                                                                                              
│      terraform show                         │                                                                                              
│      terraform validate                     │                                                                                              
│      terraform shell                        │                                                                                              
│      validate component                     │                                                                                              
│      describe component                     │                                                                                              
│      describe dependents                    │                                                                                              
│                                             │                                                                                              
│                                             │                                                                                              
│                                             │                                                                                              
│                                             │                                                                                              
│                                             │                                                                                              
│                                             │                                                                                              
│                                             │                                                                                              
│                                             │                                                                                              
╰─────────────────────────────────────────────╯                                                                                              
↑/k up       /   filter                    enter      execute                                                                                
↓/j down     esc clear filter              esc/ctrl+c quit                                                                                   
←/h left     tab flip stacks/components                                                                                                      

 INFO  Executing command="atmos terraform plan dotnet --stack deploy/templates"

   Error 

   Error: stack name pattern must be provided in 'stacks.name_pattern' config or 'ATMOS_STACKS_NAME_PATTERN' ENV variable
d
For reference, here is the component using that module: https://github.com/cloudposse-terraform-components/aws-github-repository
j
right, but that component implies patterns we don't use
d
Copy code
INFO  Executing command="atmos terraform plan dotnet --stack deploy/templates"

   Error 

   Error: stack name pattern must be provided in 'stacks.name_pattern' config or 'ATMOS_STACKS_NAME_PATTERN' ENV variable
I believe this is an Atmos YAML issue, not related to the component necessarily
j
i am making a component from the TF above
d
Every single component must have the values that are required by your name template. I'm assuming that this instance might not have one of them. That's defined in your Atmos.yaml
Could you share what your Atmos.yml is? In particular, what value do you have for 
stacks.name_pattern
?
j
that's the problem - if i use atmos/examples/quick-start-simple/atmos.yaml at main · cloudposse/atmos, it adds to the name
if i use my existing atmos.yaml bundled with my service catalog, it adds 
module.this.id
as a prefix
d
I'd recommend creating your own Atmos.yaml or just overriding the values you need. Atmos, by default, should load in with default configuration, so you don't have to define an Atmos.yaml necessarily. However, you might want to override just the name pattern so that it only uses values you have
j
Copy code
import:
  - mixins/providers/aws
  - mixins/providers/github

components:
  terraform:
    github/repository:
      backend_type: s3
      backend:
        s3:
          acl: "bucket-owner-full-control"
          encrypt: true
          key: "terraform.tfstate"
          region: "us-east-1"
          use_lockfile: true
          workspace_key_prefix: "github"
      metadata:
        component: terraform-github-repository
        type: abstract # ensures this component cannot be used as a standalone item in a stack
      settings:
        validation:
          github/repository:
            schema_type: jsonschema
            schema_path: "github/repository/defaults.json"
            description: Validate 'github/repository' component variables using JSON Schema
      vars:
        allow_squash_merge: true
        default_branch: main
        delete_branch_on_merge: true
        enabled: true
        rulesets:
          main_branch_protection:
            name: main_branch_protection
            enforcement: active
            target: branch
            conditions:
              ref_name:
                include:
                  - "~DEFAULT_BRANCH"
            rules:
              deletion: true
              non_fast_forward: true
              pull_request:
                required_approving_review_count: 1
              required_status_checks:
                required_check:
                  - context: "ci/build"
                strict_required_status_checks_policy: true
        squash_merge_commit_message: PR_BODY
        tags:
          Terraform: "cloudposse/terraform-github-repository/v1.1.0"
        visibility: internal
d
Or, you can change your atmos.yaml to be only the values that you need. For example, look at how we do it for this basic use case https://github.com/cloudposse-examples/atmos-pro-example-basic/blob/main/atmos.yaml
However, the culprit is very likely just the name_template. You should just change that one value to something that makes sense as for an ID for stacks in your organization
j
what is needed for "bare bones" without atmos.yaml
d
You can delete Atmos.yaml entirely, and it will use the default options. But the default options include the opinionated name template. Let me double check what the default values are. I'm trying to find docs on it
(cc @Erik Osterman (Cloud Posse) @Andriy Knysh (Cloud Posse), do we have documentation on the default values for Atmos.yaml if Atmos.yaml is not provided?)
j
I have a feeling I have to do more than just isolate the code. I think I have to make a component similar to cloudposse-terraform-components/aws-github-repository: This component is responsible for provisioning a Github repository, but without the patterns that break our stuff
d
I think I might be missing some of the context. Could you help fill me in? It should be relatively straightforward. We're probably just missing some of the requirements 1. Where is this component deployed? Is it in its own repo, or is it part of your infrastructure repo or a larger mono repo? 2. Do you already use Atmos for other components? Do you have a convention for stack IDs? 3. Is there any reason you don't want to reuse our component?
j
the catalog item I noted about is built on https://github.com/cloudposse/terraform-github-repository/tree/v1.1.0, but the problem is that atmos is adding 
modules.this.id
produced by the context.tf file as a prefix to all repo names
and we can't use https://github.com/cloudposse-terraform-components/aws-github-repository as is because it implies things (or modules) we don't use
d
ID should be a fallback if name isn't provided
j
d
In your custom component call to the module, are you setting context?
Copy code
module "repository" {
  source  = "cloudposse/repository/github"
  version = "1.1.0"

  enabled = local.enabled

  # keep this
  name = var.repository_name

  # delete this:
  context = module.this.context
...
}
j
No because I am pulling the TF directly
d
Could you share your full local module block?
j
I am vendoring the TF
So there's no module block
d
Let me let Andriy and Erik fill in, but my theory is this 1. The module, https://github.com/cloudposse/terraform-github-repository/tree/v1.1.0, includes within it a context.tf that is used to set the repo name 2. The call to that module, i.e., the aws-github-repository component, sets the repo name with 
var.name
3. That same component call to the module intentionally does not set 
var.context
. Therefore, the ID is always just the name
e
You mean you are vendoring the child module or the root module?
j
root module. there is no child module
e
This is the child module https://github.com/cloudposse/terraform-github-repository/ This is the root module (component implementation) https://github.com/cloudposse-terraform-components/aws-github-repository that calls the child module.
j
got it, so then i am vendoring the child module
e
It would be helpful to know what about the root module didn't work for you
j
mainly the use of the account-map
e
Aha, agree but why don't we help you with that. In general, we’re trying to eliminate the account map.
j
I'm all ears
e
@Dan Miller (Cloud Posse) do you recall why this module introduced the account map?
d
I believe it's only used in providers.tf. It should be able to be ignored with the same pattern as the other components
e
@Dan Miller (Cloud Posse) do you have a 
generate
block he can use to replace providers?
d
absolutely. We're currently working on putting together a blog post to explain this better, but in the meanwhile, you can just add the following to your stack config and it will override the vendored providers.tf
providers.yaml
providers.yaml
j
very close...
Copy code
import:
  - mixins/providers

components:
  terraform:
    github/repository:
      backend_type: s3
      backend:
        s3:
          acl: "bucket-owner-full-control"
          encrypt: true
          key: "terraform.tfstate"
          region: "us-east-1"
          use_lockfile: true
          workspace_key_prefix: "github"
      metadata:
        type: abstract # ensures this component cannot be used as a standalone item in a stack
#      settings:
#        validation:
#          github/repository:
#            schema_type: jsonschema
#            schema_path: "github/repository/defaults.json"
#            description: Validate 'github/repository' component variables using JSON Schema
      source:
        uri: <http://github.com/cloudposse-terraform-components/aws-github-repository//src|github.com/cloudposse-terraform-components/aws-github-repository//src>
        excluded_paths:
          - "**/providers.tf"
        version: v0.3.0
      vars:
        owner: cfsbny
        region: us-east-1
        repository:
          allow_merge_commit: true
          allow_squash_merge: true
          allow_rebase_merge: true
          allow_auto_merge: false
          auto_init: true
          default_branch: main
          delete_branch_on_merge: true
          gitignore_template: TeX
          license_template: GPL-3.0
          squash_merge_commit_message: PR_BODY
          visibility: internal
only thing not working is exclusion of 
<http://providers.tf|providers.tf>
when using JIT vendoring
if I can ensure providers.tf isn't download, it'll work
d
make sure you have 
auto_generate_files: true
https://atmos.tools/stacks/generate#auto-generation
j
so there's no way for JIT vendoring to simply exclude the providers.tf?
d
you can exclude it in "excluded_paths"
j
Copy code
source:
        uri: <http://github.com/cloudposse-terraform-components/aws-github-repository//src|github.com/cloudposse-terraform-components/aws-github-repository//src>
        excluded_paths:
          - "**/providers.tf"
        version: v0.3.0
excluded_paths not being honored...
5 Views
Open in Slack
PreviousNext
Powered by