How do you pin providers across all components ? ...
# atmos
r
How do you pin providers across all components ? We ran into an issue today due to the newest https://github.com/cloudposse/terraform-provider-utils/releases/tag/v1.32.0 provider and need to pin across all of our components to
v1.31.0
Most likely due to us using atmos 1.168.0 alongside v1.32.0 of the utils provider. Im sure if we used the latest atmos with the latest utils provider, it probably would have worked fine
a
1.32.0 was broken today (b/c of some GH actions issues)
we just re-released it again 30 mins ago, can you please try again?
r
it might be cached, our stuff is still broken
a
(not sure if it was updated in the TF and Tofu registries yet)
r
it would be better to probably release a v1.33.0
a
i can trigger that (apologies for breaking it)
🙌 1
r
no worries
is there a way to pin our providers using some kind of atmos yaml magic, perhaps https://atmos.tools/components/terraform/providers ?
it would be nice to do this, then we can use renovate to slowly bump versions
e
Yes just use a generate block
r
hmmm but would it overwrite the versions.tf file that we vendored in ?
e
You can use overrides
🤔 1
r
i’d rather the generate block, or similar, override the versions.tf’s terraform.required_providers.cloudposse/utils.version to == 1.31.0 and keep everything else as-is
a
you mentioned it broke something. what was broken?
r
you mentioned it broke something. what was broken?
it caused all the atmos terraform pipelines to fail until we pinned it back to 1.31
a
what error?
r
Copy code
Error: invalid stack manifest: template: catalog/services/snip/_defaults.yaml:3:57: executing "catalog/services/snip/_defaults.yaml" at <.settings.config.short_stage>: map has no entry for key "settings"
File being processed: catalog/services/snip/_defaults.yaml Import chain: → orgs/snip/snip/us-east-1.yaml```
failed during terraform-plan, after init
e
terraform.required_providers.cloudposse/utils.version to == 1.31.0 and keep everything else as-is
Perry sure you can use
<http://_override.tf|_override.tf>
for that Combined with
generate
block to generate that file
🤔 1
👀 1
a
RB you are using very old atmos which didn't detects those issues, we need to upgrade you anyway can you please send me (DM) that stack manifest?
e
No
a
i just want to see what old atmos config causes the issue
r
ya they are working on the upgrade this week for the friday call 😄
i can’t share the stack manifest without getting some red-tape approvals
unless you folks have some way of anonymizing stack yaml
e
@Zack has a slick way he generates reproduce code
r
it would be nice to have some docs on version pinning in the atmos docs too 🙂
r
i’ll share the convo in this discussion with the ops team, thanks again for weighing in here
is there still going to be a 1.33 🙏
e
a
yes we can create a doc, but that will not solve your issue, if you update to the latest atmos, you will still get the same error pining is a temporary fix
r
yes, of course, completely understood
Doesn't talk about version pinning for your use case
We can add a separate doc for provider pinning
👍 1
🙌 1
a
RB, from the error, it looks like you have a template with
settings.config.short_stage
, but the value is not provided. The old atmos just ignored it should be an easy fix (find the template, provide the value - we can help if help is needed)
e
Does the terraform overrides make sense? It’s the terraform native way of overriding anything
When you combine that with atmos file generate, those files can be written automatically
r
Yes it makes sense, but at the moment, since it’s late, ops is just looking for a quick and dirty fix
I think with the “generate”, we’d also need to run the atmos generate command in the pipeline
Since we’re kind of paving a new path here and using an ancient version, we’re a little reluctant to try the feature over just pinning the version manually, so we’ll go with that strategy for now, and then when time permits, poc the more scalable strategy, and probably do it with the latest version soon enough 🙂
e
No, generation happens automatically if enabled
But yes, for an old version, will need to use some other way :-)
r
Hmm yep, same issue with v1.33.0 as we had with v1.32.0 so it looks like we definitely have to pin to v1.31.0 at least for now Thanks again for trying a new 1.x release for us Andriy
The minimum version now for the latest 1.x utils provider is atmos 1.207.x or 1.209.x https://github.com/cloudposse/terraform-provider-utils/blob/8aedc996bfeee9207b2d3b82e4f54526f70d8e91/go.mod#L6
a
tou are right, we need to release a new provider version as soon as we release a new atmos version
you still need to check the template error (if it's the only error, should be easy to fix) and update atmos and provider version - you can't stay at the old version forever
r
we’re on a newer version of atmos now, youve probably already heard
i tried the atmos generation via claude and it failed 😞
Copy code
POC failed. New error confirms a Terraform constraint I missed:

Error: Duplicate required providers configuration
A module may have only one required providers configuration.

Terraform does not merge required_providers blocks across .tf files. A module has at most one. My earlier "Terraform merges multiple required_providers blocks" claim was wrong.

What this means

The "centralize one provider via generate:, leave others in <http://versions.tf|versions.tf>" approach is impossible. The only ways to make this work would be:

1. Generate the entire <http://versions.tf|versions.tf> from YAML — delete each component's <http://versions.tf|versions.tf> and declare ALL providers (aws, null, local, utils, github, ...) in generate.versions.tf.json. Then leverage Atmos's inheritance/deep-merge so a baseline generate: block in stacks/orgs/org/_defaults.yaml carries utils, and component-level YAML adds the rest. Atmos merges the maps before rendering. Larger refactor; would need its own POC.
2. Use a string template in generate: — emit raw HCL via Go template. Same single-block problem; you'd still need to include all required_providers.
looks like there is a related issue here https://github.com/cloudposse/atmos/issues/1194
e
Do you mean the
generate
block?
Because the linked issue, I don't believe is implemented.
r
yes the generate block
should this work?
Copy code
generate:
        <http://versions.tf|versions.tf>:
          terraform:
            required_providers:
              utils:
                source: cloudposse/utils
                version: "1.35.0"
e
Yes, that would replace a file called
versions.tf
r
oh so it won’t “merge” into that file
e
No, it's code generation.
r
ahhh 😞
e
It has no idea what it's doing 😃
Instead, try
versions_override.tf
r
hmmm let me try that
z
https://developer.hashicorp.com/terraform/language/files/override You're basically generating these on the fly. But you can also use atmos to generate whatever
r
yes, it worked for me 🙂 thanks all
Copy code
_versions_override.tf.json:
          terraform:
            required_providers:
              utils:
                source: cloudposse/utils
                version: "1.35.0"
🙌 1
e
thanks to @Zack for thoroughly testing and helping us fix the
generate
block
🙌 1
He uses it extensively
🤨 1
party parrot 1