https://cloudposse.com logo
Join Slack
Powered by
Trying to implement a blue/green pattern in atmos....
# atmos
z
Trying to implement a blue/green pattern in atmos. DAG doesn't work across multiple stacks yet does it?
Thinking through this I have resources that will be in account level stacks, logical stage level stacks, stage-blue & stage-green level stacks, and a stack that depends on the stage-blue/green stack
I'm thinking the only way for me to orchestrate this at the moment is to have a complex workflow that will deploy my "logical" stage, and deploy everything together in a unit. Like there's no "stack of stacks" type of thing in atmos, right?
b
There is cross-stack dependencies: atmos.tools/stacks/dependencies/components#… Any chance that's what you're looking for?
z
Does it handle the dependencies of the dependency? I had codex look through atmos' codebase on Friday and it was telling me it would have issues 🤔
Maybe I'll just try it out
b
Yeah I don't know, I would think it resolves the entire chain, but I'll leave that to the atmos team to answer for sure
e
Does it handle the dependencies of the dependency?
It should; it computes a complete DAG
DAG doesn't work across multiple stacks yet does it?
Yes, it should.
What happens if you run:
Copy code
atmos list dependencies
Like there's no "stack of stacks" type of thing in atmos, right?
There's a new concept of "compositions"
It's like a slice of a stack
Not sure if it will help with this; it wasn't the original design intent.
We have a PRD discussing "feature flags" and blue/green is a use-case there. github.com/cloudposse/atmos/blob/…/feature-flags.md
Would feature flags like this work for you?
b
ooh would this do something like enabling a dr instance of a stack with a second flag to control if the dr instance is cold/warm/hot?
e
yes, exactly
it formalizes that concept
b
success
Thank you!!!!
e
haha, well, we haven't started implementing the "feature flags" yet 😃
Would be great to get some interest/feedback on it.
b
d'aww, count me in
z
This isn't working for us in our monorepo pattern unfortunately
Untitled.md
Untitled.md
some AI formalizing, but pretty accurate We have a mono repo that houses our entire "landing zone" type deployment. We have a single atmos.yaml. There are 2 accounts. We can only be logged into a single account at a time, physically, we cannot be authed to both at the same time.
DAG tries to hydrate everything. One login cannot access state backend to another etc. So we either need to 1. redo our repository structure to have a split of atmos working environments - two different entrypoints for running atmos commands, moving our stack detection logic around 2. Some kind of feature in atmos to have it scope its evaluations to specific sets of files that we tag with specific metadata or something?
Is there anything that matches this functionality already?
The feature flags thing is promising, if it also creates some kind of logical boundary that would stop further templating that would work for us. Else we need to start refactoring
e
We’ve done some work recently to introduce something called an error mode. That way atmos gracefully degrades when it doesn’t have the ability to compute the value. In this case, it will report that the value is computed rather than nil or result in an error.
I understand what you’re saying I think. Basically you would like transitive dependencies not to be scoped to the stack filter.. I feel like we implemented that for @PePe Amengual .
👀 1
This error mode thing I think is in the next release.
I’ll take a closer look at the doc you shared when I get back to my desk
z
Doing some testing and it seems like DAG doesn't work with 
!terraform.state
for things that are not provisioned yet, even in the same stack?
e
It does, but you need to use mock values
Though there was a recent bug fix
Also, we're formalizing the concept of mock values in the next release
z
specifically getting this error 
failed to execute describe stacks: terraform state not provisioned for component
.... this is when doing a 
atmos terraform plan --all -s $stack
after dag-ing everythin
e
Can you 
--use-version=main
z
Copy code
Error: invalid version output format: invalid version output format: 'main'

   :bulb: Version must be a PR number, pr:NNNN, sha:XXXXXXX, or semver (e.g., 1.2.3)
e
what version of atmos are you using?
z
Copy code
atmos version                                 

 █████  ████████ ███    ███  ██████  ███████ 
██   ██    ██    ████  ████ ██    ██ ██      
███████    ██    ██ ████ ██ ██    ██ ███████ 
██   ██    ██    ██  ██  ██ ██    ██      ██ 
██   ██    ██    ██      ██  ██████  ███████ 
                                             

👽 Atmos 1.223.0 on darwin/arm64
I'll get the sha I guess lol
e
use 
--use-version=ref:main
🙌 1
z
same error unfortunately. I'll try to comb these tracelogs closer. Hard to pinpoint the error. The thing it's failing on is in the same stack which is odd
Ah, think I found it
e
The thing it's failing on is in the same stack which is odd
That would make it consistent 😃
Can you 
--use-version=2755
A lot of small fixes in that too
z
We have !terraform.state pretty much everywhere
there was one that was nested somewhere, no default
p
for those cases you need to give a default value, because you can't get a value that has not been written yet
this is also where you might want to use format() to derive values
z
yeah we basically have no defaults because we were explicitly using workflows to have things run in order so it's all over the place 🫠
It's kind of working 🤔 but not actually applying the dependencies in order
yeah for some reason it's not applying a component that all the other components depend on first
e
are you using 
--affected
or 
--all
if using 
--affected
, that would make sense if it was not affected by changes in the PR
did you run 
atmos list dependencies
?
does the tree view look correct?
z
The tree looks correct like this:
Copy code
atmos list dependencies \
    --stack $stack \
    --direction forward \
    --process-functions=false \
    --process-templates=false \
    --skip=terraform.state --format yaml
I can't not skip things because of the whole split account permissions issue This is the apply I'm running and it seems to just run the top component from the atmos list dependencies command which is weird, not the things it depends on
Copy code
atmos terraform apply --all -s $stack
again this is a greenfield deployment of this stack so nothing exists. I added defaults to every single 
!terraform.state
call and 
atmos.Resolve !terraform.state
BTW this only shows components directly in the stack, and not the cross stack dependencies that I have listed. For example the uds-rds component has dependencies on another stack with our eks cluster component in it.
Copy code
atmos list dependencies \
    --stack $app-dev1 \
    --direction forward \
    --process-functions=false \
    --process-templates=false \
    --skip=terraform.state --format yaml

- component: cloudposse-irsa-dbeaver-external-secrets
  stack: $app-dev1
  depends_on:
    - component: $platform-kms
      stack: $app-dev1
    - component: uds-rds
      stack: $app-dev1
- component: cloudposse-irsa-external-secrets
  stack: $app-dev1
  depends_on:
    - component: $platform-kms
      stack: $app-dev1
    - component: uds-rds
      stack: $app-dev1
- component: $platform-cluster-codepipeline-app
  stack: $app-dev1
  depends_on:
    - component: $platform-efs
      stack: $app-dev1
    - component: $platform-kms
      stack: $app-dev1
    - component: uds-rds
      stack: $app-dev1
- component: $platform-datasync
  stack: $app-dev1
  depends_on:
    - component: $platform-efs
      stack: $app-dev1
- component: $platform-efs
  stack: $app-dev1
  depends_on: []
- component: $platform-kms
  stack: $app-dev1
  depends_on: []
- component: $platform-uds-pkg-dbeaver
  stack: $app-dev1
  depends_on:
    - component: cloudposse-irsa-dbeaver-external-secrets
      stack: $app-dev1
    - component: uds-rds
      stack: $app-dev1
- component: uds-rds
  stack: $app-dev1
  depends_on:
    - component: $platform-kms
      stack: $app-dev1
e
--format yaml
I find 
--format=tree
easier
z
Copy code
atmos list dependencies \
    --stack $app-dev1 \
    --direction forward \
    --process-functions=false \
    --process-templates=false \
    --skip=terraform.state --format tree

 Dependencies
Stack       Component                                      Type
$app-dev1  cloudposse-irsa-dbeaver-external-secrets       terraform
            ├──▶ uds-rds
            │  └──▶ $platform-kms
            └──▶ $platform-kms

$app-dev1  cloudposse-irsa-external-secrets               terraform
            ├──▶ uds-rds
            │  └──▶ $platform-kms
            └──▶ $platform-kms

$app-dev1  $platform-cluster-codepipeline-app                  terraform
            ├──▶ uds-rds
            │  └──▶ $platform-kms
            ├──▶ $platform-efs
            └──▶ $platform-kms

$app-dev1  $platform-datasync                                  terraform
            └──▶ $platform-efs

$app-dev1  $platform-efs                                       terraform
            └──(none)

$app-dev1  $platform-kms                                       terraform
            └──(none)

$app-dev1  $platform-uds-pkg-dbeaver                           terraform
            ├──▶ cloudposse-irsa-dbeaver-external-secrets
            │  ├──▶ uds-rds
            │  │  └──▶ $platform-kms
            │  └──▶ $platform-kms
            └──▶ uds-rds
               └──▶ $platform-kms

$app-dev1  uds-rds                                        terraform
            └──▶ $platform-kms
e
so in this case, it's not doing 
platform-kms
first, you're saying?
z
right
it should be doing 
kms
and 
efs
first
instead it's doing 
cloudposse-irsa-dbeaver-external-secrets
even though it has dependencies
list dependencies even confirms that
tried using 
--affected
instead of 
--all
and that tried to do some evaluations on the state backend for the other account and errored out
e
Looking at these again all of these seem to be cross stack
Any of them not?
z
none of them are cross stack
Atmos doesn't even attempt to apply things that are cross stack with the commands I'm feeding it
e
Copy code
$app-dev1  $platform-cluster-codepipeline-app                  terraform
            ├──▶ uds-rds
            │  └──▶ $platform-kms
            ├──▶ $platform-efs
            └──▶ $platform-kms
isn't 
$platform
another stack?
z
no, the stack is on the left hand side in the tree view. $app-dev1
if you look at the yaml it's more explicit
👍 1
e
Did you try 
--use-version=2755
?
z
ohhh that might have fixed something
🤦‍♂️ yep you told me to use this 2 hours ago
e
Phew!
Regarding cross-stack dependencies behavior with 
--stack
and `--all`; what should happen? do you need that? you could imagine that we wouldn't want a dev stack that happens to have a dependency on prod, to apply prod first.
Copy code
atmos terraform apply --stack=dev --all
z
my gut feeling is that if you're doing 
--stack
and 
--all
, that should create a DAG to do everything it takes to deploy that targeted stack?
the idea I've been kicking around in my head again is the whole like, stack of stacks thing basically like 4 stacks to get to the end goal of getting a specific application deployed in this cluster in this environment, greenfield, or if 2 of the 4 stacks are deployed already
some kind of stackfile metadata over and above the 
stacks.name_template
basically
e
yea, like in cloudformation
z
it looks like this is working 🥲
e
We'll cut a release this week. Hopefully on Wednesday.
🙏 1
z
I guess for now I'll work on creating more workflows to tie these stacks together
That feature flag thing isn't soon soon, is it?
e
We drafted that a while ago, but I haven’t received too much interest yet until today.
did you get a chance to look at the design?
z
I skimmed it, seemed to make sense. Can give it harder look tomorrow.
so this is basically mixins, just with different deploy time logic, you can run the same stack but depending on what features you import, and deploy at runtime it could use different variables
I'm not sure this meets my stack relationship need
e
basically mixins, just with different deploy time logic
Flag driven
e.g. 
--feature=sox
and 
ATMOS_FEATURES=sox,k8s
I'm not sure this meets my stack relationship need
Yes, not related to relationships
Regarding relationships, did you see atmos.tools/changelog/tags-and-labels
z
Will give this a shot. Any idea if it works with DAG? - nvm I see "--all"
is this an AND type filter or an OR type filter
e
and
👍 1
z
I see
• *Tags*— a simple list of words, for basic categorical membership. Filtering matches_any_of the given tags.
• *Labels*— a set of key-value pairs, for richer metadata. Filtering requires_all_of the given key=value pairs, the same semantics as a Kubernetes label selector.
I have to read better
e
Filtering matches_any_of the given tags.
So this is OR. But we should probably provide someway to make it AND
z
use Labels for AND
✅ 1
🤷
Hmmm, still not sure how to tie multiple stacks together though
with my layering logic
e
wouldn't it be to label (aka name) your layers?
how to tie multiple stacks together though
Ok, yes, they would need to be in the same "stack"
z
say I want to apply my $app-dev1 stack. I have dependencies on $app-$account, $app-dev when you apply a stack, that's where my labels would get put on for $app-dev1
yeah it's still not aware of other stacks
the components in other stacks
p
I skimmed to the thread, do not get mad if I ask something you already said: you have an app-stack and you change something in say ( another stack or repo with other stacks) and you want to apply app-stack if those core-dependencies change?
z
more context here https://sweetops.slack.com/archives/C031919U8A0/p1784577352708039?thread_ts=1784567206.798959&cid=C031919U8A0 @PePe Amengual but yeah ideally I'd be able to target one of my blue/green deployment stacks, and it would also deploy the things that it depends on. Or if there was some kind of logical grouping, an atmos terraform apply $SOMETHING that would apply all of those stacks, accounting for DAG
e
so maybe add a "color" label? 😃
z
I think right now I'd need workflows for each logical deployment to imperatively deploy stacks in order each with the --all switch for each stack to do DAG
Not sure, because there are stacks that aren't necessarily blue/green, they are shared between both 🫣
p
this type of trigger would not work?
Copy code
depends_on:
          1:
            file: "stacks/network/us-east-1.yaml"
🤔 1
z
what does that do normally?
p
we have it pointed to a json file that can be changed by AI or a pipeline, if that changes, describe affected will pick it up as a change and then we apply
also you could use a hook: ? I wonder if a hook could trigger a curl call? or maybe you could use it to save and retrieve a SSM parameter that will be the flip for the blue green deployment
I use hooks to read a parameter from SSM, to then trigger a specific pipeline depending of the value
🤔 1
z
like use atmos to store a ssm parameter for active slot or something to fire off a specific pipeline maybe
p
yes
I actually do that now and it does work
but I do believe that the depends_on will be better if you can just add stack names to it as well ( if the file: thing does not work)
Open in Slack
PreviousNext
Powered by