https://www.runatlantis.io/ logo
Join Slack
Powered by
This message was deleted.
# atlantis-community
s
This message was deleted.
o
but, if I understand correctly - it’s HA scaling, atlantis will not shard plans/applies between replicas
r
It should if it's saving them to a shared file system like efs
o
Why? Atlantis is getting hook from SCM
And its replicas are not aware about each other
One replica of Atlantis will get hook and will process whole amount of work for specific MR/PR
r
Why? Atlantis is getting hook from SCM
unsure how that’s related. If the pod has an efs mount and multiple pods can share an efs mount and the files are saved to the efs mount, then all the pods can access it.
In the blog you posted, they used something similar to efs, they used a shared NFS which is essentially the same thing
c
The big problem with EFS is that it is...slow.
r
yes, very true! I know EFS has been sped up quite a bit tho in the last 6 to 12 months
Maybe atlantis needs some changes so we can put plans in a separate directory from the code so we can put code in an EBS and only plans on an EFS to make sure the speed of the drive is less affected
c
Well, we tried it a couple months ago and it was still far too slow for us to use.
r
Most likely due to code and plans being colocated. Thats probably the change needed.
Unless there is an alternative to EFS for shared-storage across pods ?
c
Not that I know of.
r
o
@RB files can be shared across pods, however from what I see specific Atlantis pod is in response for processing plans of specific PR/MR
specific pod will get hook and will plan all plans which are part of this hook
r
oh you’re saying that if you have 10 atlantis pods, the webhook will poke 1 pod, and that one pod will trigger all the plans in the PR what you want is the webhook pokes the LB, the LB pokes the server pod, and the server pod kicks off a separate runner pod for each job. I think there is a ticket for this.
d
More work needs to be done to break apart the individual components of Atlantis to scale better. BoltDB -> Redis for project locks is just one step. Cloning the repo and the created plan files is another.
Atlantis was originally coded to run as a singleton binary. Sure, the current method of having the replicas not be aware of each other is true, but you quickly run into problems as you scale of have multiple PRs running for the same repo across different replicas
Especially in the code for cloning the repo, it will collide as the chance for two replicas to try and clone the same repo in the same project exponentially increase
We greatly encourage people to not only submit issues but also submit proposals/ADRs on how to break out these systems to make them more maintainable and scalable.
o
1. Redis is better over SomeDB for sure, because it’s more generic and there is no managed SomeDBs in AWS/GCP/Azure e.t.c. 2. Pods are Kubernetes-specific, issue is Atlantis doesn’t scale. Replicas of Atlantis could listen Redis to process plans/applies from there
o
@Chastity Blackwell,
Well, we tried it a couple months ago and it was still far too slow for us to use.
Could you please expand on what did you try exactly?
c
Using EBS as the storage device.
o
@Chastity Blackwell, EBS was slow for you?
o
EBS is RWO (read write once) and may be used only by one replica at same time (exception only possible when several pods are running on same EC2 instance, but there is no value from such scaling in this case). EFS and NFS-like systems are RWM (read write many)
Without replication EBS and EBS-like storage mechanisms (block storages) aren’t suitable, because several replicas will be not able to use same volume
c
Yeah, EBS was very slow. Like...took 5 times as long to do a plan as with ephemeral storage.
o
That's interesting. EBS works flawlessly for me. It's the EFS that's slow. 🤷
c
Er....wait.
Sorry, I meant EFS 😛
o
...
c
Fucking Amazon and their acronyms 😛
😁 2
d
ECS, EFS, EBS, EKS 😆
To kinda cap on this, I've been thinking about serializing the plans to JSON and storing them in the DB (bolt vs redis) as a feature. This would help with the filesystem stuff
👍 2
🧠 1
o
@Dylan Page There could be backend-agnostic kv-storage interface probably
Or something cloud-native
Or redis
Because bolt isn’t, right?
d
exactly
o
Sometimes I think I wouldn’t like to have Atlantis in Kubernetes
I would like to have it somewhere in ECS Fargate with Elasticache
☝️ 1
d
in its current state, its painful
o
Separated from rest of infrastructure, because Atlantis is point which can compromise all infrastructure and the only way to prevent that is to build zero-trust using thin managed services without much runtime
But someone would use EKS or something else of course
Of course it’s painful, however redis or something generic like SQL database as backend for storing plans could provide such possibility, something exotic - not
c
I run atlantis in fargate for exactly that reason
I don't want to have to stand up kubernetes to run the app I need to bootstrap everything else
o
running on fargate with elasticache redis as well
d
Yup, it is a common use case as some teams use Atlantis to provision their k8s clusters via TF
☝️ 1
r
o
@RB IOPS never was a problem on EFS, problem is a latency and EFS/NFS nature
r
Oh doh! You're right
What about using ebs instead of efs? I saw that fargate / ecs now supports ebs for storage https://aws.amazon.com/about-aws/whats-new/2024/01/amazon-ecs-fargate-integrate-ebs/
o
EBS is ok, but it’s RWO (read-write once), what means only one node at same time can mount it
👍 1
r
Yes, agreed. It would only work for single node deployments. I believe the atlantis terraform-aws-modules fargate deployment is a single deployment so users of that may find it useful to use ebs over efs
3 Views
Open in Slack
PreviousNext
Powered by