https://www.runatlantis.io/ logo
Join Slack
Powered by
This message was deleted.
# atlantis-community
s
This message was deleted.
t
Are you using EFS? I ran into that problem and had to change my EFS throughput mode 
throughput_mode = "elastic"
v
Yes, I am using EFS. Unn let me try that
t
If you check the EFS graphs you should see spikes to your current maximum throughput, that's how I figured out what the problem was.
v
You are right, I noticed that before but did not make the connection
Thanks for the input, I will try your solution
t
You're welcome!
v
Changed it manually in AWS, ran the plan again and still got the same error. I don't think I need to redeploy the container, right?
t
Hm, no I wouldn't think so
v
Yeah, thought so too. And there was no throughput abuse this time either
t
How many CPUs are you giving the container?
v
Copy code
CPU | Memory
2 vCPU | 4 GB
t
You might check the CPU metrics on the container too and bump that up. We run 8 vCPU and 16 GB RAM
Might be worth trying 4 vCPU
v
Unn, I see. Will try that as well. Thanks!
I had noticed already that they are a bit on the edge whenever I run multiple plans, but with parallel plan disabled, I was not having any issues until now
j
We bumped ours up to 16vCPU, 32GB RAM yesterday because we have huge repos with dozens of parallel plans
Terraform is very CPU hungry
t
How many parallel plans do you allow?
v
The default one which is 15
j
We were only allowing 5 parallel plans until yesterday. Now we are going to use the default of 15
v
I see. I was going to move to 5 also to avoid this. I am not even sure I need 15 😃
Moving to 4vCPU and 8GB of RAM fixed it, alongside with EFS change. But at the same time, doesn't it feel a little overkill to allocate that many resources to Atlantis?
🙌 1
j
It depends on how much work atlantis is doing. Even locally if you were to run 
terraform init && terraform plan
in parallel for several directories containing a few hundred resources your CPU utilization would spike. Most modern dev laptops have way more power than 4vCPU/8GB RAM.
The problem is terraform running in parallel, not atlantis itself
v
Yeah, that's a good point. Some of my plans most of the time will run against multiple tfstate and deploy many resources at once.
Thanks a lot for the help on this @Jason Reslock!
j
You can also tune terraform a bit but it would require custom workflow to set the appropriate 
parallelism
flags for plan and apply: https://developer.hashicorp.com/terraform/cli/commands/plan#parallelism-n The default is 10 even if your atlantis parallelism is set to 5. I interpret this as "terraform will fork 10 processes by default to do things unless you tell it to use less"
Specifically I am talking about this option:
Copy code
-parallelism=n - Limit the number of concurrent operations as Terraform walks the graph. Defaults to 10.
v
Ah yes, I use this locally in some cases. But I did not consider that in the Atlantis level. Will check this out, thanks again!
6 Views
Open in Slack
PreviousNext
Powered by