https://www.runatlantis.io/ logo
Join Slack
Powered by
I'm running Atlantis on ECS Fargate. I'm curious a...
# atlantis-community
c
I'm running Atlantis on ECS Fargate. I'm curious about issues with timeouts... I've seen the multi-jobs kicked off from a project end up being "Killed" or resulting in timeouts like this: 
│ Error: failed to read schema for module.k8s-cluster.module.external_secrets_irsa_role.aws_iam_policy.vpc_cni in <http://registry.terraform.io/hashicorp/aws|registry.terraform.io/hashicorp/aws>: failed to instantiate provider "<http://registry.terraform.io/hashicorp/aws|registry.terraform.io/hashicorp/aws>" to obtain schema: timeout while waiting for plugin to start
is there a way to fix these?
p
might be related to the cache sharing?
c
not sure what that is exactly. I am using EFS in this deployment
p
Copy code
Note: The plugin cache directory is not guaranteed to be concurrency safe. The provider installer's behavior in environments with multiple terraform init calls is undefined.
c
they were all sharing the default workspace... I just made each project use it's own so let's see if that helps this
p
let us know how it goes
c
before I felt like if one job failed it was cleaning up the cache because almost immediately the others were failing so hopefully the separate workspaces will... obviously more storage but if it helps avoid them stepping on each other then should be good
p
yes
c
nope still getting: 
signal: killed
p
what about resources?
memory , cpu ? are you hitting any limits?
c
looks like cpu and memory only hit about max of 50% utilization from ECS dashboard I'm curious though since I have 10 projects running at once with this repo shouldn't ECS scale in some way? Not sure if that's option with the atlantis terraform module to allow for some sort of scaling option
p
Atlantis can’t scale horizontally
only vertically
c
i just increased the memory from 2Gi to 3Gi to see if it helps even though from AWS Console it doesn't look like it's using the whole thing... cpu is only 1024 so not sure if that needs to be increased or not
so at 3Gi only got 1 more success others still "signal: killed"
You'd think in a container scenario it would take each job and be able to turn them into individual (child containers)... so each has it's own resources... the service and main atlantis task would be static just create "jobs" for each job atlantis runs... not sure if this is the way it works in Kubernetes deployment that each job would turn into another pod or not but having a supervisor/controller seems like best way to scale... it's not really AutoScaling because its still 1 instance/service you are hitting it just scales out the containers that run each job... this way your dedicating resources for each job so they don't collide with each other in anyway... if running inside Docker then each would become another container running and the main atlantis container just "manages" those for their status
p
atlantis was not designed like that
look for a issue called Atlantis HA there is a long discussion there
c
is there a limit of concurrent jobs for repo like this with multiple projects? I bumped to 4Gi but still having timeout issues so rather than bumping memory to something ludacris like 10Gi to get all of these to work it might be better to have them run only 2 at a time?
p
yes, there is some caveats on that
Share storage and redis locks and such
is not worker pool/queue/job
c
I'm using the Fargate install with EFS and Atlantis does support Redis which could be Elasticache so if the code change is made then the locking issue shouldn't be a problem and would allow to scale out...
I guess I could disable parallel plan and apply to see if that helps with the resource issues
wish there was a way to have a parallel pool size ... i know someone brought it up in another issue... but basically seems like parallel has default of 10? would be nice to set this to 2 or 4 based on resources given to task
p
with parallel runs it gets trickier and TF is not thread-safe either
c
I set
Copy code
+ {
                          + name  = "ATLANTIS_PARALLEL_POOL_SIZE"
                          + value = "4"
                        },
but only see 1 plan running... not sure why that is but hopefully it stops my timeout issues... at that point I might as well disable parallel all together if it's not going to run 4
also the Remove lock and discard plan in the web UI you click Yes and it does nothing if you refresh it's gone but that button refresh should be fixed
o
more memory (16Gi), more cpu (4 vCPU), get rid of efs in favour of ebs, limit the parallel pool size to < 20 and you should be fine
also this env var is helpful:
Copy code
TF_PLUGIN_CACHE_MAY_BREAK_DEPENDENCY_LOCK_FILE=true
I'm using the Fargate install with EFS and Atlantis does support Redis which could be Elasticache so if the code change is made then the locking issue shouldn't be a problem and would allow to scale out...
we're using redis with ecs fargate and it works flawlessly, using 
cache.t2.micro
for over a 1.5 years now and around 500 terraform stacks
4 Views
Open in Slack
PreviousNext
Powered by