Hello Guys, I'm a happy user of atlantis and I ran into a serious problem today, when adding new resources. Namely, I received a message in the log that the server is killing terraform in action. In the log I got this information:

{"level":"error","ts":"2024-04-22T14:48:52.225Z","caller":"events/instrumented_project_command_runner.go:78","msg":"Error running plan operation: running \"/usr/local/bin/terraform plan -input=false -refresh -out \\\"/home/atlantis/.atlantis/repos/md-terraform/aws-infrastructure/158/default/envs/deploy/apps/deploy_apps-default.tfplan\\\"\" in \"/home/atlantis/.atlantis/repos/md-terraform/aws-infrastructure/158/default/envs/deploy/apps\": signal: killed\nAcquiring state lock. This may take a few moments...\n","json":{"repo":"md-terraform/aws-infrastructure","pull":"158"},"stacktrace":"github.com/runatlantis/atlantis/server/events.RunAndEmitStats\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:78\ngithub.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandRunner).Plan\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:38\ngithub.com/runatlantis/atlantis/server/events.runProjectCmds\n\tgithub.com/runatlantis/atlantis/server/events/project_command_pool_executor.go:48\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).run\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:262\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:306\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:365"}
---
signal: killed","json":{"repo":"md-terraform/aws-infrastructure","pull":"158","duration":160.985073771},"stacktrace":"github.com/runatlantis/atlantis/server/core/runtime/models.(*ShellCommandRunner).RunCommandAsync.func1\n\tgithub.com/runatlantis/atlantis/server/core/runtime/models/shell_command_runner.go:158"}

locale plan executes correctly. The plan takes more than 2minutes. I was able to determine that this is not a problem of lack of resources, simply atlantis server sends a kill signal to the terraform plan running in the background. I am using the latest atlantis v0.27.2 image.

signal killed sounds like server event

I use ECS for orchestration. Logging into ECS task I see that I still have more than 50% memory available (Fargate profile with 4vCPU and 9GB memory).

what about your cloudwatch logs?

These information are from Cloudwatch logs from the task.

I mean other metrics like task health, count etc

ugh.. I see that in a high peak - before kill signal, the memory was used in 100%...

there you go

Yes, when I made it in my local environment, this plan has size 3.6M

size of file?

yes

the memory consumption is RAM so the calculation of the plan is what is important

I understand. In this image of atlantis the default version of terraform is v1.6.3

Not sure if it will help, but I had a similar issue but in my case, it was due to multiple plans getting stuck. People here guided me to check EFS throughput, which fixed the issue -> https://atlantis-community.slack.com/archives/C5MGGAV0C/p1706108211224259 For reference, in case useful