https://www.runatlantis.io/ logo
Join Slack
Powered by
This message was deleted.
# atlantis-community
s
This message was deleted.
āž• 1
r
@PePe Amengual any idea?
@Dylan Page
r
p
the default workflow could potentially run in another workspace but I have no tested it
you will need to try to define a custom workflow named default in the server side config
and see if you can override that way
if not then you need to create a yaml file
remember you can create yaml files dynamically using pre_workflow_hooks so if you have hundreds of repos you can use that to create that dynamically without having to push it to every repo
šŸ‘€ 1
please read de docs, there are details on how to do that too
r
can you give us a reference for generating an Atlantis yaml? this docs https://www.runatlantis.io/docs/pre-workflow-hooks.html#dynamic-repo-config-generation were not enough for us to understand how to generate an atlantis.yaml file
p
you need to generate a file, the atlantis.yaml file
you can use a script or do echo ā€œ{atlantis.yaml_content}ā€ > atlantis.yaml
first you need to try this in one repo, add an atlantis.yaml file, see if you can override the default workspace , then you worry on how you do that on the pre_workflow
you need to create the atlantis.yaml first so after the testing you are going to do in that repo you can use that as the content for the output in the pre_workflow_hook
r
do you have an example script? and how do we even uploads such script to atlantis?
p
you will not upload the script to atlantis
you will do something like :
r
yeah okay we will run a bash script that will generate a yaml file? do you have a working example ?
p
Copy code
repos:
    - id: /.*/
      pre_workflow_hooks:
        - run: echo "version: 3
projects:
- dir: project1
  autoplan:
    enabled: false" > atlantis.yml
          description: Generating config files
you will need to make sure the echo command formats the file correctly
so the idea will be to do this in a one liner
r
we actually need autoplan enabled šŸ˜„
p
obviously that is an example file
r
yes yes šŸ˜„
p
that is why I said, you need to create and test your yaml file first
then worry about how to do it in pre_workflow_hook
e
I think that now I understand how the pre_workflow_hook work, and it sound just like what we need. Thanks @PePe Amengual! I will try to implement it this way šŸ‘
p
no problem
e
Hey, @PePe Amengual I configured the pre_workflow_hook and it seems like it successfully generates an atlantis yaml And the needed flow runs as autoplan (great so far)
But now for some reason the init is stuck
I saw an issue on this problem but it's not so clear what to do. https://github.com/runatlantis/atlantis/issues/1081
Locally, the init works just fine, as well as in or github actions automation. Do you know a possible reason why it's stuck? @PePe Amengual Thanks in advance šŸ™‚
p
mmmmm try without input =false , maybe wants to ask about the workspace select or backend?
you can try --reconfigure or one of the new options to clean the workspace every time
e
Hey @PePe Amengual how are you? I'm back with another question šŸ™‚ I've set the init step with only the "-reconfigure" flag, but it seems that Atlantis uses "-input=false" , "-upgrade" by default or something. is it true? Or something is just stuck there?
p
you can customize the init command on the workflow
just use input=true
e
I ended up running the command like this
I tried all sorts of flag combinations, but it doesn't work - still gets stuck on the init stage
also tried Terraform versions 1.5.5, 1.5.7
@PePe Amengual I also get this error sometimes when I try to see the details of the run. Where is this token coming from?
p
did you enabled debug in atlantis too?
the invalid key is not that important
can you run init manually on the atlantis instance?
and have you checked memory utilization?
one thing I will try is to start an instance, assign a instance role similar to eks, clone the repo and run init and see what happens
I know is not ideal but this looks like something specific to the particular project
e
@PePe Amengual Thanks for the reply šŸ™‚ I tried running the flow manually in the atlantis pod on EKS: ā€¢ fresh installed Atlantis with helm ā€¢ exec to the pod ā€¢ clone the code from GH ā€¢ init in the tf directory - successful ā€¢ plan in tf directory - also successful but with a few missing permissions to cloudwatch ā€¢ everithing went super quick (memory and cpu seem okay) I will try fix the missing permissions for the service role, but it seems it doesn't interfere with the init process
p
that is interesting , save version of terraform and such?
same.env variables for everything ?
e
Yes, I also tried to return to the most basic configs: init and plan
But the outcome is the same - it's stuck on the state list stage from s3 (200)
But it doesn't seem to get to the final stage of the init
The last screenshot is from the pod run (terraform plan -var-file=stg=tfvars)
This is the pod log when I trigger the event PR comment: atlantis plan -p atlantis-plan-stg-eu
p
are you trying on the same PR?
or you created a new PR?
I wonder if you close and open the PR and then run plan what would happen
e
I actually opened many PRs with multiple atlantis installations from scratch, tried reinstalling the namespace too and even create a new branch for the testing with PR
I'm suspecting the permissions of the proccess are different from the permissions that the service account has. That's my be the reason the when I'm trying to run terraform init from the pod, it finishes successfully
But when the PR process is executed it can perform sertain actions like s3:getObjects
OR! Iver notices that in the pod the directories are: atlantis-data/repos/<gh_company_name>/<repo_name>/<PR_num>/default and stg
so the atlantis.yaml is creted under default and possibly the init is running in default too.
p
you know, I had this issue once (on ECS) and what happened was that I was getting 403 errors
and those were not exposed int he log but I saw them in TF logs
can you set TF_LOG=Debug and run this again?
I wonder if, for whatever reason, for example, you have access to the bucket but not to s3:GetObject?
@James Burriss same as here
šŸ‘€ 1
e
The log is on debug And the problem is exactly this I think!
The workflow lists the objects successfully and the next step is to getObjects but it's stuck
p
time to give explicit access to that bucket and try s3:* for test, or maybe check if the policy has a typo ?
e
When I try to execute the commands from the atlantis pod (terraform init, plan), they all finished successfully. So it's like the pods execution policies are fine but the atlantis process struggles with it. Can there be a difference? šŸ¤”
p
it literally runs the terraform binary on the container image that is all
and it add the -input-false and such to the parameters
you have the same problem with older TF versions?
e
I tried also terraform version 1.3 and the result was the same. Can you suggest a specific version to use?
p
any 1.4
e
Hey, quick question: Is it okay that I have dynamoDB table configured in my s3 backend config? Or can it be the reason that Atlantis lock not created?
d
Atlantis locks are completely separate from Terraform locks
this 1
e
Hey @PePe Amengual @Dylan Page When I create the atlantis.yaml file in the repository. I can see it's being clone properly and placed inside each project folder. So respectively, each workflow runs properly. But when I replace the atlantis.yaml from the repo, with pre_orkflow_hooks config, to generate the atlantis yaml 'on the go', for each repo - the atlantis.yaml file is created on the same level as the project folders and probably not being read. I assume that because when I use the same config from the repo level atlantis.yaml I see that the workspace is now 'default' instead of 'stg'
This is what I get with a repo level atlantis.yaml
And this is the configuration of the pre_workflow_hooks (that creates the atlantis.yaml in the pod, but outside the workspace folders)
p
ahhhhhh interesting
You should not have any atlantis.yaml committed to the repo
the generated atlantis.yaml needs to be at the root
that is where atlantis reads it from
how is it going @Evgeni Sakhatsky
e
Hey, sorry for not replying for the last few days. I tried to investigate the issue with AWS support, and we established that the Atlantis role successfully assumes the Management role from the management account and can list the state bucket and it's objects. Then we test the "Terraform init" command from the pod on EKS and the init finished successfully too. Only when the command runs from Atlantis, the process gets stuck somewhere around the s3:GetObjects action, and the message is not very informative. I would be so glad if you can take another look šŸ™‚
p
is it possible for you to setup atlantis in ECS or a instance ,manually and test it out there?
do you have a sidecar or any other EKS special security service that could block calls to apis?
e
I don't think that there are any special security restrictions. The only thing is maybe the bucket ownership. I tried to store the terraform state in 2 different buckets. 1 that I created and one that was created previously by someone else. Both buckets apparently have the same configuration, but the one I made, works and the premade don't. (I really need to use the premade one.) I will try to install atlantis on an ECS. I just wonder, what can be the conclusion in case that it will work on ECS
p
wait, you said your bucket worked?
and the existing one did not?
e
I did. Yes
p
did you compare NACLs and policies?
and Region? ( region it was created)
e
Yes, compared it with AWS support too, and investigated the bucket logs. They say that there's no difference
p
Buckets are global namespace but authentication/replication happens somewhere else
well there is some difference otherwise you will not have this problem
but running on ECS is to make sure there is no EKS related permissions issues that could cause this
šŸ’” 1
e
Yup, that's what I think too
p
if in ECS you can use the premade bucket then we will know
e
On the paper - Both buckets are from the same region, have the same properties and Permissions
But I'll test it with ECS too
Hey there! I'm back on the task and I solved the permissions issue This was solved by using a newer version of the helm chart --version 4.18.0 And removing the fixed terraform version from the workflow in the values.
I have another question tho - I'm trying to use variables in the config and they are not being read. $PLANFILE returns a blank space Same as other variables (WORKSPACE etc.)
Hey @PePe Amengual Can I use external secrets instead of hardcoded tokens in the Github app config?
p
where?
if Atlantis is running inside aws for example , you can get secrets from Parameter store or secret manager and expose them to the container
e
Yes, I mean, if the secret is already in the cluster (from aws) I can just refer the secret from the values file?
And please please please @PePe Amengual One more super important thing that I can't find in the docs. Is there a way to change the workflow based on the git event? I mean that if the PR is set to branch master then run a different flow from the main one (or Don't run at all) Can I do that?
p
you can set workflows base on the branch name
there is a branch parameter
and if the secret is exposed to the container as an ENV variable for example you should be able to read it in your workflow
e
Thanks!
Open in Slack
PreviousNext
Powered by