This message was deleted.
# atlantis-community
s
This message was deleted.
j
We self host gitlab, and have tons of problems with atlantis. The largest being, anything after the FIRST
external pipeline
no longer shows up under the MR itself, so you have to go into CICD/Pipelines and find the job to see what its doing
Not to mention Gitlab doesnt have a webhook for when you make a Draft MR ready šŸ˜•
r
So possibly the changes I suggest on the Issue, it should start working as expected.
I plan to build one Atlantis image for us with the change and perform several tests with pipelines currently broken
but the hardest part may be pushing the change upstream
m
we're on self hosted gitlab v15.11 we have zero issues.
j
So we were on 15.x, now we are on 16.x same issue though, and thats in 2 different Gitlab Environments
r
Thank you all for the confirmation that this affects more people. Can I kindly ask you to vote on that issue so that it can be assessed as soon as possible?
Also, @Mustafa Mujahid can you confirm that you have custom pipelines using gitlab-ci file on projects with atlantis pipelines and you don't see the split pipelines behavior? The issue is only visible with Atlantis pipelines on projects that also have other pipelines
m
yes, we have a AtlantisGatekeeper gitlabci that makes sure that the MR cannot be merged until a successful
atlantis apply
has not been run
how does it look like, this split pipeline behavior ?
j
@Mustafa Mujahid 1. How the hell did you name your atlantis pipeline? I would love to add this same thing to ours 2. The
split
pipeline for us behaves like this. First push, the
external
pipeline shows under the MR's pipelines. Every push after that creates a new pipeline with
external
that is NOT showing up under the MR, you have to go into CICD/Pipelines and find it
m
name your atlantis pipeline?
what are you referring to ? what do yours look like? regarding the split, I guess i haven't had an MR recently with multiple pushes, but honestly i never find myself looking at the actual CIs running, just wait for the plan to pop-up. so perhaps i need to to have a look specifically to push multiple times and explicitly look at the CI.
j
your
Atlantis Gatekeep
How are you doing, so it understands to wait for
external
Atlantis takes so long to run in our environment, our developers always thinks is broken/stuck
so the job not showing up in the UI just makes it worse
m
yes, we make sure that the Gatekeeper always fails unless it detects the
if name == "atlantis/apply" and status == "success":
. we do this by using a post hook to scrape the external jobs and look for this in
.gitlab-ci.yaml
. As long as the gatekeeper fails gitlab does not allow merging based on the setting in gitlab.
j
Ahh got it, i was really hoping there was a way to just refer to an external pipeline without doing scraping
m
unfortunately not
also you do have šŸ‘€ emoji now, that informs people that atlantis is working
j
Yes, that does work for us. But once a plan is taking 20+ minutes, they assume its broken, and start trying to push more and more
or if Atlantis dies mid plan, nothing happens, it never recovers
m
hmmm, yeah, if it stuck then yes, i see the predicament šŸ¤”
j
We run in EKS, so if a node scales or w/e, and atlantis is moved
m
I will try it out, and let you know here if i have the same behaviour
well I would say that's really not the way to go with atlantis, the pod has to be completely stable.
j
I would say thats more of an atlantis problem,
not being designed to work in a modern env
same with atlantis doing all its work in a single pod, instead of spinning up a new pod for each MR
Atlantis is the best out there, but its def. lacking in working in a modern setup
m
it's a stateful workload, you can't run stateful workloads on ephemeral infra.
j
thats not a fair statement
if i have 9 MR's
TF handles the locking already
they is no reason all that work needs to run inside teh same pod
the same pod running the web UI, is running 9 diff. project plans. no reason for that
r
how does it look like, this split pipeline behavior ?
Similar to what @Justin S mentioned, when we upgraded Atlantis we started to have the pipeline of Atlantis decoupled from the main pipelines, meaning it would not appear as your screenshots. Examples below • Expected behavior (first pic) with single pipeline • Current behavior (second pic) with dual pipelines (probably duplicate is not the best wording) Additionally in some cases it would only start if we activated temporarily the
Enable merged results pipelines
option on Merge Requests. Reverting the option would not revert to single pipeline and would forever be with dual pipelines
j
ahh @ @Ricardo Silveira so ours looks like yours. except the
right side
may end up being duplicated n+1 times
šŸ™Œ 1
m
I really don't want to argue about this, as i don;t really care how you're running your stuff šŸ˜„ , it works completely fine for us, running even >200 projects per MR
r
I see, even more weird than ours then. The issue I created also states that removing the added code previously reverts the behavior to be ok, but I really don't understand what the original author was solving with it šŸ˜• That is why I wanted to first understand his problem (or any other person that had similar issue to him) and only then propose changes
m
@Ricardo Silveira I will test this out, i haven't had a loko at the CIs, i'll have a look
šŸ™Œ 1
r
So probably I should've started by pinging @Michel Z. Santello (I think I got the right person) to understand what the original issue was šŸ™‚
m
hey, so I tried it out. I did multiple pushes to this MR, and the jobs showed up in the CI
Gitlab
v15.11
Atlantis
v0.24.4
I will continue observing this, perhaps it happens on specific instances.
r
Thanks for giving it a try @Mustafa Mujahid šŸ™
šŸ‘ 1