hey folks I m trying to deploy Atlantis onto an Azure Kubern Atlantis #atlantis-community

hey folks. I'm trying to deploy Atlantis onto an A...

Ryan Draga

04/24/2024, 6:49 PM

hey folks. I'm trying to deploy Atlantis onto an Azure Kubernetes cluster. I have all the credentials from the

values.yaml

stored in Key Vault secrets which are delivered to the cluster by way of External Secrets. For some reason though it seems like the resulting yaml inside the secret is either corrupt or I've formatted it wrong, and I can't tell which it is.

PePe Amengual

04/24/2024, 7:24 PM

the pod dies before any logs?

PePe Amengual

04/24/2024, 7:25 PM

can you enable debug in atlantis and see if you get any error?

Ryan Draga

04/24/2024, 7:49 PM

so I'm using

environmentSecrets

values.yaml

to specify environment variables. this is what that looks like:

Ryan Draga

04/24/2024, 7:49 PM

Copy code

environmentSecrets:
  - name: ATLANTIS_GH_USER
    secretKeyRef:
      name: atlantis
      key: atlantis-github-user
  - name: ATLANTIS_GH_TOKEN
    secretKeyRef:
      name: atlantis
      key: atlantis-github-token
  - name: ATLANTIS_GH_WEBHOOK_SECRET
    secretKeyRef:
      name: atlantis
      key: atlantis-github-webhook-secret
  - name: ATLANTIS_WEB_USERNAME
    secretKeyRef:
      name: atlantis
      key: atlantis-web-username
  - name: ATLANTIS_WEB_PASSWORD
    secretKeyRef:
      name: atlantis-basicauth
      key: password
  - name: ATLANTIS_API_SECRET
    secretKeyRef:
      name: atlantis
      key: atlantis-api-secret

Ryan Draga

04/24/2024, 7:50 PM

the secrets are all correct

Ryan Draga

04/24/2024, 7:51 PM

and yet

Ryan Draga

04/24/2024, 7:51 PM

this is what the pod logs say

Ryan Draga

04/24/2024, 7:51 PM

PePe Amengual

04/24/2024, 7:53 PM

this doesn’t have to be the name of the ENV variable too?

name: atlantis

Ryan Draga

04/24/2024, 7:53 PM

why would it need to be?

Ryan Draga

04/24/2024, 7:54 PM

it's the name of the kubernetes secret that contains the actual environment variable value

PePe Amengual

04/24/2024, 7:54 PM

I’m not a Kubernetes person

PePe Amengual

04/24/2024, 7:55 PM

the error you see if because atlantis is not seeing one of the ENV variables that is required

PePe Amengual

04/24/2024, 7:55 PM

even if the value is wrong

Ryan Draga

04/24/2024, 7:58 PM

except I'm supplying those via a kubernetes secret

Gabriel Martinez

04/25/2024, 10:47 AM

If you shell to the container and run

env

do you see these env variables defined there?

Gabriel Martinez

04/25/2024, 10:49 AM

If not, please share the yaml content of your secret, with the values redacted

Ryan Draga

04/25/2024, 12:58 PM

I can't shell to the container, it crashloops quickly enough that I'm not able to get a shell

Ryan Draga

04/25/2024, 1:00 PM

so this is the yaml in

values.yaml

Ryan Draga

04/25/2024, 1:00 PM

Copy code

environmentSecrets:
  - name: ATLANTIS_GH_USER
    secretKeyRef:
      name: atlantis
      key: atlantis-github-user
  - name: ATLANTIS_GH_TOKEN
    secretKeyRef:
      name: atlantis
      key: atlantis-github-token
  - name: ATLANTIS_GH_WEBHOOK_SECRET
    secretKeyRef:
      name: atlantis
      key: atlantis-github-webhook-secret
  - name: ATLANTIS_WEB_USERNAME
    secretKeyRef:
      name: atlantis
      key: atlantis-web-username
  - name: ATLANTIS_WEB_PASSWORD
    secretKeyRef:
      name: atlantis-basicauth
      key: password
  - name: ATLANTIS_API_SECRET
    secretKeyRef:
      name: atlantis
      key: atlantis-api-secret

Ryan Draga

04/25/2024, 1:01 PM

and this is the redacted secret data

Ryan Draga

04/25/2024, 1:02 PM

Copy code

apiVersion: v1
data:
  atlantis-api-secret: [REDACTED]
  atlantis-gitconfig: [REDACTED]
  atlantis-github-token: [REDACTED]
  atlantis-github-user: [REDACTED]
  atlantis-github-webhook-secret: [REDACTED]
  atlantis-web-password: [REDACTED]
  atlantis-web-username: [REDACTED]
immutable: false
kind: Secret
metadata:
  annotations:
    <http://kubectl.kubernetes.io/last-applied-configuration|kubectl.kubernetes.io/last-applied-configuration>: |
      {"apiVersion":"<http://external-secrets.io/v1beta1|external-secrets.io/v1beta1>","kind":"ExternalSecret","metadata":{"annotations":{},"name":"atlantis","namespace":"atlantis"},"spec":{"data":[{"remoteRef":{"key":"secret/atlantis-github-user"},"secretKey":"atlantis-github-user"},{"remoteRef":{"key":"secret/atlantis-github-token"},"secretKey":"atlantis-github-token"},{"remoteRef":{"key":"secret/atlantis-github-webhook-secret"},"secretKey":"atlantis-github-webhook-secret"},{"remoteRef":{"key":"secret/atlantis-api-secret"},"secretKey":"atlantis-api-secret"},{"remoteRef":{"key":"secret/atlantis-web-username"},"secretKey":"atlantis-web-username"},{"remoteRef":{"key":"secret/atlantis-web-password"},"secretKey":"atlantis-web-password"},{"remoteRef":{"key":"secret/atlantis-gitconfig"},"secretKey":"atlantis-gitconfig"}],"refreshInterval":"1m","secretStoreRef":{"kind":"ClusterSecretStore","name":"mi-infra-worker-extsec"},"target":{"name":"atlantis"}}}
    <http://reconcile.external-secrets.io/data-hash|reconcile.external-secrets.io/data-hash>: 32d77666fd0e3c172f24aa68ac294aae
  creationTimestamp: "2024-04-24T19:29:09Z"
  labels:
    <http://reconcile.external-secrets.io/created-by|reconcile.external-secrets.io/created-by>: 19ff3c0e100bcc589760c06d99fdccf6
  name: atlantis
  namespace: atlantis
  ownerReferences:
  - apiVersion: <http://external-secrets.io/v1beta1|external-secrets.io/v1beta1>
    blockOwnerDeletion: true
    controller: true
    kind: ExternalSecret
    name: atlantis
    uid: 7558f3dd-1421-47b1-8843-a5b139e50819
  resourceVersion: "319569919"
  uid: 27d76e5c-151e-4731-9522-d8d7c6ed3d98
type: Opaque

Gabriel Martinez

04/25/2024, 1:04 PM

The redacted part is in base64 right?

Ryan Draga

04/25/2024, 1:05 PM

yup

Gabriel Martinez

04/25/2024, 1:07 PM

Looking at all this it seems ok. Can you confirm on the pod spec that the env configs for the secrets are there?

Ryan Draga

04/25/2024, 1:08 PM

huh. Welp, there's the problem.

Ryan Draga

04/25/2024, 1:09 PM

Copy code

Environment:
      ATLANTIS_DEFAULT_TF_VERSION:  1.7.4
      ATLANTIS_LOG_LEVEL:           info
      ATLANTIS_SSL_CERT_FILE:       /etc/tls/tls.crt
      ATLANTIS_SSL_KEY_FILE:        /etc/tls/tls.key
      ATLANTIS_DATA_DIR:            /atlantis-data
      ATLANTIS_REPO_ALLOWLIST:      [REDACTED]
      ATLANTIS_PORT:                4141
      ATLANTIS_ATLANTIS_URL:        [REDACTED]

Ryan Draga

04/25/2024, 1:10 PM

but if I'm using

environmentSecrets

, why isn't the chart pulling it in?

Gabriel Martinez

04/25/2024, 1:10 PM

Checking the logic now

Gabriel Martinez

04/25/2024, 1:11 PM

It should work https://github.com/runatlantis/helm-charts/blob/f51501b30461220cf6af5647263a0bbf75fc5fe8/charts/atlantis/templates/statefulset.yaml#L237

Gabriel Martinez

04/25/2024, 1:12 PM

Are you on the latest chart version?

Gabriel Martinez

04/25/2024, 1:14 PM

I would also run a helm template locally to make sure the secrets are part of the statefulset on the env

Ryan Draga

04/25/2024, 1:15 PM

Copy code

apiVersion: v1
appVersion: v0.27.2
description: A Helm chart for Atlantis <https://www.runatlantis.io>
home: <https://www.runatlantis.io>
icon: <https://www.runatlantis.io/hero.png>
keywords:
- terraform
maintainers:
- name: lkysow
- name: jamengual
- name: chenrui333
- name: nitrocode
- name: genpage
- name: gmartinez-sisti
name: atlantis
sources:
- <https://github.com/runatlantis/atlantis>
version: 4.25.0

Gabriel Martinez

04/25/2024, 1:15 PM

Also confirm that the statefulset has those configs. If they were added after with the pod already failling, sometimes they are not updated on the pod.

Ryan Draga

04/25/2024, 1:16 PM

yeah so the statefulset doesn't have them

Ryan Draga

04/25/2024, 1:16 PM

should I maybe just do a helm delete and do a fresh install?

Gabriel Martinez

04/25/2024, 1:17 PM

Try the local helm template to see if they show up

Ryan Draga

04/25/2024, 1:18 PM

looks like they don't

Gabriel Martinez

04/25/2024, 1:18 PM

Let me try it, we might have a bug 🐛

👍 1

❌ 1

Gabriel Martinez

04/25/2024, 1:24 PM

it worked here

Copy code

→ cat test.yaml
environmentSecrets:
  - name: ATLANTIS_GH_USER
    secretKeyRef:
      name: atlantis
      key: atlantis-github-user
  - name: ATLANTIS_GH_TOKEN
    secretKeyRef:
      name: atlantis
      key: atlantis-github-token
  - name: ATLANTIS_GH_WEBHOOK_SECRET
    secretKeyRef:
      name: atlantis
      key: atlantis-github-webhook-secret
  - name: ATLANTIS_WEB_USERNAME
    secretKeyRef:
      name: atlantis
      key: atlantis-web-username
  - name: ATLANTIS_WEB_PASSWORD
    secretKeyRef:
      name: atlantis-basicauth
      key: password
  - name: ATLANTIS_API_SECRET
    secretKeyRef:
      name: atlantis
      key: atlantis-api-secret

Copy code

→ helm template atlantis . -f test.yaml -s templates/statefulset.yaml | yq .spec.template.spec.containers[0].env
- name: ATLANTIS_GH_USER
  valueFrom:
    secretKeyRef:
      name: atlantis
      key: atlantis-github-user
- name: ATLANTIS_GH_TOKEN
  valueFrom:
    secretKeyRef:
      name: atlantis
      key: atlantis-github-token
- name: ATLANTIS_GH_WEBHOOK_SECRET
  valueFrom:
    secretKeyRef:
      name: atlantis
      key: atlantis-github-webhook-secret
- name: ATLANTIS_WEB_USERNAME
  valueFrom:
    secretKeyRef:
      name: atlantis
      key: atlantis-web-username
- name: ATLANTIS_WEB_PASSWORD
  valueFrom:
    secretKeyRef:
      name: atlantis-basicauth
      key: password
- name: ATLANTIS_API_SECRET
  valueFrom:
    secretKeyRef:
      name: atlantis
      key: atlantis-api-secret
- name: ATLANTIS_DATA_DIR
  value: /atlantis-data
- name: ATLANTIS_REPO_ALLOWLIST
  value: "<replace-me>"
- name: ATLANTIS_PORT
  value: "4141"
- name: ATLANTIS_ATLANTIS_URL
  value: http://

looks like your values file is not being correctly sourced.

Ryan Draga

04/25/2024, 1:24 PM

any idea why?

Gabriel Martinez

04/25/2024, 1:25 PM

how are you deploying it?

Ryan Draga

04/25/2024, 1:25 PM

helm install atlantis runatlantis/atlantis -f values.yaml

Ryan Draga

04/25/2024, 1:27 PM

oh wait hold on

Ryan Draga

04/25/2024, 1:27 PM

I think I might've found the problem. Let me try to redeploy

💡 1

Ryan Draga

04/25/2024, 1:42 PM

okay I think that did it. Chalk it up to ID-10T error and letting Copilot run amok.

Ryan Draga

04/25/2024, 1:43 PM

I had a duplicate

environmentSecrets

block, and a couple of autoco-wrecked-ed secret names

Ryan Draga

04/25/2024, 1:43 PM

the pod is up at least, but it's not stable yet

Gabriel Martinez

04/25/2024, 1:47 PM

Almost there 😊

Ryan Draga

04/25/2024, 2:38 PM

okay, so how would I configure the helm chart to use

ClusterIP

instead of

NodePort

for the

service

block?

Ryan Draga

04/25/2024, 2:38 PM

that's the last piece

Ryan Draga

04/25/2024, 2:38 PM

everything else is working now

Gabriel Martinez

04/25/2024, 11:08 PM

Here https://github.com/runatlantis/helm-charts/blob/f51501b30461220cf6af5647263a0bbf75fc5fe8/charts/atlantis/templates/service.yaml#L24?

Open in Slack

Previous Next