https://cloudposse.com logo
Join Slack
Powered by
I now suddenly have issues with Atmos Terraform Dr...
# github-actions
m
I now suddenly have issues with Atmos Terraform Drift Remediation - it fails on authentication to Kubernetes - 
getting credentials: exec: executable aws not found
. We changed atmos version from 1.80 to 1.83, though. Is it not supported?
They're using exactly the same workflow file:
Copy code
jobs:
  remediate-drift:
    if: github.event.action == 'labeled' && contains(github.event.issue.labels.*.name, 'apply')
    name: Remediate Drift
    runs-on: ["self-hosted", "terraform"]
    steps:
      - name: Remediate Drift
        uses: cloudposse/github-action-atmos-terraform-drift-remediation@v2
        with:
          issue-number: ${{ github.event.issue.number }}
          action: remediate
          atmos-version: ${{ vars.ATMOS_VERSION }}
          atmos-config-path: ${{ vars.ATMOS_CONFIG_PATH }}
Last week it was working fine, now it can't find aws -_-
Copy code
2024-07-09T04:57:58.8486056Z kubernetes_manifest.ec2_node_class["default"]: Refreshing state...
2024-07-09T04:57:58.8486559Z 
2024-07-09T04:57:58.8487036Z Planning failed. Terraform encountered an error while generating this plan.
2024-07-09T04:57:58.8487606Z 
2024-07-09T04:57:58.8487614Z 
2024-07-09T04:57:58.8487996Z Error: Failed to determine GroupVersionResource for manifest
2024-07-09T04:57:58.8488459Z 
2024-07-09T04:57:58.8488741Z   with kubernetes_manifest.ec2_node_class["default"],
2024-07-09T04:57:58.8489707Z   on <http://ec2-node-class.tf|ec2-node-class.tf> line 26, in resource "kubernetes_manifest" "ec2_node_class":
2024-07-09T04:57:58.8490581Z   26: resource "kubernetes_manifest" "ec2_node_class" {
2024-07-09T04:57:58.8490982Z 
2024-07-09T04:57:58.8491110Z Get
2024-07-09T04:57:58.8492081Z "<https://CA639ED0C99EB1C5F6EF28CFA19B151B.gr7.ap-southeast-2.eks.amazonaws.com/api?timeout=32s>":
2024-07-09T04:57:58.8493149Z getting credentials: exec: executable aws not found
2024-07-09T04:57:58.8493554Z 
2024-07-09T04:57:58.8494120Z It looks like you are trying to use a client-go credential plugin that is not
2024-07-09T04:57:58.8494812Z installed.
2024-07-09T04:57:58.8495016Z 
2024-07-09T04:57:58.8495477Z To learn more about this feature, consult the documentation available at:
2024-07-09T04:57:58.8496772Z       <https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins>
2024-07-09T04:57:58.8497665Z exit status 1
2024-07-09T04:57:58.8497877Z 
2024-07-09T04:57:58.8498069Z Terraform apply failed
2024-07-09T04:57:58.8510018Z ##[error]Process completed with exit code 1.
Copy code
2024-07-09T04:56:36.7059673Z Current runner version: '2.317.0'
2024-07-09T04:56:36.7065783Z Runner name: 'platform-runner-f7gcw-4z6vs'
2024-07-09T04:56:36.7106854Z Runner group name: 'Default'
2024-07-09T04:56:36.7107832Z Machine name: 'platform-runner-f7gcw-4z6vs'
2024-07-09T04:56:36.7110563Z ##[group]GITHUB_TOKEN Permissions
2024-07-09T04:56:36.7112679Z Contents: read
2024-07-09T04:56:36.7113113Z Issues: write
2024-07-09T04:56:36.7113600Z Metadata: read
2024-07-09T04:56:36.7114039Z ##[endgroup]
2024-07-09T04:56:36.7116415Z Secret source: Actions
2024-07-09T04:56:36.7117113Z Prepare workflow directory
2024-07-09T04:56:36.7904054Z Prepare all required actions
2024-07-09T04:56:36.8080857Z Getting action download info
2024-07-09T04:56:37.1516171Z Download action repository 'cloudposse/github-action-atmos-terraform-drift-remediation@v2' (SHA:65aabe2c0189b45b603be815dae1d839d23bd90a)
2024-07-09T04:56:38.0288473Z Getting action download info
2024-07-09T04:56:38.3326806Z Download action repository 'actions/github-script@v6' (SHA:d7906e4ad0b1822421a7e6a35d5ca353c962f410)
2024-07-09T04:56:39.0027454Z Download action repository 'cloudposse/github-action-atmos-terraform-apply@v2' (SHA:3ec18a48ec4acec3c2760036343603171f17938b)
2024-07-09T04:56:39.7414471Z Getting action download info
2024-07-09T04:56:40.1422584Z Download action repository 'actions/checkout@v4' (SHA:692973e3d937129bcbf40652eb9f2f61becf3332)
2024-07-09T04:56:40.8160936Z Download action repository 'cloudposse/github-action-setup-atmos@v2' (SHA:7137afeb830d41684b464f9677a13566d4bcb550)
2024-07-09T04:56:41.4994630Z Download action repository 'hashicorp/setup-terraform@v3' (SHA:651471c36a6092792c552e8b1bef71e592b462d8)
2024-07-09T04:56:42.1969633Z Download action repository 'actions/setup-node@v4' (SHA:60edb5dd545a775178f52524783378180af0d1f8)
2024-07-09T04:56:42.9492869Z Download action repository 'cloudposse/github-action-atmos-get-setting@v1' (SHA:644b6ec2d51b24455905f7158804a41962c6d6b2)
2024-07-09T04:56:43.6624238Z Download action repository 'jaxxstorm/action-install-gh-release@v1.11.0' (SHA:71d17cb091aa850acb2a1a4cf87258d183eb941b)
2024-07-09T04:56:47.1942062Z Download action repository 'aws-actions/configure-aws-credentials@v4.0.2' (SHA:e3dd6a429d7300a6a4c196c26e071d42e0343502)
2024-07-09T04:56:47.8832345Z Download action repository 'cloudposse/github-action-terraform-plan-storage@v1' (SHA:0b13c5198170a1636557f008e1737c662ed2eb86)
2024-07-09T04:56:48.6733651Z Download action repository 'infracost/actions@v3' (SHA:d5dd739fb5eae81f31bf09fea5c290186a91964a)
2024-07-09T04:56:49.5715885Z Complete job name: Remediate Drift
2024-07-09T04:56:49.6204174Z A job started hook has been configured by the self-hosted runner administrator
2024-07-09T04:56:49.6356034Z ##[group]Run '/etc/arc/hooks/job-started.sh'
2024-07-09T04:56:49.6366017Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-09T04:56:49.6366579Z ##[endgroup]
2024-07-09T04:56:49.6646354Z [0;37m2024-07-09 04:56:49.656  DEBUG --- Running ARC Job Started Hooks[0m
2024-07-09T04:56:49.6648104Z [0;37m2024-07-09 04:56:49.657  DEBUG --- Running hook: /etc/arc/hooks/job-started.d/update-status[0m
2024-07-09T04:56:49.7212378Z ##[group]Run cloudposse/github-action-atmos-terraform-drift-remediation@v2
2024-07-09T04:56:49.7213119Z with:
2024-07-09T04:56:49.7213397Z   issue-number: 128
2024-07-09T04:56:49.7213714Z   action: remediate
2024-07-09T04:56:49.7214023Z   atmos-version: 1.83.1
2024-07-09T04:56:49.7214440Z   atmos-config-path: ./rootfs/usr/local/etc/atmos/
2024-07-09T04:56:49.7214922Z   debug: false
2024-07-09T04:56:49.7215447Z   token: ***
2024-07-09T04:56:49.7215734Z ##[endgroup]
e
Are you able to downgrade in the meantime?
m
the lookups would stop working %) and I'm not sure it would help, but I can try
It didn't help, it still complains about missing 
aws
executable -_- Last week's runs weren't touching kubernetes, so I guess that's why it worked
a
The root cause of the issue is that 
.github/workflows/atmos-terraform-drift-remediation.yaml
misses 
- uses: unfor19/install-aws-cli-action@v1
which exists both in 
atmos-terraform-plan-matrix.yaml
and 
atmos-terraform-apply-matrix.yaml
🤦‍♂️ 1
👍 1
e
@Igor Rodionov ☝️
i
@Marat Bakeev @Alexander Matveev is right. You need to install aws cli like we had in plan and apply workflows. The case is that the Kubernetes component uses AWS CLI in 
provider,tf
to auth on the K8s API. This is an edge case. In theory, there is a wide range of third-party APIs that need different CLIs to auth a service. For k8s, I will do the fix in our workflows; that’s ridiculous. We fixed it in the plan and apply, but missed it in drift remediation. Thanks for pointing out.
@Marat Bakeev @Erik Osterman (Cloud Posse) Here is the PR to fix our refarch https://github.com/cloudposse/refarch-scaffold/pull/678
e
Aha, yes, in the older EKS component this is true. The latest release does not. @dan I think this is a painless upgrade. Cc @Jeremy G (Cloud Posse)
j
Not exactly painless upgrade, as it requires multiple manual steps, but not terrible, either. Takes about 20 minutes per cluster.
e
@Matt Calhoun ☝️
Open in Slack
PreviousNext
Powered by