Looks like our GHA are failing with: `Error savin...
# github-actions
j
Looks like our GHA are failing with:
Error saving credentials: error storing credentials - err: exit status 1, out: not implemented
I see multiple issues discussing this (https://github.com/docker/login-action/issues/515), but none of them have an outright solution. Has anyone encountered this? We're using https://github.com/cloudposse/github-action-docker-build-push for our build/push
e
Just a random guess, host machine is out of disk space
j
doesn't seem to be the case
@Jeremy G (Cloud Posse) does this ring a bell? I saw you commenting back in 2020 on this issue: https://github.com/awslabs/amazon-ecr-credential-helper/issues/102
e
@Igor Rodionov ☝️
j
the weird thing is we've made no changes to the set up. this is all the same as it was before and it just fails
of course it is right before the very moment we need it 😬 lol
i
@johncblandii know this case
sec
try this
Copy code
tee -a /github-runners/docker-config.json <<EOF
{
  "credHelpers": {
		"ecr.us-east-1.amazonaws.com": "ecr-login",
		"ecr.us-east-2.amazonaws.com": "ecr-login",
		"ecr.us-west-1.amazonaws.com": "ecr-login",
		"ecr.us-west-2.amazonaws.com": "ecr-login",
		"ecr.af-south-1.amazonaws.com": "ecr-login",
		"ecr.ap-east-1.amazonaws.com": "ecr-login",
		"ecr.ap-southeast-3.amazonaws.com": "ecr-login",
		"ecr.ap-south-1.amazonaws.com": "ecr-login",
		"ecr.ap-northeast-3.amazonaws.com": "ecr-login",
		"ecr.ap-northeast-2.amazonaws.com": "ecr-login",
		"ecr.ap-southeast-1.amazonaws.com": "ecr-login",
		"ecr.ap-southeast-2.amazonaws.com": "ecr-login",
		"ecr.ap-northeast-1.amazonaws.com": "ecr-login",
		"ecr.ca-central-1.amazonaws.com": "ecr-login",
		"ecr.eu-central-1.amazonaws.com": "ecr-login",
		"ecr.eu-west-1.amazonaws.com": "ecr-login",
		"ecr.eu-west-2.amazonaws.com": "ecr-login",
		"ecr.eu-south-1.amazonaws.com": "ecr-login",
		"ecr.eu-west-3.amazonaws.com": "ecr-login",
		"ecr.eu-north-1.amazonaws.com": "ecr-login",
		"ecr.me-south-1.amazonaws.com": "ecr-login",
		"ecr.sa-east-1.amazonaws.com": "ecr-login"
  }
}
j
add that to the user data?
i
yea
j
we have this in there:
Copy code
mkdir -p /root/.docker
echo '{ "credsStore": "ecr-login" }' >/root/.docker/config.json
mkdir -p /home/ec2-user/.docker
echo '{ "credsStore": "ecr-login" }' >/home/ec2-user/.docker/config.json
should the path match?
i
no
j
ok, gotcha
i
my path is from our module
that docker-config.json should be like that
j
ok, have to step away, but i'll run this in a few
j
For GHA, did you give the job write permission?
Copy code
permissions:
  contents: write
IIRC, in a GitHub Action using official Docker actions, you do not need to have any credential helpers or
.docker/config.json
, just use the
docker/login-action
action (which is done for you by cloudposse/github-action-docker-build-push).
j
yes, it has that
and yes, that's the action we're using. it just started failing out of nowhere
j
There was an outage. Is it still failing?
j
yes. it started last week
was still failing today
is failing now cuz of the
tee
above, but i'm pushing a fix
j
I don't know, then. I haven't seen new problems crop up recently.
j
yeah, reverted and it is still failing
Confirmed this was in place
Copy code
permissions:
  contents: write
  id-token: write
cloudposse/github-action-docker-build-push@1.13.0
confirmed adding
credHelpers
didn't help. I created a reduced test case and
docker/login-action
continues to fail no matter what I do with the same error:
Error saving credentials: error storing credentials - err: exit status 1, out:
not implemented
I'll see what else I can find out. Definitely the worst timing to hit this bug
just adding here for context as I go. testing on
ubuntu-latest
works perfect fine on GHA. testing on a linux2 bastion and I could easily replicate this problem
I changed this section and everything worked on the bastion. Testing this on the runners as well. So far it looks promising 🤞🏽
Copy code
mkdir -p /root/.docker
echo '{}' >/root/.docker/config.json
mkdir -p /home/ec2-user/.docker
echo '{}' >/home/ec2-user/.docker/config.json
🤞 1
confirmed we're back green
🎉 1
j
So what changed?
j
there is a new ami being used with the filter that came out this month
so could be something in that ami
1 sec...
amzn2-ami-hvm-2.0.20230808.0-x86_64-ebs
that date coincides with when i saw some errors
i believe this one was fine
amzn2-ami-hvm-2.0.20230727.0-x86_64-ebs
or maybe the one before that. i didn't track those as i didn't consider it to be an AMI change