https://nf-co.re logo
Join Slack
Powered by
Heads up: the Biocontainers ECR mirror (see <launc...
# random
p
Heads up: the Biocontainers ECR mirror (see launch blog post from 2022) are going to be deleted next week. These are not the same as the regular biocontainers hosted on quay.io. We don't think that anyone is using these - if you are, shout quickly!
j
Oh interesting, why is that?
a
I use the quay.io biocontainers mirror, glad to hear that is still supported!!
👍🏻 1
a
Hi Phil, we were using them and found this out the hard way. A quick scan of GitHub - https://github.com/search?q=public.ecr.aws%2Fbiocontainers&amp;type=code suggests we are also not the only ones using these.
p
Ouch, hope it didn't cause too much of a problem 😢
To be clear: these were not and never have been anything to do with nf-core. I just posted the above to try to help get the word out.
a
Ouch, hope it didn't cause too much of a problem
Ah it did.
I just posted the above to try to help get the word out.
Can you point me to any announcements about this deprecation (other than the update on the blogpost)? I can't seem to find anything anywhere about this
p
You'll need to ask the BioContainers folks. They've known about this since near the start of the year. I'm not aware of any announcements that they put out though sorry.
a
j
If that helps at all (there was a dicussion maybe about an altenrative source)
a
If that helps at all (there was a dicussion maybe about an altenrative source)
Thanks I'm aware of the alternative repos and that the solutions to our problem are rather simple. But we either need to deploy new pipeline versions into our production systems or hack together our cwltool overrides parameters to inject alternative docker requirements into our biocontainers steps. Also spent about around $1k on automated analyses over the weekend that failed at the final step because a biocontainer repo wasn't present. Ideally there'd have been a lot more fanfare around this. Even from the AWS team @Angel Pizarro
🫠 1
a
Sorry about that. We assumed that BioContainers didn’t warn their user base about it, as it is their project. Having said that we should have been more vocal about it for our customers
a
We assumed that BioContainers didn’t warn their user base about it,
Then all the more reason to announce something??
we should have been more vocal about it for our customers
Yes. Definitely. Was there ANY announcement from AWS on this? Given the original blog post was an AWS blog post I assumed (obviously incorrectly) this was an AWS supported project as AWS was investing in the genomics space? We use the public ecr gallery for a few of our AWS services but it does really reduce our trust in the gallery if repositories are just going to disappear without as much as a peep from AWS. Having a look at the matrix chat linked above from James, it seems as if the biocontainers team were under the impression that this was an AWS decision, rather than their own?
a
As I said, it is our mistake and we should have been more vocal about it. I’m not going to discuss specific details about how we came to this decision.
a
So in summary, AWS decided to delete a public container registry repository with thousands of images and told no one?
This is also a potentially massive security risk, given the repository itself is deleted, a bad actor could come along and regenerate the same repository with the same image names and tags with their own malware.
a
No that’s not correct. The BioContainers project decided to not continue with a mirror they would have to pay for. The mirror was always owned and operated by the BioContainers project.
a
No that’s not correct.
Isn't it? Who made the decision to delete the repository then? Don't want to go around in circles here but I'm really surprised and incredibly frustrated that AWS has taken such an uncharacteristically reckless sunsetting approach to this, particularly given the security risk I've outlined above. Could AWS not restore the repository for a short period of time (6 months or so) which would be a much more appropriate time for a system to be in a deprecated state and THEN delete the images in the repository before then later on deleting the repository?
p
The problem here is that AWS doesn't own Biocontainers. They donated compute credits to the Biocontainers folks, but they don't have any direct control over the AWS account or any contents. They can sponsor projects but not drive them.
It's the same for the credits that we get from AWS for nf-core. We basically put in a grant proposal saying what we want to do (eg. in our case run CI mostly) and roughly how much it'll cost. We provide an AWS account number and they deposit (time-limited) credits into the account.
AWS have zero access to that account
It'll be the same for Biocontainers. The responsibility for the Biocontainers mirror on AWS is the Biocontainers community / administrators, it's basically nothing to do with AWS.
I've been cc'ed on some of the email threads with the Biocontainers folks about this since last year. The conclusion about credits was decided around spring. As the deadline for deletion approached I was getting alarmed that I hadn't seen any outreach from Biocontainers about it, which is why I posted the message at the top of this thread, to make some effort to get the word out to anyone in our community who might be using them.
This is at the nasty end of the spectrum for what can unfortunately happen in cases like this, where open source communities have cross-dependencies. Biocontainers are used by nf-core and we hardcode the quay.io addresses, so we are suitably paranoid about those. The AWS mirror was seen as a good thing by our community but not directly referenced anywhere, so the level of caution from us running nf-core was much lower. We have no ownership or control over the Biocontainers community at all. AWS has no ownership or control over the Biocontianers community.
I understand it's frustrating because the lines between the three can be blurry and the three need to work together in your setup. It's not the first time that we've had some issues with our dependency on Biocontainers, this is a big part of why we're moving away from using them (see blog post). In the future we will use Seqera Containers for nf-core pipelines, relying only on bioconda and not biocontainers. Then we effectively have more ownership of the entire stack and hopefully things like this will be less likely to happen (Seqera has guaranteed hosting packages on the Seqera Containers registry for 5 years, with our without AWS credits, for exactly this reason - see the FAQ on seqera.io/containers).
a
Thanks for the explanations Phil, but the vague distribution of responsibilities, terrible communication, and improper sunsetting techniques is the part that baffles me about all this. More than happy to update repos in pipelines with even a little notice -- not happy to hack-patch-redrive $1K worth of analytics on a Monday morning. Biocontainers don't seem to think that there were in charge of this account, nor claim they were ever notified about it (see snippet below). AWS said it was their decision but deny responsibility for the project? No one has put their hand up to say, 'hey that's our fault, it was our project and we deleted the repository of thousands of images without any attempt to find / notify those potentially using them'. These issues for us happened outside of the nf-core context so it's only through the help of the colleague that I found this post. And if I had miraculously found this post and 'shouted quickly' as you suggested at the top of the thread, what would have happened? Would the repository not have been deleted? And who would have stopped that from happening? Clearly I'm not being told the full story here and that's probably the most frustrating part of all
p
I think what you're seeing is the full story. The AWS mirror failed precisely because no-one thought that they owned it, and no-one took ownership of it spiderman pointing There's no conspiracy here.
➕ 1
And if I had miraculously found this post and 'shouted quickly' as you suggested at the top of the thread, what would have happened?
I would have attempted to raise the alarm on your behalf and contacted the people involved that someone was using the mirror.
👍 1
a
Thanks for chiming in here Phil. @Alexis Lucattini I feel your pain, but continuing this thread (trying to assign blame) is not going to help anyone. It would be better if we can focus on a way forward to mitigate the risk in the future.
a
Not trying to assign blame here, but trying to assign responsibility to fix what needs to be fixed. No one has addressed the security vulnerability I've raised. Nor has AWS has not put any announcement out about this. Are these two issues that you can address Angel?
a
I’ve got a blog post on general guidance for resilient bioinformatics that includes how to protect against supply chain attacks. As Phil mentioned, AWS has never owned the account and had no access to anything in it. We are not able to recreate any of the repo or assets. That would need to be done by the project owner. I will work to see if there is a way to block another organization from taking the BioContainers repo name.
👍 1
Open in Slack
PreviousNext
Powered by