I have just read the new Maintainers Minutes and wanted to point out a few things: HuggingFace • ❌ Is actually just a

git-lfs

, so actually doesn’t provide much difference to GitHub (which also supports git-lfs) Smartarse comment: It was Git LFS until a few months ago, now it is based on Xethub, which they acquired. AWS S3: • ❌ We were very worried about ingress/egress costs (particularly in our very parallelised CI tests) There could be ways around that. But it is indeed the issue that keeps us from using it more often. • ❌ We weren’t sure on the longevity of services like 42basepairs This site is run by Robert Aboukhalil, one of my role models (for his foresightedness to realize the potential of WebAssembly long before others). It uses Biowasm, which is open-source software, so it should not be too hard to implement something similar, if it were to go away. But I also do not think that we have a UI for Test datasets really at the moment? • ❌ Would maybe need to ask for open-source credits… but no idea if available Tigris S3 offers 5GB for free, which might already suffice for the current test data? I also checked Bunny, but the pricing seems comparable to Cloudflare. ✅ S3 filesystem Smartarse comment 2: Object storage is not a POSIX compatible file system. If you want that, you will need to use either a EFS or run some additional tools around S3 that give it file-system like behavior. There is e.g. S3FS (doesn’t have a good reputation) or JuiceFS or MountpointS3, but since we are mostly just downloading files, a simple HTTP Cache on top of S3 would probably suffice. For #CE5LG7WMB, going through OpenDAL is probably the easiest.

Thank you very much @Matthias Zepper! This is exactly what we were looking for!

To add my $0.02...

❌ We were very worried about ingress/egress costs (particularly in our very parallelised CI tests)

Tests are using run-on, which means that they're running on AWS. As long as compute + storage is in the same region, there should be little to no ingress / egress cost. Also we have a lot of AWS credits that we are encouraged to try to use..

❌ We weren’t sure on the longevity of services like 42basepairs

We wouldn't be dependent on this, it's just a nice add on. So seems a little irrelevant..?

(cloudflare) ❌ Would maybe need to ask for open-source credits… but no idea if available

Thats a biiiig ❌ - it's taken us years to build the relationship with AWS, and we've struggled with azure, failed with Google despite many efforts. Likely that it will be very difficult to do this.

Tests are using run-on, which means that they’re running on AWS. As long as compute + storage is in the same region, there should be little to no ingress / egress cost.

True, I forgot that RunsOn deploys directly into the AWS account. Because it developer’s build applications “wrong” and use the public internet in between egress charges still occur.

...like Fusion, which we already get for free

I am aware that Fusion has a similar purpose and even had a brief look before writing my response. But “FusionFS driver is auto-installed by Wave service” to me sounded like it is tightly coupled to containers. That precludes Conda tests and and also third-party users, who may or may not use nf-core test data for their pipelines as well.

Slack channel, upload files to slack. (Use a slack workflow to then upload??)

Why upload to Slack? You could probably create pre-signed URLs with it, everything else should be on the website.

conda / 3rd party

The test data would still work here, just with regular Nextflow staging (bit slower, needs local file storage if big - but data is small so shouldn't make much difference). It's just that we have Fusion + native Nextflow already, so not sure that we need to go chasing another solution. I'm generally not very worried about egress fees though. This will be nothing like the volume of iGenomes and we should have plenty of credits for it. Indeed we're already doing exactly this for the pipelines that get tested the most (especially rnaseq) and have done for several years and it's been fine. My guess is that rnaseq and sarek get tested 10-100x more than any other pipeline, as they're commonly used to check / benchmark systems. If they're already on s3 and it's not a problem then I very much doubt that we will hit any issues with the rest of it.

why slack?

Because I was thinking about how to handle moderation. We don't want any old people to be able to upload arbitrary data to our public bucket, so we need some kind of staging area (hence slack uploads). But yeah it could be done in any range of methods, was just spitballin'

Are you sure we don't have the volume of iGenomes? Every single test run for every mdule will be pulling from there... and what happens if credits stop being renewed? Or just based on the rnaseq/sarek experience?

iGenomes is about 5PB off the top of my head. But the storage size makes almost no difference. The vast, vast majority of the cost is data egress (something like 99% off the top of my head again).

ok fair enough

Let me know if you want me to join. Back at work from Monday.

Btw, while I won't be there (ping @Simon Pearce), there are more points on the maintainers agenda on Friday to talk about this if you can make it @Phil Ewels: )

I wasn't planning on as I have a clash, am I needed?

No not needed! You just said you might be interested 😉