YEa

for now

Nothing in delete_me!

We really need to either rename that folder and / finally come up with a nice way of indexing the data (at least for modules). paranoid

Cross-posting for y'all to look at https://nfcore.slack.com/archives/C069B2Z5P1R/p1749043006644589

OK the responses so far:

nf-core test-datasets survey_Submissions_2025-06-18.csv

There are definitely som trends, anyone willing to caterogise the responses and make some stats?

OK we got up to 20 responses, we can use this to plan out at the next not-over-holiday-period maintainers meeting!

actually Updated results

I can't wait to burn this repo to the ground

To be fair we could already do a spring cleaning of the branches

I have just read the new Maintainers Minutes and wanted to point out a few things: HuggingFace • ❌ Is actually just a

git-lfs

, so actually doesn’t provide much difference to GitHub (which also supports git-lfs) Smartarse comment: It was Git LFS until a few months ago, now it is based on Xethub, which they acquired. AWS S3: • ❌ We were very worried about ingress/egress costs (particularly in our very parallelised CI tests) There could be ways around that. But it is indeed the issue that keeps us from using it more often. • ❌ We weren’t sure on the longevity of services like 42basepairs This site is run by Robert Aboukhalil, one of my role models (for his foresightedness to realize the potential of WebAssembly long before others). It uses Biowasm, which is open-source software, so it should not be too hard to implement something similar, if it were to go away. But I also do not think that we have a UI for Test datasets really at the moment? • ❌ Would maybe need to ask for open-source credits… but no idea if available Tigris S3 offers 5GB for free, which might already suffice for the current test data? I also checked Bunny, but the pricing seems comparable to Cloudflare. ✅ S3 filesystem Smartarse comment 2: Object storage is not a POSIX compatible file system. If you want that, you will need to use either a EFS or run some additional tools around S3 that give it file-system like behavior. There is e.g. S3FS (doesn’t have a good reputation) or JuiceFS or MountpointS3, but since we are mostly just downloading files, a simple HTTP Cache on top of S3 would probably suffice. For #CE5LG7WMB, going through OpenDAL is probably the easiest.

First RFC: https://github.com/nf-core/proposals/issues/70

https://nfcore.slack.com/archives/C043FMKUNLB/p1764681174724119 Any hints for this? :)

@Jim Downie was doing a nice #C07KND2JP54, and that got me thinking about this slack channel that I had forgotten about

I remember that a while back, I did my own branches that was a sub-section of the modules branch for some pipelines, so that it was simpler to deal with just these data for me

And I know we tend to go down the line of having a branch per pipeline + one for the whole modules repo, and it takes forever to get it locally

Probably a dumb idea, but how about we split some data in some other branches to lower the size?

seeing now people copying over test data from pipeline branches to modules and vice versa. They could just leave them in one place, no (preferably

modules

i guess)?

We really need a better solution 😞

It seems that

modules

is the main source of test data, but at the same time (as I recently mentioned elsewhere) it has become extremely large and must be handled with great care, otherwise it becomes an unintentional "zip bomb"

I think I only had data anymore via the web interface

S3 bucket with some kind of review/submission interface?

On my HPC I need to submit

git pull

because it otherwise gets killed on the head node 🫠

Solutions we've thought about previously: • git (current solution) ◦ ideally always do a shallow check out directly of the branch you need ◦ Good for CI as all internal to GitHub • AWS ◦ We have credits, but data egress is expensive so a bit worried it'll burn a lot • Cloudflare ◦ Free egress, but we don't have any credits • Zenodo, figshare etc ◦ Not reliable enough Any other ideas welcome!

Another one, that maybe could be facilitated via @sven_nahnsen or someone long term in Germany - could see if deNBI cloud would give us object storage. Theoretically could be free, and I think we could get a lot of support from different areas of German academic groups (e.g. NFDI4Microbiota are using nf-core a lot)

I just asked Claude about what it'd cost to use Cloudflare R2 and it came up with this (based on the pricing page). If this is accurate and the cost really is $0-5 USD per month, then this might be a good option (Seqera could own to mitigate risk).

James has started an RFC to make our test-data set up better, but doesn't have time to champion it at the moment: github.com/nf-core/proposals/issues/70#… Does anyone here want to pick up this RFC and move it forward?