https://nf-co.re logo
Join Slack
Powered by
seqkit_stats is also problematic in the number of ...
# mag
j
seqkit_stats is also problematic in the number of files it creates in work/, having 55,000 bins it generates half a million files because it runs a single process per bin. This is a roadblock on all clusters we have access to in Canada. Can you think of any workarounds?
technically I think I could delete all 
.command.*
files in my workdir, it's dirty but they are the problem here
j
You might want to try the `nf-boost`plugin
👀 1
This cleans work directories on the fly
(just breaks resume)
j
that's my current problem, I can't break resume because I can't make it through the pipeline without busting because I cap the 1M files/user limit right after binning, even despite not having gunzip anymore (running with dev)
For future development would we be open to a bundling up of that step in a batched process, e.g. all bins from an assembler/binner combination processed together? It could be optional, but it would help making the pipeline scalable, as this is only a 4x60 coassembly and I got 56k bins from it (I did not even use semibin), which means 10 files per process minimum = 560k files generated in work only for seqkit_stats. I'm not sure how complex that would be to implement this though, @Diego Alvarez Saravia what do you think?
more stats: in a single assembly of 240 metagenomes, I end up with 110,428 bins, so >1M work files created for SEQKIT_STATS only 🙃
d
My bad, I forgot to remove the transpose when I flattened the seqkit processes. I've already opened a PR for that, pending review 😅
aaw yeah 1
3 Views
Open in Slack
PreviousNext
Powered by