https://nf-co.re logo
Join Slack
Powered by
In my crusade to make mag fully free of channel "l...
# mag
d
In my crusade to make mag fully free of channel "locks", I have a last one: gunzipping bins. Right now each binner emits multiple gzipped bins. Those get flattened and fed to single gunzip processes, then joined back together for downstream steps. This is bad for two reasons: it creates a lot of small jobs (several thousand), and the flatten/process/merge means there's a sync point, so nothing downstream starts until all the gunzipped bins are ready. Two options I can think of: 1. A 
gunzip_batched
process that does all the bins at once. Con: it's a new local module (though maybe we can ask if there's interest to publish it to nf-core). 2. Let the downstream tools handle the gzipped files directly, since they support it either natively or with an explicit gunzip when they detect the extension (on a quick check Das Tool and CAT_pack will need small adjustments). Con: probably less efficient since each process has to decompress, but the bins are small so it shouldn't be a big deal. I personally think that 2 is the best option, but it needs more changes and testing than 1. Maybe we can implement 1 for now and migrate to 2 when we decide to go for breaking changes (mag v6). Opinions?
🚀 1
j
Are you absolutely sure all downstream tools can accept the gzipped input? I might be mixing up pipelines but I seem to remember finding that there were a couple of tools that didn't support it and indeed it was quite inefficient to gunzip within each step...
I think 1 would be preferable for now indeed. And I think an official module may be acceptable in this regard
We already have a special 
find_concatenate
for example
d
I have this nice summary by Claude. I checked a few ones and they're good. Def will need a double-check if we go that route
j
So only CAT_pack is missing support?
That would be easy to implement then I guess
d
And 
RENAME_PREDASTOOL
 , but that's local
👍 1
j
Ah then go for it I guess then
I don't think it necessary means more changes/testing per se
d
Alright, I will give it a go.
And it's done: github.com/nf-core/mag/pull/1065 It went quite well.
🎉 1
j
following with much interest
I'm having space issues on our clusters' scratch quotas and I'm pretty sure I'm about to bust because of the unzipping If I wanted to interrupt this run, pull your new PR in my local code (I'm running from the dev branch from my last PR), and resume... is there a clean way to clear out the unzipped fastas from my work/ without affecting anything else, e.g. ensuring my resume will pickup the cached steps from SPLIT_FASTA and restart from there? Is this a good idea? I'm in the steep of the learning curve so sorry if I'm missing something elementary here but I want to do all I can not to have to start over the binning process. I assume I could point nextflow clean to the right hashes but I'm not sure how to compile all 45,000 of them.
d
If you only apply that PR by patching your copy of mag, cache should not break, because the change only touches that part. That being said, I'm not completely sure that your quota is getting filled because of that. After all, bins should small... but well you have 100k, multiply that for a few MB and you enter the TB territory.
j
yeah currently my scratch quota is 10Tb or 1M files and my work dir just hit it despite my results being published elsewhere. I'm trying to figure out ways to reduce the amount of intermediate files being written in work throughout that pipeline. This is a 240 sample single assembly project.
d
If you already passed the assembly step, you can start mag in assembly_input mode with a clean working directory, that could help you to manage the usage. The con is that worst case scenario, you will need to re-run the binning.
Another tip (more for files / inodes) is that if you plan to use BUSCO, use 
busco_clean = true
. BUSCO generates a lot of small files, and that adds a cleanup step to reduce the burden on th filesystem (and quota).
❤️ 1
j
yeah just the unzip process was creating ~500,000 files in work because each unzip generates about 10 files 🙃 removed them the dirty way and I'm installing the latest dev revision, this is insanely well timed
🙌 1
d
Just a notice: dev has many changes compared to stable, and it could possibly make your cache useless. I don't remember if we have some change on preprocessing and/or assembly, but from there we definitively do.
j
I was already running from dev since my skip-fastqc PR on june 24th, and did not have cache problems. Fingers crossed for this revision...
🤞 1
Any idea why I'm getting these errors now?
Copy code
ERROR ~ Error executing process > 'NFCORE_MAG:mag:ASSEMBLY:SHORTREAD_ASSEMBLY:METASPADES (121_J14)'

Caused by:
  Process requirement exceeds available CPUs -- req: 48; avail: 1
this only happened since I switched to the latest commits on dev. It also happens for phix removal, which is the first non-cached step the pipeline tries to do. I tried skipping it just to see if I'd hit another bump and it happens at the next process the resume can't cache. I'll restart binning anyways but i thought it was worth reporting (let me know if i should open an issue with more details)
d
That's weird. Did you have spades fixed with 1 cpu? Maybe it's some kind of disagreement between the scheduler and the process, someway? I'll try to reproduce it.
j
ok that looks like a cluster-specific problem. Adding
Copy code
executor = 'slurm'
    resourceLimits = [ cpus: 128, memory: 512.GB, time: 7.days ]
to config fixed it. Not sure why I have to be explicit about this now and why this only applies to one cluster where I didn't have that error before. Gemini suggests that nextflow evaluates local limits before passing jobs to slurm and somehow this dev revision seems to pick that up as a hard limit. I'll inquire to see if that's something this cluster really does, and maybe we could have a fix for those cases in the conf/base.config or somewhere else... not sure By the way the only thing that prevented caching up to where I was was phix removal. Skipping it allowed me to continue from where I was (post-binning) with the latest dev revision.
🎉 1
Can you think of anything you added between the PRs 1057 and 1065 that could have triggered this? might be something changed on the cluster's end but looking for anything to give them
d
I don't recall any recent change on pre-processing steps.
2 Views
Open in Slack
PreviousNext
Powered by