Could I get some thoughts on this issue. Basically I added an optional VCF output to

glnexus

but it transpires that it dies with a funny exit code (

9

) if the

glnexus

process gets OOM (I think it bubbles through to the

bgzip

and breaks there). I’m not sure if there is a better way to handle this failure or if I should just drop the pipe and revert the module for now?

Oh this is a good one that we should also maybe add specs about too... My instinct is either 1. Leave it, but add comment saying it'll probably break and the next person who needs it should fix it 2. Maybe come up with some fancy pipe capturing (they do that a lot in bioconda) to capture the 9 exit code and allow it to continue 3. Remove it But also: have you tried giving the nextflow process a memory overhead so it has more than what the tool actually gets given (e.g. multiply by tast.ext × 0.8)? To me it doesn't sound like it's completely broken: minimal tests work and the broken retry is the bigger issue (is that correct?) which is why I say 1/2 first But also I understand with 2 maybe you don't want to invest time in this...

So the background is that I added the feature because I wanted to avoid writing out a BCF I never actually needed. I've found that even without this step the tool seems to break a lot on our cluster with OOM (so I've resorted to patching it to use

mem * .75

- although this still fails sometimes, and boosting the mem by a huge amount overall). The issue mainly here is that the pipe is effectively broken for retries. My feeling is I should probably either fix the pipe issue, or remove the VCF option altogether. For fixing it I'm not sure whether there is already some preferred method for this, and I'm also unsure if the exitcode would be affected by a different HPC system (the underlying OOM issue on our cluster throws a

slurmstepd killed OOM

message)

I personally would just say then comment with an inline comment and then in the meta that this may break and leave it to the next person 😬. However, could you 'manually' generate the bcf outside of the tool? As in with another process ?

What do you mean by manually generate the BCF? BCF is the only output format glnexus gives, I'm just piping through bcftools to save a process/disk space. I assume everyone else using glnexus currently just uses a bcftools_view downstream to convert.

Ah sorry, maybe misunderstood (was rather groggy, was 5.30am 😅)

Your regular reminder that

BCF

is not a compressed format

Isn't it 👀 (Ive never actually used it 😅)

It’s binary, you can have

bcf.gz

too

Oooohhhh

I mean I suppose strictly speaking its compressed compared to plain text VCF.

Huh interesting

But anyway…. I will PR to remove the VCF output option

Well regardless of terminology quibbles: https://nf-co.re/docs/guidelines/components/modules#compression-of-input-and-output-files Basically this is your stamp of approval for dropping if the uncompressed/(read: plain text larger 😉) version if it also regularly breaks the module