HI there! I believe the aim of this pipeline is to take the output from sarek seamlessly. As far as I know, sarek does not output the the necessary inputs for CNA analysis. CNVkit outputs need to be formatted. Please let me know if I am missing something.

Yes that is the idea. I think there are some changes as to how cnvkit is used in sarek and when that is finalized we will adapt the pipeline as well.

OK. I think I have found another bug. The manta output from sarek has this line in the header which is causing an issue for the bcftools norm process.

##source=GenerateSVCandidates 1.6.0

Here is the error:

ERROR ~ Error executing process > 'NFCORE_VARIANTPRIORITIZATION:VARIANTPRIORITIZATION:INPUT_PREPROCESSING:BCFTOOLS_NORM (HD827.HD827-manta.generatesvcandidates 1.6.0)'

Caused by:
  Process `NFCORE_VARIANTPRIORITIZATION:VARIANTPRIORITIZATION:INPUT_PREPROCESSING:BCFTOOLS_NORM (HD827.HD827-manta.generatesvcandidates 1.6.0)` terminated with an error exit status (255)


Command executed:

  bcftools norm \
      --fasta-ref Homo_sapiens_assembly38.fasta \
      --output HD827.HD827-manta.generatesvcandidates 1.6.0.norm.vcf.gz \
      -m -both -cw -Oz --write-index=tbi \
      --threads 6 \
      HD827.manta.diploid_sv.vcf.gz

Command exit status:
  255

Command output:
  (empty)

Command error:
  [E::hts_open_format] Failed to open file "1.6.0.norm.vcf.gz" : No such file or directory
  Failed to read from 1.6.0.norm.vcf.gz: No such file or directory

I edited the header in all of the manta vcf files to remove the space between the

generatesvcandidates

and

1.6.0.norm.vcf.gz

which seemed to fix the bug. I get the error with version 1.0.0 and dev.

I am a little confused - it seems to me that at some point there are blanks introduced into filenames and then the files are not found anymore?

This is a filename parsing / channel naming issue in the pipeline rather than a

bcftools

problem itself. The process is generating this output filename:

--output HD827.HD827-manta.generatesvcandidates 1.6.0.norm.vcf.gz

Notice the space:

HD827.HD827-manta.generatesvcandidates 1.6.0.norm.vcf.gz
                                           ^

Because of that,

bcftools

interprets: •

HD827.HD827-manta.generatesvcandidates

as the output file •

1.6.0.norm.vcf.gz

as an input argument So it tries to open:

1.6.0.norm.vcf.gz

Which does not exist, causing:

Failed to open file "1.6.0.norm.vcf.gz"

HOwever, the 1.6.1 comes from the the vcf header. It is not found in my samplesheet anywhere.

Ok I think that first thing should be an "easy" fix, can you open an issue over at the repository describing what you did and supply nextflow.log file etc?

At least that is the cause of the first error. "Fixing" the vcf header to remove the space stops the bcftools norm command, but causes downstream errors, which are beyond me.

Its a good catch but I would not expect the pipeline to work with manta output at this moment anyways:

Ah yes. Shall I post the bug anyway?

Maybe as a note? So that we keep it in mind when we start working on manta

I already posted it as a bug... I'll change it to a note

changed it, thanks!

The problem is that in Unix shells and scripts (and by extension in Nextflow

script

blocks that are

bash

code), any variable should be quoted (with

"

to allow expansion, e.g. for

bash

[environment] variables, or

'

to use the content verbatim; Nextflow variables would probably fall into the latter category). This is contrary to most programming languages and generally inconvenient (although one gets use to it), and many people don't even about it. In nf-core pipelines (and probably most Nextflow pipelines) this quoting is therefore omitted, which means that variables containing file names with spaces in them don't work (as you just saw). The "solution" that is pervasive in nf-core is to prevent such file names, typically at the step of sample sheet validation. (This can force the users to employ bizarre work-arounds. For example, I had a case where an input FASTQ was in a directory with spaces in the name, and it was not up to me to change that, so I had to create a "space-less" path via symlinks and use that in the sample sheet. This can work, but soon as the path gets "canonicalized" [i.e. all links resolved] somewhere, it breaks again.) This works with input files, but if a file is generated by some tool in the pipeline, this approach does not help, as you have seen. (I am not familiar with the pipeline and the tools, but the

/--tool: invalid choice: 'generatesvcandidates_1.6.0'

error also may well be a consequence of this same problem.) The real and proper solution to this: Quote all of your variables, in every

script

block (any anywhere else a shell would interpret them). This should have been done from the beginning of nf-core, and doing it for hundreds of thousands of lines of legacy code is a lot of work (maybe something to delegate to AI??), but it saves you kludgy and unnecessarily restrictive work-arounds (like that annoying sample sheet validation, or "manually" removing any spaces that were introduced -- there is always a chance you'll miss one!) and bugs like these. So my suggestion would be to start here, with the modules in this pipeline.