ok it looks like its random. is there like a best ...
# nostupidquestions
b
ok it looks like its random. is there like a best practices for storing the sample order?
j
Yes, Nextflow follows the dataflow set of principles, so is designed to be asynchronous: https://www.nextflow.io/docs/latest/overview.html#processes-and-dataflow Best practises depends where you want to set the order, but assuming you've already grouped all the VCFs in a single list prior going into the module, you could just do something like a
.map(vcfs -> [vcfs.sort()] )
m
yeah, I'm with James here, I never assume order, and I always re-order when I need to
👍 1
b
ok and maybe just a follow up stupid question but how to feed that info through should I emit it through different modules/workflows all the way to my sub workflow of interest. or is ok to hit the params.input directly ? or is this not recommended
like so ? (also I would make a pull request to the nf-core pipeline if so)
Copy code
Channel.fromPath(params.input)
.splitCsv(header: true)
.map { it.sample }
.unique()
.collect()
.map { it.join('\n') }.collectFile(name: 'sample_order_file.txt')
.set { ch_sample_order }
a
This doesn't sound like a good idea. You are probably going to process your files before you want to merge them, i.e. fastq -> trimming -> mapping -> variant calling -> merging. Sending files through a process basically randomizes the order within a channel. You should therefore determine the sample order right before the process that takes the list of files.
I usually use
groupTuple
and its sort function to collect multiple channel entries. I then add the list of sample ids into a new joint meta map.
b
@Alexander Nater thanks for the reply. so my goal here is to sort the samples names and pass ch_sample_order into a bcftools view command to sort a merged vcf after its created. this is already very deep deep inside the pipeline. the merged vcf cannot be sorted prior to its creation and as you mention its existing sample order is randomized (its one-liner that creates it is taking a list of bams) when this vcf is originally being created. so I need to do the sorting downstream after its been created. [well theoretically we could do the sort on the input bams but to me that is far less traceable then forcing a specific sort order via names from the samplesheet] in the use-case, multiple multi-sample vcfs are being merged and the software performing that merge is not currently robust to multi-sample vcfs with different sample orders. the meta map is probably way too large a refactor than I reasonably manage given that its an nf-core pipeline. my hope here is to fix a singular bug with a mininmal amount of code. the channel thing above is theoretically sufficient to pull the sample order straight from the sample sheet which to me is at least a straight forward solution? but this is not ideal I guess?
a
Is the problem here that you want a very specific order of samples or just a robust, reproducible order? I don't see a reason why you can't enforce the order when you generate the files. You have to wait anyways for all upstream processes to finish before you can create the multi-sample vcf file. Just make sure that the list of bam files that you provide to the process has the correct order.
Or do you mean that the tool doesn't respect the order of bam files and just outputs a multi-sample vcf file with random sample order? I think Freebayes was doing that at some point.
b
the goal is robust reproducible order. its manta in this case and runs on a "list" of bams. and manta outputs a vcf with samples ordered identical to the order of the bams it is passed [and they come in randomly]. however all the other tools (e.g. delly, gatk, etc..) run one-by-one on the bams. so I guess you're saying the best solution is just to sort the order of bams before they go into manta... ok I went back through and see that indeed in all other tools there is a param forcing the sorting the of vcfs before merging so its just that manta doesn't have this step that is causing the bug. so sorting the bams before running should solve the issue