Chenyu Jin (Amend)
06/08/2026, 9:14 AM(sample_ID, sample_fq)
and the expected output is:
(sample_ID, unclassified_fq)
My main issue is how to correctly structure the workflow so that:
• input list are automatically split into around 128 samples per list
• each list of samples are processed in parallel within the same task/job
• each sample’s outputs are correctly tracked and associated with its sample_ID
• I avoid launching one SLURM job per sample but still maintain clean sample-level outputs
What is the recommended way to design this (e.g. in Nextflow / GNU parallel / custom scripting), and how should I manage input/output bookkeeping for each sample in a single parallelized job? I tried to input and output converting from channel to a list (.csv) in inputing and outputing but I am afraid that this will not resume properly.
I have been multithreading to the maximum (128) since it requires the whole node to run large-mem kraken2 job. Or is there any other way apart from parallelizing to save core hours usage on cluster?Niklas Schandry
06/08/2026, 9:41 AMbuffer() . I think the other things you are trying to do are more complex, because they go a bit against nextflows one sample - one job approach. You would probably need to find a way via parallel or xargs or something that would allow you to then generate the command to launch kraken2 correctly / several times in parallel (not familiar with kraken2)? I would probably then collect all outputs into a single emit (probably named by sample_id) and have some channel stuff (combination of flatten() and map() maybe) to turn them back into a proper channel. The main issue (to me) is probably to figure out if the shared memory thing in slurm actually works as intended in the nextflow generated sbatch script.James Fellows Yates
06/08/2026, 10:58 AMChenyu Jin (Amend)
06/08/2026, 10:59 AMJames Fellows Yates
06/08/2026, 10:59 AMJames Fellows Yates
06/08/2026, 10:59 AMJames Fellows Yates
06/08/2026, 11:00 AMJames Fellows Yates
06/08/2026, 11:00 AMJames Fellows Yates
06/08/2026, 11:00 AMJames Fellows Yates
06/08/2026, 11:00 AMJames Fellows Yates
06/08/2026, 11:00 AMJames Fellows Yates
06/08/2026, 11:00 AMJames Fellows Yates
06/08/2026, 11:00 AMChenyu Jin (Amend)
06/08/2026, 11:17 AMclassified = meta.single_end ? "\${PREFIX}.classified.${sequence_type}" : "\${PREFIX}.merged.classified.${sequence_type}"
unclassified = meta.single_end ? "\${PREFIX}.unclassified.${sequence_type}" : "\${PREFIX}.merged.unclassified.${sequence_type}"
I am also interested in seeing how the output val(meta), path(unclassified) etc., could be parsed to val(sample_id), path(unclassified), though I did not find them in the shared main.nf. Could you point me where this workflow is called and its output parsed?James Fellows Yates
06/08/2026, 11:22 AMJames Fellows Yates
06/08/2026, 11:24 AMI am also interested in seeing how the output val(meta), path(unclassified) etc., could be parsed to val(sample_id), path(unclassified), though I did not find them in the shared main.nf. Could you point me where this workflow is called and its output parsed?I'm not sure what you mean exactly here, but are you talking about within the module?
Chenyu Jin (Amend)
06/08/2026, 11:35 AMJames Fellows Yates
06/08/2026, 11:35 AMJames Fellows Yates
06/08/2026, 11:35 AMJames Fellows Yates
06/08/2026, 11:36 AMJames Fellows Yates
06/08/2026, 11:36 AMJames Fellows Yates
06/08/2026, 11:37 AMNiklas Schandry
06/08/2026, 11:40 AMid ). This can be kind of useful if you need to pass things through a process, i.e. you need something afterwards that does not match the process input, you can simply store it somewhere in meta and map it back out later. This can also avoid a lot of joins etcChenyu Jin (Amend)
06/08/2026, 11:41 AMChenyu Jin (Amend)
06/08/2026, 11:41 AMChenyu Jin (Amend)
06/10/2026, 10:48 AM.collectFile from the previous process output, then batching, and run through kraken2 to output a list which contains the sample_ID and output_files in each row.
However, managing the samples with a list than nf channel, now it does not resume KRAKEN2 any more despite a successful run. Since the kraken2 process is at the very beginning of my workflow, not being able to resume from here will break the resumability of the majority of the workflow.
Is there any way to restore the resume? Or is it not possible?
here are the codes:
preprocessed_reads
.map { it.join(',') }
.collectFile(name: "${workflow.runName}.csv", newLine: true)
.set { input_kraken2 }
input_kraken2.view { "input_kraken2: $it" }
//split the input of kraken2 into 128 samples per batch
input_kraken2
.flatMap { csv_file ->
// Read the file content and process it
def lines = csv_file.text.split('\n').findAll { it.trim() } // Remove empty lines
// Create batches of size 2 (batches of lines, not parsed CSV)
def batches = lines.collate(5)
// Return one item per batch
return batches.withIndex().collect { batch, index ->
def basename = csv_file.getBaseName()
def filename = "${basename}.sub${index + 1}.csv"
def content = batch.join('\n') + '\n'
return [filename, content]
}
}
.collectFile() { filename, content ->
[filename, content]
}
.set{input_kraken2_subsets}
input_kraken2_subsets.view { "1_subsets: $it" }
input_kraken2_subsets
.map{it}
.map { it -> tuple(it, params.KRAKEN2_FILTER_DATABASE) }
.set { input_kraken2_subsets_tuples }
input_kraken2_subsets_tuples.view{ "2_subsets_tuples: $it" }
KRAKEN2( input_kraken2_subsets_tuples )
KRAKEN2.out.out_list.view { "5_kraken_out: $it" }
KRAKEN2.out.out_list
.collectFile(name: "${workflow.runName}.kraken2_out_all.csv", newLine: true)
.view { "6_after_collectFile: $it" }
.splitCsv(header: false, sep: ',', strip: true)
.view { "7_after_splitCsv: $it" }
.map { row -> tuple(row[0], file(row[1]), file(row[2])) }
.view { "8_kraken2_out_list_ext: $it" }
.set { kraken2_out_list_ext }
//parse the list output by KRAKEN2 to again sampleID and unclas or clas pair
kraken2_out_list_ext
.map { it ->
params.USE_KRAKEN_CLASSIFIED == "disable"
? tuple(it[0], it[1]) // unclassified
: tuple(it[0], it[2]) // classified
}
.set { kraken_out1 }
kraken_out1.view { "9_kraken_out1: $it" }James Fellows Yates
06/10/2026, 11:02 AMJames Fellows Yates
06/10/2026, 11:02 AMJames Fellows Yates
06/10/2026, 11:02 AMJames Fellows Yates
06/10/2026, 11:03 AMcsv fileJames Fellows Yates
06/10/2026, 11:03 AMJames Fellows Yates
06/10/2026, 11:03 AMNiklas Schandry
06/10/2026, 11:16 AM=, usage of .set{} is discouraged (https://docs.seqera.io/nextflow/reference/operator#set)Chenyu Jin (Amend)
06/10/2026, 11:21 AMChenyu Jin (Amend)
06/10/2026, 11:36 AM