https://nf-co.re logo
Join Slack
Powered by
<@U07MHF3NYVB> <@U039DEJ1RJS> (:arrow_left: I saw ...
# demultiplex
t
@Delfina Terradas @Matthias De Smet (⬅️ I saw your names in the lateset commits) 🔴 I recently upgraded the BCL_DEMULTIPLEX swf to the latest f3086da commit (March, 19th). Ever since one of my tests is not passing anymore with
Copy code
ERROR ~ No such file or directory: <https://raw.githubusercontent.com/nf-core/test-datasets/modules/data/genomics/sarscov2/illumina/bcl/200624_A00834_0183_BHMTFYDRXX.tar.gz/*.xml>

   -- Check script '/<path>/subworkflows/nf-core/bcl_demultiplex/main.nf' at line: 65 or see '/<path>/.nf-test/tests/a9f9a82b17265ea90d24d15669e96ce2/meta/nextflow.log' file for more details
  ERROR ~ Pipeline failed.
Line 65 is
Copy code
def xml = files(file(flowcell).resolve("*.xml"), checkIfExists: true)
I am using nf-core's public SARS-CoV-2 Illumina BCL test data. The tarball's files are
Copy code
├── Config
├── CopyComplete.txt
├── Data
├── InterOp
├── Recipe
├── RTA3.cfg
├── RTAComplete.txt
├── RunInfo.xml
├── RunParameters.xml
└── SampleSheet.csv
hence, there are some XML files present. Can you tell what's going on here?
m
Copy code
<https://raw.githubusercontent.com/nf-core/test-datasets/modules/data/genomics/sarscov2/illumina/bcl/200624_A00834_0183_BHMTFYDRXX.tar.gz/*.xml>
You need to untar the archive first, otherwise the glob doesn’t work
👍 1
t
Nice catch, thank you. gratitude thank you❗ Now I am wondering about two followups: 1. Two lines prior (L63) in the bcl_demultipliplex/main.nf it seems to not be an issue with the interop dir. I am wondering if this just silently fails for tarballs because existence of the interop dir is not strictly required as L64 just provides a fallback solution if dir not present. 2. I'll hotfix my test, no question. It might be a reasonable addition for the future though that BCL_DEMULTIPLEX swf is able to handle tarballs as the actual BCLCONVERT module within was designed in a way of being able to handle a path or a tarball. The surrounding swf is now limiting this degree of freedom since the multiqcsav addition.
a
Yeah open a ticket for this if this is of interest, then one could tackle it in a subsequent release. Since there were so many changes already and bugfixes, we pushed for a new release to have something stable to refer to again.
🆗 1
t
Can I put this issue of the 
bcl_demultiplex
subworkflow back on the radar? Afaik this isn't solved yet and I am humbly but honestly wondering how this is not a bigger topic. A couple month ago when I played with some nf-core test data it was fine to unpack the tarball. But this is absolutely not feasible for a genomics core unit to do this on a regular basis for real-world service data. We'd be talking about terabytes of disc space overhead and hours of CPU time for decompression. My current plan is that I am about to patch the MQCSAV module out of the subworkflow to get things done 😕 Am I just the first one using this subworkflow with tarballs and for routine sequencing? 😅
m
No sure what the issue is here 🤔 you can still have a tarball as input, but you have to untar it before using it as in input for the subworkflow. Earlier versions just untarred it within the 
bcl2fastq/bclconvert
module, so the decompression time should be equal 🤷🏻 all you need is the plumbing before the subwf
gratitude thank you 1
Am I just the first one using this subworkflow with tarballs and for routine sequencing? 😅
I might think so 🤔 as you mentioned yourself, the compression/decompression time overhead is way too big for larger flowcells, so we just eat the storage cost in favor of processing speeds
For our cloud stuff we run a continuous (selective) sync between the NAS attached to the sequencer and our cloud provider, so the transfer delay is kept at a minimum
And locally we just mount the NAS directly to the image 😄
t
@Matthias De Smet thanks for our detailed insights and hints!
Earlier versions just untarred it within the 
bcl2fastq/bclconvert
module
This was a super important pointer
m
t
Yes , I just looked into BCLCONVERT in more detail and found the 
tar -x
down the line
2 Views
Open in Slack
PreviousNext
Powered by