I was just looking at `CAT/FASTQ` and I realised t...
# modules
a
I was just looking at
CAT/FASTQ
and I realised that it saves output to
.gz
even if you give it uncompressed
fastq
. I think this is somewhat surprising given the description of the input channel gives no indication that files are expected to be `.gz`:
Copy code
List of input FastQ files to be concatenated.
This is also different to the behaviour of
CAT_CAT
which uses the input and output file extensions to magically handle compression. I’m wondering if this has been raised at all before or if there is some scope to consider changing
CAT_FASTQ
?
j
I think this might be because the module specifications strongly encourage always compress outputs where possible. And no bioinformatics tool worth its salt will not support gzipped fastq files 😅 But I have occasionally come across these cases, so I think it would be fine to have parity
a
The issue is not that the output is
gzipped
the issue is that the output is not actually
gzipped
if the inputs are not already compressed but they are still appended with
.gz
j
Oh I see!nn
Then yeah bas
Bad*
a
I think it’s fine to enforce compression, as long as that’s what actually happens. Instead we have this confusing mismatch in behaviour with
CAT_CAT
and also an input channel description that makes no mention of the need to supply compressed inputs
👍 2
j
Agree
I guess same assumption applies then... but with indeed bad behaviour
Agree it should be fixed
a
I think there might be some overlap between the compression logic here and the discussed options for a
"compression aware"
BGZIP module here
party this 1
m
@Arthur the more messages I read from you, the more #team-maintainers vibes I get
😂 1
💯 1
a
Hmm can I get some more thoughts on this. I am looking at the possible logic now and I am not sure what the “best”/“correct” outcome should be. Current: •
CAT_CAT
can decompress/compress from/to
gzip
•
CAT_FASTQ
assumes gzip input and cats to
.gz
out - silently
cats
uncompressed files to a
.gz
which is not actually compressed •
SAMTOOLS_BGZIP
compresses various compressed/uncompressed single-file input to a single-file
bgzip
output. Marketed towards being used exclusively with fasta files For an updated
CAT_FASTQ
: • Should it handle different compression options for input (i.e.
bgzip
,
xz
etc) • What is the optimal/correct compression for output?
gzip
vs
bgzip
? (
gzip
for consistency with legacy logic?) Realistically the only
fastq
specialism here is for
single_end
vs
paired
. Is there scope for some other new module for concatenation of arbitrarily compressed input (similar to new
SAMTOOLS/BGZIP
) to arbitrarily compressed output? Given that
SAMTOOLS/BGZIP
doesn’t actually have any special code that would preclude arbitrary input types, that module could be reasonably re-purposed for that use.
j
I've never seen other compression methods in the wild (just papers saying we should use them), so I would vote for sticking with bog standard gzip for simplicity (or until we hit a case) Also bgzip is NOT actually backwards compatible with standard gzip as they report (learnt the hard way recently). So again i personally would stick with standard gzip in the CAT_FASTQ module For the last question I see what you mean... but I only just woke up so needed to process it in my head first
a
Not wanting to derail this topic further, but what were the gzip compatibility issues? I’m aware of one case:
It is worth noting that there is a known bug in the Java GZIPInputStream class that concatenated gzip archives cannot be
successfully decompressed by this class. BGZF files can be created and manipulated using the built-in Java util.zip package,
but naive use of GZIPInputStream on a BGZF file will not work due to this bug.
j
.... It was indeed a Java program 👀fry
But even so, I think java is common enough this could be problematic
👍 2
m
Ok, I think bgzip is useful, but indeed samtools/bgzip is currently restricted to fasta
👍 1
j
OK more awake now, Yeah I think it's fine to have the BGZIP module more flexible with etc.
And would help keeping GZIP/BGZIP distinct
m
My main issue with bgzip/gzip is that it often use the same extension, so it's a bit of a mess
💯 2
e
Crazy idea: What if we moved the fastq combination to a beforeScript in the config? https://nextflow.io/docs/latest/reference/process.html#beforescript Not saying we should deprecate these modules they just always feel “wasteful”
🤔 1
m
I like the idea, but for me beforeScript are more used by HPC to do set up. How would you deal with pre-existing beforeScript coming from an HPC config?
🤷🏻 1