Arthur
05/05/2025, 2:14 AMCAT/FASTQ and I realised that it saves output to .gz even if you give it uncompressed fastq.
I think this is somewhat surprising given the description of the input channel gives no indication that files are expected to be `.gz`:
List of input FastQ files to be concatenated.
This is also different to the behaviour of CAT_CAT which uses the input and output file extensions to magically handle compression. I’m wondering if this has been raised at all before or if there is some scope to consider changing CAT_FASTQ?James Fellows Yates
05/05/2025, 4:05 AMArthur
05/05/2025, 4:06 AMgzipped the issue is that the output is not actually gzipped if the inputs are not already compressed but they are still appended with .gzJames Fellows Yates
05/05/2025, 4:07 AMJames Fellows Yates
05/05/2025, 4:07 AMJames Fellows Yates
05/05/2025, 4:07 AMArthur
05/05/2025, 4:07 AMCAT_CAT and also an input channel description that makes no mention of the need to supply compressed inputsJames Fellows Yates
05/05/2025, 4:07 AMJames Fellows Yates
05/05/2025, 4:08 AMJames Fellows Yates
05/05/2025, 4:08 AMArthur
05/05/2025, 4:09 AM"compression aware" BGZIP module hereMaxime Garcia
05/05/2025, 6:17 AMArthur
05/12/2025, 1:50 AMCAT_CAT can decompress/compress from/to gzip
• CAT_FASTQ assumes gzip input and cats to .gz out - silently cats uncompressed files to a .gz which is not actually compressed
• SAMTOOLS_BGZIP compresses various compressed/uncompressed single-file input to a single-file bgzip output. Marketed towards being used exclusively with fasta files
For an updated CAT_FASTQ :
• Should it handle different compression options for input (i.e. bgzip, xz etc)
• What is the optimal/correct compression for output? gzip vs bgzip? (gzip for consistency with legacy logic?)
Realistically the only fastq specialism here is for single_end vs paired. Is there scope for some other new module for concatenation of arbitrarily compressed input (similar to new SAMTOOLS/BGZIP) to arbitrarily compressed output? Given that SAMTOOLS/BGZIP doesn’t actually have any special code that would preclude arbitrary input types, that module could be reasonably re-purposed for that use.James Fellows Yates
05/12/2025, 4:07 AMArthur
05/12/2025, 4:18 AMIt is worth noting that there is a known bug in the Java GZIPInputStream class that concatenated gzip archives cannot be
successfully decompressed by this class. BGZF files can be created and manipulated using the built-in Java util.zip package,
but naive use of GZIPInputStream on a BGZF file will not work due to this bug.
James Fellows Yates
05/12/2025, 4:30 AMJames Fellows Yates
05/12/2025, 4:30 AMMaxime Garcia
05/12/2025, 7:02 AMJames Fellows Yates
05/12/2025, 7:03 AMJames Fellows Yates
05/12/2025, 7:04 AMMaxime Garcia
05/12/2025, 7:05 AMEdmund
05/15/2025, 4:33 PMMaxime Garcia
05/15/2025, 4:39 PM