A bunch of nf-core pipelines have in their instruc...
# modules
c
A bunch of nf-core pipelines have in their instructions that their input must be compressed with
bgzip
. But databases like NCBI only offer
gzip
-compressed genomes for download. To ease the life of users and pipeline maintainers, I propose a new module,
samtools/bgzip
, that takes a FASTA file as input that can be either not compressed, or gzip-compressed, or bgzip-compressed, and outputs a file that is guaranteed to be bgzip-compressed. I welcome your comments, or your reviews if you think that it is ready to go! https://github.com/nf-core/modules/pull/7433
👀 1
m
I think you might have overlooked https://nf-co.re/modules/tabix_bgzip/ 😉
c
Is
tabix/bgzip
really going to "Do The Right Thing©" on an input that is gzipped? What I want is to be sure that anything that is fed to
samtools/faidx
is not going to crash the module.
s
there is only one way to find out
😆 1
m
I’d say write some tests 🤷🏻 That’s why we have nf-test.
this 2
Also don’t be afraid to add functionality and ping the maintainer team for a review
a
So, couple of things: •
tabix/bgzip
I believe will just end up decompressing the file if it’s already compressed (either by gzip or bgzip) and won’t recompress it •
bgzip
itself is fully interoperable with
gzipped
files [GitHub]
I think perhaps the correct course of action here could be to add a new module perhaps called
tabix_recompress
or something? Maybe like:
Copy code
zcat -f ${input} | bgzip -c > ${output}
c
Thanks for the heads-up on the behabiour of
tabix/bgzip
, I did not realise that sanitises its output by decompressing files even if they were already bgzipped. In
samtools/bgzip
I test if the file was compressed with bgzip or plain gzip, and if it was bgzip I just return it. Would a PR along these lines be welcome? Note that it requires to add
conda-forge:file
to the containers and environment.
Copy code
COMPRESS_TYPE=\$(file -L -b $fasta)
    case "\$COMPRESS_TYPE" in
        "gzip compressed data, extra field"*)
            # Assuming the "extra field" implies BGZF, do nothing.
            ln -s $fasta ${prefix}.bgzip.fa.gz ${output}
            ;;
        gzip*)
            # Recompress non-BGZF gzipped files
            zcat $fasta | bgzip $args --threads ${task.cpu} ${output}
            ;;
        *)
            # Compress
            bgzip $args --threads ${task.cpu} --stdout $fasta > ${output}
            ;;
    esac
For sure, compared to the all the computations made on the bgzipped files, it does not cost much to decompres and recompress them immediately after, but still it feels like a waste of electricity…
m
anyone familiar with grabix ? This tool can also check if a file is bgzipped
a
You could also just do this I think (which has no extra deps):
Copy code
((bgzip -cd ${input} | bgzip -c ) || bgzip -c ${input}) > $output
c
Just in case, I asked the htslib maintainer to provide a simple way to test for BGZF compression. https://github.com/samtools/htslib/issues/1884
🙌🏻 1
🙌 1
m
I just saw this option in the manual `-g, --rebgzip use an index file to bgzip a file``
Perhaps it’s possible to use the
gzip
index to
bgzip
the file?
a
I mean you’d still end up de- and recompressing an already bgzip file, which is what I think Charles is trying to avoid
c
Anyway, after one day of hard work on that module, I started to use it to implement CRAM support in my favourite #pairgenomealign pipeline, only to realise that I can not represent pairwise genome alignments in CRAM format when genome length is longer than 2^28 - 1 = 268,435,455 because of how cigar lines are binary-encoded in BAM/CRAM files, which just kills the need I had for bgzip compression sanitiser. Therefore, I am not going to push hard further to change the
tabix/bgzip
module… Nevertheless, I will try to do something if the issue I opened on htslib leads somewhere, or if you ping me to open a PR based on something like the patch that I showed above.
Hello again, the htslib maintainers pointed me to the
htsfile
tool that was exist for 10 years already. So I proposed a patch to
tabix/bgzip
this morning. https://github.com/nf-core/modules/issues/7444
a
I think you fundamentally misunderstand the purpose of the
tabix/bgzip
module. It is intended to either decompress or compress based on the given input. Given it is a widely used module used in at least 10 nf-core pipelines alone (not to mention any non-nf-core ones) I don’t think fundamentally changing the way the module works is the right thing to do. Hence why I proposed you create a new module something like
tabix/recompress
c
Sure, but then why was I asked to look at
tabix/bgzip
when I proposed
samtools/bgzip
which is basically doing what you call
tabix/recompress
? https://github.com/nf-core/modules/pull/7433
a
I presume that it was not clear what your aim was or that the proposer assumed you might use it to first decompress then recompress over 2 processes.
c
What I want is a module for which it is guaranteed that the output will be bgzipped.
In any case, since it is ready, I am file using it as a local module if there is no appetite for it 🙂
a
That’s fine, it’s just explicitly not the expected behaviour of
tabix/bgzip
(whether that’s a bit confusing or not).
Personally I have never liked the use of extension testing to determine if a file is compressed or not, so I am quite happy to see
htsfile
replace that. It could potentially replace the current logic in
tabix/bgzip
too if it’s not an extra dependency.