Hey @Daniel Straub, by any chance, do you remember how the

test_minigut

dataset was created for the

test

profile in #CE9MS66BS? We're having some trouble using it with an nf-core module for ALE (Assembly Likelihood Estimator), which complains about improperly paired reads. For more info, see this comment (and the previous two) from the PR where we're attempting to integrate ALE into #CE9MS66BS: https://github.com/nf-core/mag/pull/931#issuecomment-3600806922

As far as I can see test_minigut_R1/2.fastq.gz were added 7 years ago by https://github.com/HadrienG, that was even before I contributed, I think...

I see. Well, that small dataset is now giving us troublew with ALE, although it runs fine with the

test_full

profile (the CAPES_XX datasets). I've done a few tests and there are, indeed, some issues with improperly paired reads in the the BAM file:

$ gatk ValidateSamFile I=/home/ubuntu/test-ALE/work/15/11b84d8b1fbe62e2535f8e86cc5e32/MEGAHIT-test_minigut-test_minigut.bam MODE=SUMMARY
Using GATK jar /home/ubuntu/anaconda/envs/mainenv/share/gatk4-4.6.1.0-0/gatk-package-4.6.1.0-local.jar
Running:
    java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /home/ubuntu/anaconda/envs/mainenv/share/gatk4-4.6.1.0-0/gatk-package-4.6.1.0-local.jar ValidateSamFile I=/home/ubuntu/test-ALE/work/15/11b84d8b1fbe62e2535f8e86cc5e32/MEGAHIT-test_minigut-test_minigut.bam MODE=SUMMARY
INFO	2025-12-02 21:59:33	ValidateSamFile	

********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** 
<https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)>
**********
********** The command line looks like this in the new syntax:
**********
**********    ValidateSamFile -I /home/ubuntu/test-ALE/work/15/11b84d8b1fbe62e2535f8e86cc5e32/MEGAHIT-test_minigut-test_minigut.bam -MODE SUMMARY
**********


21:59:33.858 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/home/ubuntu/anaconda/envs/mainenv/share/gatk4-4.6.1.0-0/gatk-package-4.6.1.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
[Tue Dec 02 21:59:33 EET 2025] ValidateSamFile INPUT=/home/ubuntu/test-ALE/work/15/11b84d8b1fbe62e2535f8e86cc5e32/MEGAHIT-test_minigut-test_minigut.bam MODE=SUMMARY    MAX_OUTPUT=100 IGNORE_WARNINGS=false VALIDATE_INDEX=true INDEX_VALIDATION_STRINGENCY=EXHAUSTIVE IS_BISULFITE_SEQUENCED=false MAX_OPEN_TEMP_FILES=8000 SKIP_MATE_VALIDATION=false VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=2 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
[Tue Dec 02 21:59:34 EET 2025] Executing as ubuntu@bioinfosrv on Linux 5.15.0-69-generic amd64; OpenJDK 64-Bit Server VM 17.0.11-internal+0-adhoc..src; Deflater: Intel; Inflater: Intel; Provider GCS is available; Picard version: 4.6.1.0
WARNING	2025-12-02 21:59:34	ValidateSamFile	NM validation cannot be performed without the reference. All other validations will still occur.
WARNING	2025-12-02 21:59:34	SamFiles	The index file /home/ubuntu/test-ALE/work/de/4c3c818d63d27b3b9fa817c92ff480/MEGAHIT-test_minigut-test_minigut.bam.bai was found by resolving the canonical path of a symlink: /home/ubuntu/test-ALE/work/15/11b84d8b1fbe62e2535f8e86cc5e32/MEGAHIT-test_minigut-test_minigut.bam -> /home/ubuntu/test-ALE/work/de/4c3c818d63d27b3b9fa817c92ff480/MEGAHIT-test_minigut-test_minigut.bam
ERROR	2025-12-02 21:59:34	ValidateSamFile	Value was put into PairInfoMap more than once.  7: NC_000913.3_9019
[Tue Dec 02 21:59:34 EET 2025] picard.sam.ValidateSamFile done. Elapsed time: 0.01 minutes.
Runtime.totalMemory()=285212672
To get help, see <http://broadinstitute.github.io/picard/index.html#GettingHelp>
Tool returned:
-1

Below is the output of `samtools flagstat`:

(base) ubuntu@bioinfosrv:~$ samtools flagstat /home/ubuntu/test-ALE/work/15/11b84d8b1fbe62e2535f8e86cc5e32/MEGAHIT-test_minigut-test_minigut.bam
200000 + 0 in total (QC-passed reads + QC-failed reads)
0 + 0 duplicates
199354 + 0 mapped (99.68%:-nan%)
200000 + 0 paired in sequencing
100000 + 0 read1
100000 + 0 read2
57180 + 0 properly paired (28.59%:-nan%)
198752 + 0 with itself and mate mapped
602 + 0 singletons (0.30%:-nan%)
19728 + 0 with mate mapped to a different chr
19684 + 0 with mate mapped to a different chr (mapQ>=5)

I would be concerned that whatever issue is causing the improper pairing would crop up on users' datasets and cause the same error.

@Jim Downie do you think it might be a mapping issue with Bowtie2, instead of a dataset-related issue?

I guess finding a new test dataset should become a priority 🤔

It seems to me like the main issue here is that there are multiple reads with the same ID in the

test_minigut

dataset:

$ zcat /home/ubuntu/test-ALE/work/c8/652777b44a5c406b9f8ebed2650024/test_minigut_1.merged.fastq.gz | grep "NC_000913.3_9229"
@NC_000913.3_9229/1
@NC_000913.3_9229/1
@NC_000913.3_9229/1
@NC_000913.3_9229/1

(base) ubuntu@bioinfosrv:~/test-ALE/work/de/4c3c818d63d27b3b9fa817c92ff480$ zcat /home/ubuntu/test-ALE/work/c8/652777b44a5c406b9f8ebed2650024/test_minigut_2.merged.fastq.gz | grep "NC_000913.3_9229"
@NC_000913.3_9229/2
@NC_000913.3_9229/2
@NC_000913.3_9229/2
@NC_000913.3_9229/2

Without knowing much about what ALE does, it is hard to say. It might be due to the sparsity of the test data, or it might not. I don't quite understand what the problem is with the read that is failing here. Improper pairs occur when: • different reference contigs • wrong orientations of read alignment relative to each mate • reads map to same strand • distance between pairs too big or too small I don't know which of these ALE cares about. It could be a mapping issue (though I don't understand why it would be), or it could be an assembly issue.

Again, the multiple reads with the same ID don't seem to trip up the assemblers, but for anything that cares about read pairing (like ALE), these duplicate IDs will cause an issue...

In which case, the fix is to update the test dataset so that we re-name all the reads uniquely.

It might be difficult to salvage the

test_minigut

dataset itself, unless we can assume that the orders of the reads are perfectly preserved in the R1 and R2 files. Might be easier to simply replace it with a new dataset, but I don't have any good suggestion for what this could be.

I would hope that the orders of the reads are identical between the two files also... otherwise we have bigger problems

I guess we can try updating the read names and check if the tests (and ALE) are happy with it, just for the sake of make them running.

Hey @Diego Alvarez Saravia thanks a lot for looking into this. I tried something similar as well, but ALE still complains in my test, so the duplicate read names are probably not the only issue with that dataset. I am you my script if you want to have a look. It also contains the command that I've used to test the ALE module branch form my student's fork. Please let me know if I made any mistake in handling the FASTQ files.

Your script looks good to me @Alexandru Mizeranschi, so maybe the problem with the data is indeed deeper. Probably they are not proper Illumina pairs after all (or at least part of them) or are not sorted consistently between R1 and R2.

In this case, how do we continue?

I guess, if @James Fellows Yates agrees with it, we can exclude ale from the tests (as we have done with other tools), and try to include it in the ci when we change the test dataset

I guess it's a deeper issue with the test data itself then yes, to not slow down developement and given we know it works with real data we skip in CI testing with abig message in the test configs we need to fix it when we get around to prepare better data

I'm interested in that time multiplying machine

You know, I think I know avery good workflow manager that does a lot of stuff in parallel...

My students are enthusiastic, but setting up a new test dataset would be a task a bit out of their league. Any advice for how to remove ALE from the tests would be appreciated...

In each config explicitly set whatever flag you use to activate it to false with a comment above it explaining why

Nope. We never tested it with such a low amount of memory. I think we gave it 64.GB by default

OK that's good to know, and something then we should maybe bump by default!

Just make sure to test it first, we never really did any serious profiling on that tool

No worries, that's what I'm doing now

and you're saying that happens with test_full? which datasets is that using? I remember we tested with test_full (the one based on the CAPES_XY samples) and it worked for us

Yup...

that's very strange, then

Ahhhh it's failing on all the long read assemblies!

well, it's not meant to run on long-read assemblies. it's supposed to be turned off for those

Yeah I just realised that 😅