Hi all, I have been attempting to make a new module following nf-core guidelines. I am trying to build a test:

nf-core modules test nasvar/pipeline


                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 4.0.2 - <https://nf-co.re>


? Choose container software to run the test with Docker
INFO     Generating nf-test snapshot
Displaying nf-test output
╭───────────────────────────────────────────────────────────────────────────────────────────── nf-test output ─────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                                                                                                          │
│ 🚀 nf-test 0.9.5                                                                                                                                                                                         │
│ <https://www.nf-test.com>                                                                                                                                                                                  │
│ Please cite: <https://doi.org/10.1093/gigascience/giaf130>                                                                                                                                                 │
│ (c) 2021 - 2026 Lukas Forer and Sebastian Schoenherr                                                                                                                                                     │
│                                                                                                                                                                                                          │
│ Load .nf-test/plugins/nft-anndata/0.4.1/nft-anndata-0.4.1.jar                                                                                                                                            │
│ Load .nf-test/plugins/nft-bam/0.6.1/nft-bam-0.6.1.jar                                                                                                                                                    │
│ Load .nf-test/plugins/nft-csv/0.1.0/nft-csv-0.1.0.jar                                                                                                                                                    │
│ Load .nf-test/plugins/nft-compress/0.1.0/nft-compress-0.1.0.jar                                                                                                                                          │
│ Load .nf-test/plugins/nft-fastq/0.1.0/nft-fastq-0.1.0.jar                                                                                                                                                │
│ Load .nf-test/plugins/nft-utils/0.0.9/nft-utils-0.0.9.jar                                                                                                                                                │
│ Load .nf-test/plugins/nft-vcf/1.0.7/nft-vcf-1.0.7.jar                                                                                                                                                    │
│                                                                                                                                                                                                          │
│ Test Process NASVAR_PIPELINE                                                                                                                                                                             │
│                                                                                                                                                                                                          │
│   Test [f427fc01] 'nasvar pipeline - bam - RUNX1::ETV6' PASSED (412.294s)                                                                                                                                │
│   Test [e96eaa29] 'sarscov2 - bam - stub' PASSED (3.355s)                                                                                                                                                │
│                                                                                                                                                                                                          │
│                                                                                                                                                                                                          │
│ SUCCESS: Executed 2 tests in 415.698s                                                                                                                                                                    │
│                                                                                                                                                                                                          │
│                                                                                                                                                                                                          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Displaying nf-test error
INFO     Generating nf-test snapshot again to check stability
Displaying nf-test output
╭───────────────────────────────────────────────────────────────────────────────────────────── nf-test output ─────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                                                                                                          │
│ 🚀 nf-test 0.9.5                                                                                                                                                                                         │
│ <https://www.nf-test.com>                                                                                                                                                                                  │
│ Please cite: <https://doi.org/10.1093/gigascience/giaf130>                                                                                                                                                 │
│ (c) 2021 - 2026 Lukas Forer and Sebastian Schoenherr                                                                                                                                                     │
│                                                                                                                                                                                                          │
│ Load .nf-test/plugins/nft-anndata/0.4.1/nft-anndata-0.4.1.jar                                                                                                                                            │
│ Load .nf-test/plugins/nft-bam/0.6.1/nft-bam-0.6.1.jar                                                                                                                                                    │
│ Load .nf-test/plugins/nft-csv/0.1.0/nft-csv-0.1.0.jar                                                                                                                                                    │
│ Load .nf-test/plugins/nft-compress/0.1.0/nft-compress-0.1.0.jar                                                                                                                                          │
│ Load .nf-test/plugins/nft-fastq/0.1.0/nft-fastq-0.1.0.jar                                                                                                                                                │
│ Load .nf-test/plugins/nft-utils/0.0.9/nft-utils-0.0.9.jar                                                                                                                                                │
│ Load .nf-test/plugins/nft-vcf/1.0.7/nft-vcf-1.0.7.jar                                                                                                                                                    │
│                                                                                                                                                                                                          │
│ Test Process NASVAR_PIPELINE                                                                                                                                                                             │
│                                                                                                                                                                                                          │
│   Test [f427fc01] 'nasvar pipeline - bam - RUNX1::ETV6' PASSED (413.67s)                                                                                                                                 │
│   Test [e96eaa29] 'sarscov2 - bam - stub' PASSED (3.235s)                                                                                                                                                │
│                                                                                                                                                                                                          │
│                                                                                                                                                                                                          │
│ SUCCESS: Executed 2 tests in 416.961s                                                                                                                                                                    │
│                                                                                                                                                                                                          │
│                                                                                                                                                                                                          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Displaying nf-test error
CRITICAL Ran, but found errors:
          - nf-test failed
          - nf-test failed

So my tests succeeded, but then it end with saying that nf-test failed? I also ran it with --verbose, but I did not see any other error messages. Any suggestions for what is going wrong?

This is probably best asked in #C049MBCEW06 where the experts are. Did you look at the work directories of the tests? I have no experience with testing, but as I understand it, there are two components to a test: 1) running successfully, i.e. not crashing with an error, and 2) giving the "correct" result. The idea of reproducibility dictates that each run of a process wit the same inputs should give exactly the same output, ideally verifiable with a checksum (MD5, SHA1, etc.). It seems your processes ran correctly, but if the output files failed the checksum test, the overall results could still be "fail". If there is some variable component in your test output (e.g. a time/date, or a sort order difference, etc.), you would either have to transform the output before calculating the checksum or tell the test to skip/ignore a particular checksum.