Current state of the pipeline • All merged modules are added to our pipeline (PR) ◦ added

htodemux

,

multiseq

, and

hasheddrops

to the pipeline ◦ added the preprocessing script for

htodemux

and

multiseq

in the local folder ◦ added a visualization script for

htodemux

in the local folder • Updated

htodemux

(PR),

multiseq

(PR),

hasheddrops

(PR), and

bff

(PR). I mainly fixed the data types so that they fit with the pipeline. • Due to the memory limit, we haven’t been able to merge bff. What can we do to finally merge this module? • What is the status of the remaining modules that haven’t been merged yet? (See progress table)

Thank you very much for the excellent summary! @Nico Trummer could you please help with the two questions in @Luis Heinzlmeier PR? I think they're nf-core module specific and I don't know the answers.

• Due to the memory limit, we haven’t been able to merge bff. What can we do to finally merge this module?

Memory limit in Github Actions?

I made sure the CI passes in the pipeline PR (except the readthedocs build, but that will probably stick around until the repo is moved to nf-core)

I can disable RTD builds for PRs but it doesn't hurt to just keep it on till it's moved

Thank you very much, @Nico Trummer! I'm currently trying to get the pipeline running with

test.config

! Unfortunately, the newer Seurat version doesn't allow

genes.tsv

anymore. @Nico Trummer could you maybe change the name

genes.tsv

to

features.tsv

.

Error in Read10X(data.dir = rna_matrix, gene.column = gene_col) : 
  Gene name or features file missing. Expecting features.tsv.gz

Container:
  <http://community.wave.seqera.io/library/r-seurat:5.3.0--eeb977835038859a|community.wave.seqera.io/library/r-seurat:5.3.0--eeb977835038859a>

You can directly push changes to the

hadge

branch of #C02L5UB4Y9G But probably it would be more user-friendly to rename in this case

Thank you! I have now also added a small process for renaming

genes.tsv

to

feature.tsv

. However, the files don't seem to match:

(base) luisheinzlmeier@MBC02FF4G5Q6LR hadge % zcat results_test/untar/test3_hto/features.tsv.gz       
MS-11   MS-11
MS-12   MS-12
(base) luisheinzlmeier@MBC02FF4G5Q6LR hadge % zcat results_test/untar/test3_rna/genes.tsv.gz | head   
PGBD2   PGBD2
LOC114674980    LOC114674980
TRNAE-CUC TRNAE-CUC
ZNF692  ZNF692
TRNAL-CAA TRNAL-CAA
LOC114678402    LOC114678402
ZNF672  ZNF672
LOC106995359    LOC106995359
SH3BP5L SH3BP5L

I used these files for testing:

(base) luisheinzlmeier@MBC02FF4G5Q6LR ~ % zcat /Users/luisheinzlmeier/Desktop/hadge/test_data_dev/rna/features.tsv.gz  
ENS_ID	MS-1-GGAGAAGA	Gene Expression
ENS_ID	MS-2-CCACAATG	Gene Expression
ENS_ID	MS-3-TGAGACCT	Gene Expression
ENS_ID	MS-4-GCACACGC	Gene Expression
ENS_ID	MS-5-AGAGAGAG	Gene Expression
ENS_ID	MS-6-TCACAGCA	Gene Expression
ENS_ID	MS-7-GAAAAGGG	Gene Expression
ENS_ID	MS-8-CGAGATTC	Gene Expression
ENS_ID	MS-9-GTAGCACT	Gene Expression
ENS_ID	MS-10-CGACCAGC	Gene Expression
ENS_ID	MS-11-TTAGCCAG	Gene Expression
ENS_ID	MS-12-GGACCCCA	Gene Expression
ENS_ID	MS-13-CCAACCGG	Gene Expression
ENS_ID	MS-14-TGACCGAT	Gene Expression
ENS_ID	MS-15-GCAACGCC	Gene Expression
ENS_ID	MS-16-CAATCGGT	Gene Expression
ENS_ID	MS-17-ATAGCGTC	Gene Expression
ENS_ID	MS-18-GAATCTCG	Gene Expression
ENS_ID	MS-19-CTAGCTGA	Gene Expression
ENS_ID	MS-20-AGACCTTG	Gene Expression
ENS_ID	MS-21-GAAGGAAG	Gene Expression
ENS_ID	MS-22-CTACGACA	Gene Expression
ENS_ID	MS-23-AGAAGAGG	Gene Expression
ENS_ID	MS-24-GTACGCAT	Gene Expression
ENS_ID	unmapped	Gene Expression
(base) luisheinzlmeier@MBC02FF4G5Q6LR ~ % zcat /Users/luisheinzlmeier/Desktop/hadge/test_data_dev/hto/features.tsv.gz 
ENS_ID	MS-11	Antibody Capture
ENS_ID	MS-12	Antibody Capture

I think everyone who is in the nf-core organization should have direct push access here, but if something does not work for you, you can send the file to me and i will upload for you

@Luis Heinzlmeier you should be able to make a PR. Direct pushes might be disabled and this warning message hard to understand

Thank you! I created a PR with the updated data sets. Could someone please merge them? (link to PR)

Done! Still surprised that you don't seem to have permission to do that. If this continues to be an issue and prevents you from contributing, we'll ping some people

I think it is because luis is part of the nf-core github community, but not part of the standard team

nf-core/contributors

Thanks for the help! I received the new invite. When I run the pipeline locally, all of my modules pass. When I tried to run the pipeline with Docker on CodeSpaces, this error came up:

Command error:
    docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
    See 'docker run --help'.

Then I updated the snapshot with conda or singularity using these commands:

nextflow run .  -profile test,singularity --outdir test

or

nf-test test tests/default.nf.test  --update-snapshot --profile test,conda

or

nf-test test tests/default.nf.test  --update-snapshot --profile test,singularity

Unfortunately, I still haven't passed the CI. @Nico Trummer, why are the snapshots in the CI still showing a difference? How can I debug such a problem?

Docker is not installed in the nf-core CodeSpaces, you can use apptainer instead

The hashes are different between Codespaces and the CI. I pushed the snapshot from Codespaces. Which outputs should I compare in

.nf-test

? Can I read something precise from the error in the CI? Or can I read something concrete from the difference in the hashes? E.g. (same file but different hashes) from here:

java.lang.RuntimeException: Different Snapshot:

--- /tmp/expected4709676898786390216jso +++ /tmp/found219066029070260155json   
n                                                                              
@@ -273,144 @@                          @@ +273,144 @@                         
  273         "untar/test3_rna/matrix.m   273         "untar/test3_rna/matrix.m
tx.gz"                                  tx.gz"                                 
  274     ],                              274     ],                           
  275     [                               275     [                            
! 276         "test1.base.vcf.gz:md5,c3 ! 276         "test1.base.vcf.gz:md5,36
81d7e92163710dfca2306d0d9b73f1",        eccb4bd3e2130f395985e09a66a60d",       
! 277         "test1.cells.vcf.gz:md5,9 ! 277         "test1.cells.vcf.gz:md5,b
198bde348a0dd6a40ddee8a4cc4104c",       28d8acaf17777c7cf9e2c7d01d93a66",

The commands I used to update are correct, aren't they?

Nf-test allows you to test for consistency of any file attributes, snapshot is just the quickest way for testing if something is completely the same. You can also compare the entire file content for equality, then the file content would also be shown in the CI log (which will proabably be hardly readable if the file is large). Alternatively there is also the `nft-vcf` nf-test plugin, which allows you to test for certain attributes of VCF files.

I can find the content of, for example,

test1.base.vcf.gz

in the CI log? If so, how do I find it? I couldn't find it 😅 Thanks for the plugin, I'll check it out!

The CI shows everything you test for. If you test for hashes, it will show only the hashes. It you test for text content equality, then it will show the text content (but with gzipped files it won't work well probably). But you can test for other file attributes, such as size, text content, or VCF-specific attributes using the plugin. But you need to adjust the test first, and then you will see it

Thanks Nico! I integrated the plugin and tried to run a few tests. But I am still not sure if I am on the right track. Not really sure what file attributes I could/should check that might differ between using the test on Codespaces and the CI. The file

test1.base.vcf.gz

also only was an example, also the checksum from other files differ. Maybe we can use a few minutes do discuss that on Monday. If you have any resources discussing my issue "Tests/snapshots pass on Codespaces/locally but not for the CI". Like an article, a discussion or video, I would be happy to go through these. I really would like to learn solving this myself. And I am probably not the first one having such problems.

The core of the problem is not related to nf-core or nextflow at all, we just pay a lot of attention to these things for the sake of reproducibility. But the behavior you observe means that the tool produces different output on different devices. If nextflow is used as a wrapper or not does not matter in that sense. As the problem is tool-specific, there are no general resources for this. The solution generally is getting a better understanding of how the tool behaves on different devices, and what parts of the output show slight alterations. Once one understood this, one can either ignore the problematic parts from the tests, or try to find other solutions. For example I encountered situations where the order of columns in a dataframe was not consistent, which could be solved by alphabetic sorting. In another situation floating point numbers were equal in the first 9 decimal places, but were different afterwards. This could be solved by rounding.