Bring hadge to nf-core Thank you for the review, <...
# hadge
l
Bring hadge to nf-core Thank you for the review, @Nico Trummer and @Lukas Heumos! I have incorporated your feedback, and all tests are still passing. So what's next?
n
From my perspective: 1. Merge the PR 2. Set default branch to
dev
3. Send a message to #C09H6NYHR9T for moving the repo to the nf-core organization 4. Check if the docs publish correctly 5. Change the CI configuration to use nf-core's larger runners, open a test PR to check if everything works (including the things we previously ignored due to 'disk space' issues) 6. Clean up the branch structure 7. Solve any open issues that would be classified as 'bugs', for feature issues decide what should be done before the 1.0 release and what can be done after. Create milestones and assign accordingly, to keep track 8. I will look at the entire pipeline again and check concordance with latest nf-core guidelines - there have been some recent developments in relation to the nextflow
strict
mode, see #C04QR0T3G3H 9. 1.0 release
❀️ 1
l
All right, thank you. I'll try to go through it step by step. If I need help, I'll get in touch with you.
Unfortunately, I cannot change the default branch because I don’t have sufficient permissions for this repository. @Lukas Heumos, could you grant me the necessary access?
Send a message to #C09H6NYHR9T βœ…
l
You're now admin
l
Thank you!
Unfortunately, I don't have permission to transfer the repository.
l
Oh sorry, I somehow didn't receive this ping
Likely only theislab admins can do that. Luckily I am one. I'll transfer
πŸ˜… 1
Hmm, not even I can do that. This might be a global org setting
Okay I went full rodeo now
FIrst to myself and now to mashehu
@mashehu you should have the repository now
πŸ‘ 1
n
Crazy stuff
πŸ˜‚ 1
The docs have become available now: https://nf-co.re/hadge/dev/
πŸ₯³ 1
So steps 1-4 from my list are finished
Does someone of you have access to edit the 'About' section in the repo? If yes, could you change the docs link to the nf-core docs? If not, Mashehu can give you elevated repo permissions that allow this (and also other things like configuring branch protection) Usually the primary maintainers of a pipeline have these permissions, in this case probably both Luis and Lukas should have it
quick look 1
m
i fixed the link
i will also add branch protection rules, so you might see some smaller changes when you try to merge things
n
Thanks! For your reference, the
master
branch is currently the proper nf-core release branch, the
main
branch was the default branch before we re-built everything for nf-core
l
Pretty smooth so far. Good job @Luis Heinzlmeier and @Nico Trummer!
m
okay, that branch duality might confuse some of the tooling. it is totally up to you if you want to use
main
or
master
, but there can only be one (you can just add a
_old
prefix to one or something similar)
l
Let's go with
main
then. More modern for better or worse
πŸ‘ 1
πŸ‘πŸ» 1
m
be sure to also set that in the
nextflow.config
πŸ‘ 2
l
I added nf-core's larger runners, but now the CI is skipping
Run nf-test
. Have I forgotten anything?
n
No, it's only running nf-test if changes have been made to relevant files
If you change something in a module, only this module (and any subworkflows that use this module) will be tested If you want to run all tests, you can change one of these files - just add a random modification and remove it after it was executed
l
Thank you, Nico! And we want to trigger running the test again with the new CI configuration, don't we?
n
Yes, also be aware that you can configure the disk volume like this
πŸ‘ 1
You will notice that the tests with
latest-everything
finished very quickly with a tick. But if you open the logs you will see that it actually failed. This is because the
CI
is configured to not fail on
latest-everything
, but only show you if your pipeline will work with future releases or not The failure here is associated with nextflow's strict syntax (which I already mentioned in point 8 of my list at the top). So far, complying with the strict syntax was very optional, but it will be enforced in future releases. As I've already participated in doing this for #C077PUX8DBP, I can also do this for #C067K2P6GUV
πŸ‘ 1
l
Ouhh, I see! Thank you for your help with this Nico. πŸ™
n
I created a 1.0 release review issue on github which is a bit more detailed than the list on the beginning of this thread
l
Great. I just thought through the task of cleaning up the branch structure and came across a warning. The template version is
nf_core_version: 3.3.2
but my
nf-core, version 3.5.1
. Should I then run
pip install --upgrade nf-core==3.3.2
or do we first want to update
TEMPLATE
? In the tutorial, they say
master
is the default. In our case, I keep
dev
as the default, don't I? So, can I then also follow the tutorial when
dev
is the default branch?
n
dev
remains the default until the 1.0 release (as it is the only branch that actually contains the pipeline logic until the 1.0 release PR is merged) I think this tutorial does actually mostly not apply to us, it looks like it is targeted on pipelines that already have been released. For our use case, it should be sufficient to rename the branches and change the
nextflow.config
πŸ‘ 1
And don't worry about the template stuff for now, I will take care of this over the weekend
πŸ‘ 1
l
@Lukas Heumos β€’ Do we want people who use the pipeline to cite it via Zenodo, bioRxiv, or Genome Biology? (The nf-core template suggests to cite with Zenodo, so I did both for now: Zenodo and Genome Biology) β€’ There was a recommendation to add many keywords to the repo in the "new pipeline proposal thread" in November 2023. Concering hadge, we only have a few keywords now. Do you have any ideas for more keywords? How do you like my keyword ideas? β—¦ hashing-based-deconvolution β—¦ genotype-based-deconvolution β—¦ donor-deconvolution β—¦ cell-hashing β—¦ genetic-demultiplexing β—¦ donor-assignment β—¦ hto β—¦ hashtag-oligonucleotides β—¦ lipid-tagging β—¦ Maybe the integrated tools? vireo, demuxlet, freemuxlet, souporcell, scsplit, htodemux, multiseq, hasheddrops, demuxem, hashsolo, gmm-demux
l
Do we want people who use the pipeline to cite it via Zenodo, bioRxiv, or Genome Biology? (The nf-core template suggests to cite with Zenodo, so I did both for now: Zenodo and Genome Biology)
Only Genome biology
β—¦ hashing-based-deconvolution
β—¦ genotype-based-deconvolution
They're a bit complex because of the
-based
, While not wrong, I'm not sure whether it helps with search. Given it's keywords, we might just scrap the
-based
. The rest look good!
πŸ‘ 1
If we have space for more, we can also add the tools, yeah
l
And I just saw that I don't have the permission to rename the
main
branch. πŸ˜‚ So @Lukas Heumos could you maybe rename the
main
branch to
legacy-main
or to whatever name you like :)
m
i can do that
or better, gave you admin rights for now
πŸ‘ 1
l
Also, I cannot rename
master
πŸ˜…
It worked now. Thank you for the quick help!
I am gonna run
test_full
on our cluster. To push changes to the test snapshots from the cluster to the repo, I need a fine-grained token. @mashehu could you approve my token called
HGMU Cluster
?
I am gonna run
test_full
on our cluster. To push changes to the test snapshots from the cluster to the repo, I need a fine-grained token. @mashehu could you approve my token called
HGMU Cluster
?
resolved
I am currently running hadge with a real-life dataset included in
test_full
. The hashing part already works! The genetic part has been running for about 7 hours now (includes a 50GB bam file). @Nico Trummer, if running
test_full
works on our cluster, how should I include it in the CI? You also mentioned that we could test running hadge in the cloud.
nf-core/scrnaseq
included
test_full
into
.github/workflows/awsfulltest.yml
. Is this the way we want to do it?
n
It's not could test in the cloud but actually having these tests and running them in the cloud is part of the standard release process I am myself no expert in setting up and running the AWS tests, I know there have been some issues with running the tests in the last half year, but I think/hope they are fixed by now. To provide your data for upload, I think you need to use #C09H6NYHR9T. Then the people there can probably also assist you with the remaining steps. You are lucky, as currently the nf-core hackathon is ongoing, so it is quite likely that this will be resolved quite quickly if you initiate it today/tomorrow