Hi all, I'm trying to set-up a new bare repository...
# tools
y
Hi all, I'm trying to set-up a new bare repository containing modules/subworkflows that are specific to our lab. As far as I understand, I can use the option ``--git-remote`` to specify another repository . But when I try to test the repo (
nf-core/tools version 3.3.1
) using :
Copy code
nf-core pipelines --git-remote "https://gitlab.univ-nantes.fr/xxx/nf-yyy"  list
I get the following error:
Copy code
No such option: --git-remote
is it just not implemented ? thanks, Pierre
✅ 1
m
nf-core modules
to install
modules
✅ 1
not in
pipelines
y
ok. Should I submit an issue ?
m
no,
nf-core pipelines
should not need this git-remote option
❓ 1
you want your own fork of the nf-core modules repo, right?
so you need the modules and subworkflows subcommands
not the pipelines one
this works:
nf-core modules --git-remote https://github.com/nf-side/modules.git list remote
y
no, I want to create a new empty repo that could contains our modules, subworkflows. My idea was to use the nf-core tool with multiple repo;
Copy code
nf-core module install samtools
nf-core module install --git-remote "https://gitlab.univ-nantes.fr/xxx/nf-yyy" our-amazing-tool
m
ok, so not a fork, but a clone
but that's the same idea
y
not a clone, I wanted to start from an empty repo.
m
ok, I got you
It's what I did
more or less
this is a subset of some selected modules from the nf-core/modules repo
y
ah , I see ok.
m
sorry, I was using bad analogies with
fork
and
clone
y
Ah yes, and I see that I should have used
modules list
, not pipelines. Thanks @Maxime Garcia!
🙏 1
m
yeah, you want your own personal modules repo, so you need the modules/subworkflows subcommands
Anytime Pierre
a
You can see #proj-nf-modules-template which sadly never really got off the ground I think, but there is some info there about setting up a custom remote for modules
✅ 1
p
The repo exists, so it got off the ground. Though there are some questions about it's maintenance
e
Hey! I actually tried to use that repo not for custom modules but for subworkflows, to create a collection of custom subworkflows to work on them there and then pull into my workflows but you cant install nf-core modules on that repo. I get
Copy code
ERROR    You cannot install a None in a clone of nf-core/modules
I’ve seen some workarounds where you manually copy nf-core modules into the repo but it isnt ideal. Is there some way to do this correctly?
p
Please make an issue and I'll also ping <!subteam^S08LAB76NBU> here
a
Cross-repo subworkflows are now supported by tools but not in that way. The code for the external modules is never copied into your remote. Personally I prefer to avoid using remote subworkflows altogether because I think there are too many issues and edge-cases with them in general (not just for cross repo). But cross-repo support exists. You have to define the externally dependent modules with their git path in the meta yml
e
Yes I saw that cross-repo support and I thought this feature would solve this issue 😅. I mean even just having a organisation repo to save your subworkflows of nf-core modules only would be nice. I’ll open an issue in github as Phil suggested. Thanks!
a
But this issue is solved no? It makes no sense to "install" modules into a modules repo. The remotes are just stores for source code of modules/subworkflows and are for a single "organisation" per repo. The correct work pattern with the new cross-repo support is: - store non nf-core modules - store non nf-core subworkflows which can depend on modules from other repos - use nf-core tools as normal to install cross-repo subworkflows into your pipelines. Dependent modules will be pulled from the various different modules remotes as configured in the subworkflow meta.yml If this is the pattern you have tried and you had an issue then that does need looking into. However the key point is that there is no intended or expected support for using
modules install
inside a modules repo and you shouldn't attempt to store module code from another repo in your own custom repo as that is not supported (or expected to be) either
e
Maybe I’m missing something but to create a subworkflow (with nf-core modules) dont these modules need to be present in the same repo under modules? How would I run nf-test for example on that repo to test the subworkflow if the modules arent present?
a
No, that was the whole purpose of providing cross-repo support. But then yes, you have hit upon one of the issues that this leads to - they are not testable via the current modules framework. The most likely option I see for testing (in CI or locally) is: - create a dummy pipeline repo - install the subworkflow and then run the tests
this 1
j
https://nf-co.re/docs/tutorials/external_usage/cross_organisational_subworkflows here is where we have the new docs for mixed subworkflows
e
Ok thanks a lot for your help. I will play around with that try some test subworkflows.