https://nf-co.re logo
Join Slack
Powered by
Hi! I'm trying to install de nf-core subworflow di...
# tools
m
Hi! I'm trying to install de nf-core subworflow differential_functional_enrichment and I'm encountering this error. What could it be related to? https://github.com/nf-core/differentialabundance/pull/503
Copy code
nf-core subworkflows install


                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\ 
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 3.3.2 - <https://nf-co.re>


? Subworkflow name: differential_functional_enrichment
INFO     Installing 'differential_functional_enrichment'                                                                  
ERROR    Module 'decoupler' not found in available modules                                                                
╭─ info ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                        │
│ Use the command nf-core modules list to view available modules.                                                        │
│                                                                                                                        │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ERROR  


nf-core modules list local


                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\ 
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 3.3.2 - <https://nf-co.re>


INFO     Repository type: pipeline                                                                                        
ERROR    'decoupler'
m
It seems you fixed it already? Also mind that there is usually a cached version of the modules, so you may need to renew your cache.
m
Not yet. I have fixed the problem in the subworkflow and remove the cache but it's still failing. I think there is a problem with the dependencies of the subworkflow and the module
m
it could be related to this issues?
Both issues deal with installation from custom sources for modules and subworkflows, other than the canonic nf-core ones. Therefore, I do not think so? Or are you also using another remote than nf-core/modules?
m
no, I'm not
m
I can reproduce your error also with the lasted dev version of #CE5LG7WMB and a new testpipeline, but even the verbose log doesn’t tell me much. Since I am not familiar with that part of the tools suite, I feel unfortunately not competent to help you out there. Both commands simply error out with 
ERROR    'decoupler'
Copy code
nf-core --log-file ./decoupler-subworkflow.txt subworkflows install differential_functional_enrichment

nf-core --log-file decoupler-module.txt modules install decoupler/decoupler
and the log files do not produce anything obviously suspicious to me.
a
The error here is somewhat obvious but I am surprised that it’s able to occur. Basically you have a module 
DECOUPLER
but it’s under a 
<tool>/<subtool>
structure of 
decoupler/decoupler
so the module name needs to be 
DECOUPLER_DECOUPLER
rather than the plain 
DECOUPLER
. I am surprised the module was able to be pushed into the master branch in the current state.
Looking at the original PR to add that module it seems like no CI tests ran at all 🤔. I suspect that might be how this ended up happening.
m
The error here is somewhat obvious but I am surprised that it’s able to occur.
I noticed as well that there was an issue with the name, but felt that there must be more to it since I could not install the module independently of the subworkflow either.
Copy code
nf-core modules install decoupler/decoupler
also failed.
a
What's the error there?
m
The problem is, that one can’t install the module either way, neither with 
decoupler
nor with 
decoupler/decoupler.
I naively assumed the name in the 
meta.yml
of the module would be the identifier, but also tried the path in the modules folder. This is why I gave up, since I do not know how the module installation works in #CE5LG7WMB.
But if no CI ever ran, then maybe the module is broken all the way.
a
Hmm I don’t get that issue installing with the dev version of tools
Or locally with 3.3.2
So not sure where that’s coming from on your end
m
🤔 I was using 3.4dev, but admittedly not the latest version, but used a conda env that was at a stage that I used to review the download PR from Erik, so a few months old. Bad luck it seems.
a
Copy code
nf-core modules install decoupler/decoupler


                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\ 
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 3.3.2 - <https://nf-co.re>


INFO     Installing 'decoupler/decoupler'                                                                                                
INFO     Use the following statement to include this module:                                                                             
                                                                                                                                         
 include { DECOUPLER_DECOUPLER } from '../modules/nf-core/decoupler/decoupler/main'
Except that ironically that 
include
line is incorrect…
m
Copy code
(nf_tools) matthias.zepper@matthiaszmbp nf-core-testpipeline % nf-core -v modules install decoupler/decoupler


                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 3.4.0.dev0 - <https://nf-co.re>


DEBUG    sys.platform='darwin', git_executable='git'                                                                                                                           util.py:494
Pulling from 'nf-core/modules' (<https://github.com/nf-core/modules.git>) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Waiting for responseDEBUG    Popen(['git', 'version'], cwd=/Users/matthias.zepper/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                               cmd.py:1270
DEBUG    Popen(['git', 'fetch', '-v', '--progress', '--', 'origin'], cwd=/Users/matthias.zepper/.config/nfcore/nf-core/modules, stdin=None, shell=False,                       cmd.py:1270
         universal_newlines=True)
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/Users/matthias.zepper/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                    cmd.py:1270
DEBUG    Popen(['git', 'merge', 'origin/master'], cwd=/Users/matthias.zepper/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                cmd.py:1270
DEBUG    Got '/Users/matthias.zepper/.config/nfcore/nf-core/modules' as path                                                                                                  utils.py:281
DEBUG    Found a config cache, loading: /Users/matthias.zepper/.nextflow/nf-core/wf-config-cache-9f4bce35ba6f9f759cbf54fa7.json                                               utils.py:315
DEBUG    Using config file: /Users/matthias.zepper/.config/nfcore/nf-core/modules/.nf-core.yml                                                                               utils.py:1413
DEBUG    Got '.' as path                                                                                                                                                      utils.py:281
DEBUG    Found a config cache, loading: /Users/matthias.zepper/.nextflow/nf-core/wf-config-cache-c793aa7bbd1c9ecc154688e60.json                                               utils.py:315
DEBUG    Using config file: .nf-core.yml                                                                                                                                     utils.py:1413
DEBUG    sys.platform='darwin', git_executable='git'                                                                                                                           util.py:494
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/Users/matthias.zepper/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                    cmd.py:1270
DEBUG    Popen(['git', 'merge', 'origin/master'], cwd=/Users/matthias.zepper/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)                cmd.py:1270
DEBUG    Got '/Users/matthias.zepper/.config/nfcore/nf-core/modules' as path                                                                                                  utils.py:281
DEBUG    Found a config cache, loading: /Users/matthias.zepper/.nextflow/nf-core/wf-config-cache-9f4bce35ba6f9f759cbf54fa7.json                                               utils.py:315
DEBUG    Using config file: /Users/matthias.zepper/.config/nfcore/nf-core/modules/.nf-core.yml                                                                               utils.py:1413
ERROR    'decoupler'
Maybe something with the config caches? Either way, glad that you identified the error.
a
Given I can’t replicate the install (
modules install
) error on any version I have no idea where it’s coming from. I remain fairly certain the subworkflow issue is due to the name being incorrect, because I believe we trace back to the 
<tool>/<subtool>
by parsing the 
TOOL_SUBTOOL
of the include statement
m
So I just installed the latest 
dev
version with 
pip install --upgrade --force-reinstall git+<https://github.com/nf-core/tools.git@dev>
. I deleted all cached files and repeated the install attempt unsuccessfully:
Copy code
(nf_tools) matthias.zepper@matthiaszmbp nf-core-testpipeline % rm -rf /Users/matthias.zepper/.config/nfcore
(nf_tools) matthias.zepper@matthiaszmbp nf-core-testpipeline % rm -rf /Users/matthias.zepper/.nextflow/nf-core
(nf_tools) matthias.zepper@matthiaszmbp nf-core-testpipeline % nf-core -v --log-file decoupler-install-3.4dev modules install decoupler/decoupler
The only explanation I still have left is that the previous installation attempts in my test pipeline directory left the 
modules.json
in some inconsistent state, because I did not erase my local testpipeline so far.
decoupler-install-3.4dev.txt
a
I think it's very likely that the error is related to the modules json
It's quite possibly a key error at some point in the modules json access
So I would look for 
decoupler
in there and retry
m
Thanks. I already trashed the testpipeline, but I will stop here. I spent about an hour troubleshooting this and that is as much time as I can spare for now. @Maria Antonia Saracco should make an issue on the modules directory or try fixing the subworkflow with what you said. Ultimately, a better error message in tools if it is truly a key error in 
modules.json
would also help.
👍 1
a
Yeah I also think maybe we need a "full trace" flag that disables the catch-all error catching we have that is swallowing the trace at that point.
Open in Slack
PreviousNext
Powered by