hi there, just wondering is there a way to `clean ...
# tools
t
hi there, just wondering is there a way to
clean up all cache
after running nf-core tools?
Copy code
rm -fr /home/ubuntu/.config/nfcore/
rm -rf /home/ubuntu/.nextflow/nf-core
and still seem to have some caches somewhere.
m
what error do you see?
t
when I try to run install anything (modules/subworkflows) I always see
Copy code
ERROR    'imaging_generatecompanion'
which is a modules from my repo
-g https://github.com/cellgeni/nf-modules.git
the one below is not even pointing to our repo, but using the canonical one using this command line
nf-core --verbose subworkflows install fastq_align_mapad
I still see that error..
Copy code
,--./,-.
          ___     __   __   __   ___     /,-._.--~\ 
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 3.2.0 - https://nf-co.re


Pulling from 'nf-core/modules' (https://github.com/nf-core/modules.git) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Waiting for responseDEBUG    Popen(['git', 'version'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, stdin=None, shell=False,                  cmd.py:1253
         universal_newlines=False)                                                                                                       
DEBUG    Popen(['git', 'fetch', '-v', '--progress', '--', 'origin'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules,         cmd.py:1253
         stdin=None, shell=False, universal_newlines=False)                                                                              
Pulling from 'nf-core/modules' (https://github.com/nf-core/modules.git) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Waiting for responseDEBUG    Popen(['git', 'cat-file', '--batch-check'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, stdin=<valid stream>,   cmd.py:1253
         shell=False, universal_newlines=False)                                                                                          
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, stdin=None, shell=False,       cmd.py:1253
         universal_newlines=False)                                                                                                       
DEBUG    Popen(['git', 'merge', 'origin/master'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, stdin=None, shell=False,   cmd.py:1253
         universal_newlines=False)                                                                                                       
DEBUG    Got '/home/ubuntu/.config/nfcore/nf-core/modules' as path                                                           utils.py:280
DEBUG    Found a config cache, loading: /home/ubuntu/.nextflow/nf-core/wf-config-cache-9f4bce35ba6f9f759cbf54fa7.json        utils.py:314
DEBUG    Using config file: /home/ubuntu/.config/nfcore/nf-core/modules/.nf-core.yml                                        utils.py:1385
DEBUG    Got '.' as path                                                                                                     utils.py:280
DEBUG    No config cache found                                                                                               utils.py:321
DEBUG    Running command: nextflow config -flat .                                                                            utils.py:364
DEBUG    Saving config cache: /home/ubuntu/.nextflow/nf-core/wf-config-cache-41423192c921f4ec0853f1317.json                  utils.py:354
DEBUG    Using config file: .nf-core.yml                                                                                    utils.py:1385
ERROR    'imaging_generatecompanion'
so, I was wondering this module name (
imaging_generatecompanion
)must be cached somewhere...
m
is there a line number for the last error?
also what does your
.nf-core.yml
look like?
t
line number is
commands_subworkflows.py:201
and nf-core.yaml is vanilla
Copy code
repository_type: pipeline
also, this (imaging_generatecompanion) is a module but not a subworkflow...
j
do you have this module installed in your pipeline? if so, can you share the path to it and the content of
modules.json
? It should be under your org directory
t
yes, it's installed now.
Copy code
{
  "name": "",
  "homePage": "",
  "repos": {
    "https://github.com/cellgeni/nf-modules.git": {
      "modules": {
        "sanger": {
          "imaging/generatecompanion": {
            "branch": "main",
            "git_sha": "5fa8a76629029050df04118851a3d93bee48e544",
            "installed_by": [
              "modules"
            ]
          },
          "imaging/preprocess": {
            "branch": "main",
            "git_sha": "5fa8a76629029050df04118851a3d93bee48e544",
            "installed_by": [
              "modules"
            ]
          }
        }
      },
      "subworkflows": {
        "sanger": {
          "stitch": {
            "branch": "main",
            "git_sha": "2ecc2cc028337abebe6b8938323ba845d9ac8a4d",
            "installed_by": [
              "subworkflows"
            ]
          }
        }
      }
    }
  }
}
and all the files seems to be fine
Copy code
modules
└── sanger
    └── imaging
        ├── generatecompanion
        │   ├── environment.yml
        │   ├── main.nf
        │   ├── meta.yml
        │   ├── resources
        │   │   └── usr
        │   │       └── bin
        │   │           └── generate_companion.py
        │   └── tests
        │       └── main.nf.test
        └── preprocess
            ├── environment.yml
            ├── main.nf
            ├── meta.yml
            ├── resources
            │   └── usr
            │       └── bin
            │           └── process.py
            └── tests
                └── main.nf.test
but I had to manually install the two modules and then the subworkflows. If I recall correctly, running
nf-core subworkflows install xx
will automatically install the relevant
modules
, right? Or am I hallucinating?
m
correct
t
I've been testing a lot of options. But if I recall correctly, what I did was in this order 1. manually install two modules with nf-core modules install 2. test with local main.nf 3. created a subworkflow with local main,nf 4. push to our remote 5. delete two local modules 6. try to install with nf-core subworkflows install and then the error persists...
the nf-core subworkflows isntall still download the subworkflows. but never managed to download the modules automatically
maybe somewhere with the parent folder? when I tried to switch to a completely different folder, which is not a git repo. It seems to complain. but it didn't with my current folder, which is also not a git repo.
j
everything looks good with the module
can you try to delete these
Copy code
~/.cache/nfcore
~/.cache/nf-core
~/.config/nfcore
~/.config/nf-core
t
unfortunately, still the same with
nf-core --verbose subworkflows install fastq_align_mapad
Copy code
DEBUG    Popen(['git', 'version'], cwd=/lustre/scratch127/cellgen/cellgeni/tl10/temp/nf-modules_pipeline_test, stdin=None,    cmd.py:1253
         shell=False, universal_newlines=False)                                                                                          
DEBUG    Popen(['git', 'clone', '-v', '--progress', '--', 'https://github.com/nf-core/modules.git',                           cmd.py:1253
         '/home/ubuntu/.config/nfcore/nf-core/modules'],                                                                                 
         cwd=/lustre/scratch127/cellgen/cellgeni/tl10/temp/nf-modules_pipeline_test, stdin=None, shell=False,                            
         universal_newlines=True)                                                                                                        
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, stdin=None, shell=False,       cmd.py:1253
         universal_newlines=False)                                                                                                       
DEBUG    Got '/home/ubuntu/.config/nfcore/nf-core/modules' as path                                                           utils.py:280
DEBUG    Found a config cache, loading: /home/ubuntu/.nextflow/nf-core/wf-config-cache-9f4bce35ba6f9f759cbf54fa7.json        utils.py:314
DEBUG    Using config file: /home/ubuntu/.config/nfcore/nf-core/modules/.nf-core.yml                                        utils.py:1385
DEBUG    Got '.' as path                                                                                                     utils.py:280
DEBUG    No config cache found                                                                                               utils.py:321
DEBUG    Running command: nextflow config -flat .                                                                            utils.py:364
DEBUG    Saving config cache: /home/ubuntu/.nextflow/nf-core/wf-config-cache-83d72a8175c14fcf070d5cd13.json                  utils.py:354
DEBUG    Using config file: .nf-core.yml                                                                                    utils.py:1385
ERROR    'imaging_generatecompanion'                                                                         commands_subworkflows.py:201
😞 1
j
what happens if you try to remove the imaging_generatecompanion moduel with nf-core modules remove?
t
removed
Copy code
INFO     Removed files for 'imaging/generatecompanion' and its dependencies 'imaging/generatecompanion'.
and still the same
Copy code
Pulling from 'nf-core/modules' (https://github.com/nf-core/modules.git) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Waiting for responseDEBUG    Popen(['git', 'version'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, stdin=None, shell=False,                  cmd.py:1253
         universal_newlines=False)                                                                                                       
DEBUG    Popen(['git', 'fetch', '-v', '--progress', '--', 'origin'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules,         cmd.py:1253
         stdin=None, shell=False, universal_newlines=False)                                                                              
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, stdin=None, shell=False,       cmd.py:1253
         universal_newlines=False)                                                                                                       
DEBUG    Popen(['git', 'merge', 'origin/master'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, stdin=None, shell=False,   cmd.py:1253
         universal_newlines=False)                                                                                                       
DEBUG    Got '/home/ubuntu/.config/nfcore/nf-core/modules' as path                                                           utils.py:280
DEBUG    Found a config cache, loading: /home/ubuntu/.nextflow/nf-core/wf-config-cache-9f4bce35ba6f9f759cbf54fa7.json        utils.py:314
DEBUG    Using config file: /home/ubuntu/.config/nfcore/nf-core/modules/.nf-core.yml                                        utils.py:1385
DEBUG    Got '.' as path                                                                                                     utils.py:280
DEBUG    Found a config cache, loading: /home/ubuntu/.nextflow/nf-core/wf-config-cache-83d72a8175c14fcf070d5cd13.json        utils.py:314
DEBUG    Using config file: .nf-core.yml                                                                                    utils.py:1385
ERROR    'imaging_generatecompanion'
😱 1
what is the best practice to start from a new folder? I can give it another try with a clean folder
j
I will try to reproduce this and see if it's a bug but I don't have more time today
you can try to create a fresh pipeline with
nf-core pieplines create
then try to install the subworkflows
t
will try and update here. Thanks both! @Júlia Mir @mashehu
🤞🏻 1
Copy code
nf-core --verbose subworkflows -g https://github.com/cellgeni/nf-modules.git install stitch
good and bad news:
different error
it somehow thinks
imaging_generatecompanion
is a subworkflow. but it isn't.
Copy code
,--./,-.
          ___     __   __   __   ___     /,-._.--~\ 
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 3.2.0 - https://nf-co.re


Pulling from 'cellgeni/nf-modules' (https://github.com/cellgeni/nf-modules.git) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Waiting for responseDEBUG    Popen(['git', 'version'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False,              cmd.py:1253
         universal_newlines=False)                                                                                                       
DEBUG    Popen(['git', 'fetch', '-v', '--progress', '--', 'origin'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules,     cmd.py:1253
         stdin=None, shell=False, universal_newlines=False)                                                                              
DEBUG    Popen(['git', 'checkout', 'main'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False,     cmd.py:1253
         universal_newlines=False)                                                                                                       
DEBUG    Popen(['git', 'merge', 'origin/main'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False, cmd.py:1253
         universal_newlines=False)                                                                                                       
DEBUG    Got '/home/ubuntu/.config/nfcore/cellgeni/nf-modules' as path                                                       utils.py:280
DEBUG    No config cache found                                                                                               utils.py:321
DEBUG    Running command: nextflow config -flat /home/ubuntu/.config/nfcore/cellgeni/nf-modules                              utils.py:364
DEBUG    Could not open /home/ubuntu/.config/nfcore/cellgeni/nf-modules/main.nf to look for parameter declarations - [Errno  utils.py:346
         2] No such file or directory: '/home/ubuntu/.config/nfcore/cellgeni/nf-modules/main.nf'                                         
DEBUG    Using config file: /home/ubuntu/.config/nfcore/cellgeni/nf-modules/.nf-core.yml                                    utils.py:1385
DEBUG    Could not find a config file in the directory '.'                                                                  utils.py:1328
WARNING  'repository_type' not defined in .nf-core.yml                                                             components_utils.py:44
DEBUG    Using selector: EpollSelector                                                                              selector_events.py:54
? Is this repository a pipeline or a modules repository? Pipeline
INFO     To avoid this prompt in the future, add the 'repository_type' key to your .nf-core.yml file.              components_utils.py:55
? Would you like me to add this config now? [y/n] (y): y
INFO     Config added to '.nf-core.yml'                                                                            components_utils.py:59
WARNING  Could not find a 'main.nf' or 'nextflow.config' file in '.'                                            components_command.py:107
INFO     The 'modules.json' file is not up to date. Recreating the 'modules.json' file.                               modules_json.py:628
DEBUG    Got '.' as path                                                                                                     utils.py:280
DEBUG    No config cache found                                                                                               utils.py:321
DEBUG    Running command: nextflow config -flat .                                                                            utils.py:364
DEBUG    Could not open main.nf to look for parameter declarations - [Errno 2] No such file or directory: 'main.nf'          utils.py:346
? Can't find a ./modules directory. Would you like me to create one? [y/n] (y): y
INFO     Creating ./modules directory in '.'                                                                           modules_json.py:85
fatal: not a git repository (or any of the parent directories): .git
DEBUG    Not in a git repository, skipping pre-commit hook.                                                             lint_utils.py:118
DEBUG    Popen(['git', 'checkout', 'main'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False,     cmd.py:1253
         universal_newlines=False)                                                                                                       
DEBUG    Popen(['git', 'checkout', 'main'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False,     cmd.py:1253
         universal_newlines=False)                                                                                                       
DEBUG    Popen(['git', 'checkout', 'main'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False,     cmd.py:1253
         universal_newlines=False)                                                                                                       
DEBUG    Popen(['git', 'cat-file', '--batch-check'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=<valid        cmd.py:1253
         stream>, shell=False, universal_newlines=False)                                                                                 
DEBUG    Popen(['git', 'rev-list', '--max-count=1', '432a2f50df2cc2f68c149df94606a2254be35fbb', '--',                         cmd.py:1253
         'subworkflows/sanger/stitch'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False,                    
         universal_newlines=False)                                                                                                       
DEBUG    Popen(['git', 'cat-file', '--batch'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=<valid stream>,     cmd.py:1253
         shell=False, universal_newlines=False)                                                                                          
INFO     Installing 'stitch'                                                                                               install.py:131
DEBUG    Installing subworkflows 'stitch' at modules hash 2ecc2cc028337abebe6b8938323ba845d9ac8a4d from                    install.py:132
         https://github.com/cellgeni/nf-modules.git                                                                                      
DEBUG    Popen(['git', 'checkout', '2ecc2cc028337abebe6b8938323ba845d9ac8a4d'],                                               cmd.py:1253
         cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False, universal_newlines=False)                         
DEBUG    Popen(['git', 'checkout', 'main'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False,     cmd.py:1253
         universal_newlines=False)                                                                                                       
WARNING  Could not find a 'main.nf' or 'nextflow.config' file in '.'                                            components_command.py:107
DEBUG    Popen(['git', 'checkout', 'main'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False,     cmd.py:1253
         universal_newlines=False)                                                                                                       
ERROR    Subworkflow 'imaging_generatecompanion' not found in available subworkflows                                       install.py:211
DEBUG    entering code: StateBlock(line=0,level=0,tokens=0), 0, 1, False                                                       code.py:10
DEBUG    entering fence: StateBlock(line=0,level=0,tokens=0), 0, 1, False                                                     fence.py:10
DEBUG    entering blockquote: StateBlock(line=0,level=0,tokens=0), 0, 1, False                                           blockquote.py:13
DEBUG    entering hr: StateBlock(line=0,level=0,tokens=0), 0, 1, False                                                           hr.py:14
DEBUG    entering list: StateBlock(line=0,level=0,tokens=0), 0, 1, False                                                      list.py:102
DEBUG    entering reference: StateBlock(line=0,level=0,tokens=0), 0, 1, False                                             reference.py:10
DEBUG    entering html_block: StateBlock(line=0,level=0,tokens=0), 0, 1, False                                           html_block.py:35
DEBUG    entering heading: StateBlock(line=0,level=0,tokens=0), 0, 1, False                                                 heading.py:13
DEBUG    entering lheading: StateBlock(line=0,level=0,tokens=0), 0, 1, False                                               lheading.py:10
DEBUG    entering paragraph: StateBlock(line=0,level=0,tokens=0), 0, 1, False                                             paragraph.py:10
╭─ info ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                                       │
│ Use the command nf-core subworkflows list to view available subworkflows.                                                             │
│                                                                                                                                       │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ERROR                                                                                                                     __main__.py:132
m
could it be because of the underscore in the name?
t
good point. I think I manually wrote those proceses. giving it a try now
nope. Same error..
ERROR    Subworkflow 'imaginggeneratecompanion' not found in available subworkflows
I've tried with another machine and got the same error as above. It tries to find this "module" (
imaginggeneratecompanion
) in the subworkflows. This is maybe not a caching issue but the
tools
issue?
tried earlier version of
tools==2.14.0
as well.. slightly different error.
Copy code
Pulling from 'cellgeni/nf-modules' (https://github.com/cellgeni/nf-modules.git) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Waiting for responseDEBUG    Popen(['git', 'version'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False, universal_newlines=False)                                                                                                          cmd.py:1253
DEBUG    Popen(['git', 'fetch', '-v', '--progress', '--', 'origin'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False, universal_newlines=True)                                                                         cmd.py:1253
DEBUG    Popen(['git', 'checkout', 'main'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False, universal_newlines=False)                                                                                                 cmd.py:1253
DEBUG    Popen(['git', 'merge', 'origin/main'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False, universal_newlines=False)                                                                                             cmd.py:1253
DEBUG    Using config file: /home/ubuntu/.config/nfcore/cellgeni/nf-modules/.nf-core.yml                                                                                                                                                          utils.py:1051
DEBUG    Using config file: .nf-core.yml                                                                                                                                                                                                          utils.py:1051
ERROR    'imaginggeneratecompanion'                                                                                                                                                                                                            __main__.py:1711
I am now very confused💀🫠
I have a feeling that it might be our bespoke remote repo structure. Do you have a guideline of how to set-up a remote similar to nf-core
modules
repo? I thought all we need is the
.nf-core.yml
+ empty
modules
and
subworkflows
folders. Has that changed?
OK, I am pretty sure it's a
caching
issue now. I've first cleaned up the
known
caching locations
Copy code
rm -fr ~/.cache/nfcore
rm -fr ~/.config/nfcore
rm -fr ~/.cache/nf-core
rm -fr ~/.config/nf-core
rm -rf ~/.nextflow/nf-core
And I run
nf-core --verbose modules list local
, where it has nothing to do with our own repo (https://github.com/cellgeni/nf-modules/). And I always see the same error.
Copy code
Pulling from 'nf-core/modules' (https://github.com/nf-core/modules.git) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Waiting for responseDEBUG    Popen(['git', 'version'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)     cmd.py:1253
DEBUG    Popen(['git', 'fetch', '-v', '--progress', '--', 'origin'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, stdin=None,          cmd.py:1253
         shell=False, universal_newlines=True)
DEBUG    Popen(['git', 'checkout', 'master'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, stdin=None, shell=False,                    cmd.py:1253
         universal_newlines=False)
DEBUG    Popen(['git', 'merge', 'origin/master'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, stdin=None, shell=False,                cmd.py:1253
         universal_newlines=False)
DEBUG    Got '/home/ubuntu/.config/nfcore/nf-core/modules' as path                                                                        utils.py:280
DEBUG    No config cache found                                                                                                            utils.py:321
DEBUG    Running command: nextflow config -flat /home/ubuntu/.config/nfcore/nf-core/modules                                               utils.py:364
DEBUG    Saving config cache: /home/ubuntu/.nextflow/nf-core/wf-config-cache-9f4bce35ba6f9f759cbf54fa7.json                               utils.py:354
DEBUG    Using config file: /home/ubuntu/.config/nfcore/nf-core/modules/.nf-core.yml                                                     utils.py:1385
DEBUG    Got '.' as path                                                                                                                  utils.py:280
DEBUG    No config cache found                                                                                                            utils.py:321
DEBUG    Running command: nextflow config -flat .                                                                                         utils.py:364
DEBUG    Saving config cache: /home/ubuntu/.nextflow/nf-core/wf-config-cache-af790a9274bf56bcfa3e1bd60.json                               utils.py:354
DEBUG    Using config file: .nf-core.yml                                                                                                 utils.py:1385
INFO     Repository type: pipeline                                                                                                          list.py:85
ERROR    'imaginggeneratecompanion'                                                                                             commands_modules.py:48
j
I am trying to debug this
🙏 1
would it be easy for you to try to rename your module to IMAGING_GENERATECOMPANION
with underscore
t
sure.
j
we always use the pattern <tool>/<subtool> for directories and <tool>_<subtool> for the module name, and I suspect we don't check the actual name of the module
👍 1
t
should be ok now
j
aha! now it worked for me
can you install the subworkflow now?
t
and yes! it works for me as well! I think the error from tomorrow was fixed here: https://github.com/cellgeni/nf-modules/commit/2498afd9050b1c85cab9115e13a40880cd7db58b and indeed we need the
_
as you just pointed out.
🥳 1
many thanks for your help!