Tong LI
04/24/2025, 2:56 PMclean up all cache after running nf-core tools?
rm -fr /home/ubuntu/.config/nfcore/
rm -rf /home/ubuntu/.nextflow/nf-core
and still seem to have some caches somewhere.mashehu
04/24/2025, 2:59 PMTong LI
04/24/2025, 3:03 PMERROR 'imaging_generatecompanion'
which is a modules from my repo
-g https://github.com/cellgeni/nf-modules.git
the one below is not even pointing to our repo, but using the canonical one
using this command line
nf-core --verbose subworkflows install fastq_align_mapad
I still see that error..
,--./,-.
___ __ __ __ ___ /,-._.--~\
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'
nf-core/tools version 3.2.0 - https://nf-co.re
Pulling from 'nf-core/modules' (https://github.com/nf-core/modules.git) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Waiting for responseDEBUG Popen(['git', 'version'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, stdin=None, shell=False, cmd.py:1253
universal_newlines=False)
DEBUG Popen(['git', 'fetch', '-v', '--progress', '--', 'origin'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, cmd.py:1253
stdin=None, shell=False, universal_newlines=False)
Pulling from 'nf-core/modules' (https://github.com/nf-core/modules.git) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Waiting for responseDEBUG Popen(['git', 'cat-file', '--batch-check'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, stdin=<valid stream>, cmd.py:1253
shell=False, universal_newlines=False)
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, stdin=None, shell=False, cmd.py:1253
universal_newlines=False)
DEBUG Popen(['git', 'merge', 'origin/master'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, stdin=None, shell=False, cmd.py:1253
universal_newlines=False)
DEBUG Got '/home/ubuntu/.config/nfcore/nf-core/modules' as path utils.py:280
DEBUG Found a config cache, loading: /home/ubuntu/.nextflow/nf-core/wf-config-cache-9f4bce35ba6f9f759cbf54fa7.json utils.py:314
DEBUG Using config file: /home/ubuntu/.config/nfcore/nf-core/modules/.nf-core.yml utils.py:1385
DEBUG Got '.' as path utils.py:280
DEBUG No config cache found utils.py:321
DEBUG Running command: nextflow config -flat . utils.py:364
DEBUG Saving config cache: /home/ubuntu/.nextflow/nf-core/wf-config-cache-41423192c921f4ec0853f1317.json utils.py:354
DEBUG Using config file: .nf-core.yml utils.py:1385
ERROR 'imaging_generatecompanion'
so, I was wondering this module name (imaging_generatecompanion)must be cached somewhere...mashehu
04/24/2025, 3:04 PMmashehu
04/24/2025, 3:05 PM.nf-core.yml look like?Tong LI
04/24/2025, 3:05 PMcommands_subworkflows.py:201
and nf-core.yaml is vanilla
repository_type: pipelineTong LI
04/24/2025, 3:08 PMJúlia Mir
04/24/2025, 3:09 PMmodules.json? It should be under your org directoryTong LI
04/24/2025, 3:12 PM{
"name": "",
"homePage": "",
"repos": {
"https://github.com/cellgeni/nf-modules.git": {
"modules": {
"sanger": {
"imaging/generatecompanion": {
"branch": "main",
"git_sha": "5fa8a76629029050df04118851a3d93bee48e544",
"installed_by": [
"modules"
]
},
"imaging/preprocess": {
"branch": "main",
"git_sha": "5fa8a76629029050df04118851a3d93bee48e544",
"installed_by": [
"modules"
]
}
}
},
"subworkflows": {
"sanger": {
"stitch": {
"branch": "main",
"git_sha": "2ecc2cc028337abebe6b8938323ba845d9ac8a4d",
"installed_by": [
"subworkflows"
]
}
}
}
}
}
}
and all the files seems to be fine
modules
└── sanger
└── imaging
├── generatecompanion
│ ├── environment.yml
│ ├── main.nf
│ ├── meta.yml
│ ├── resources
│ │ └── usr
│ │ └── bin
│ │ └── generate_companion.py
│ └── tests
│ └── main.nf.test
└── preprocess
├── environment.yml
├── main.nf
├── meta.yml
├── resources
│ └── usr
│ └── bin
│ └── process.py
└── tests
└── main.nf.test
but I had to manually install the two modules and then the subworkflows.
If I recall correctly, running nf-core subworkflows install xx will automatically install the relevant modules , right? Or am I hallucinating?mashehu
04/24/2025, 3:14 PMTong LI
04/24/2025, 3:15 PMTong LI
04/24/2025, 3:16 PMTong LI
04/24/2025, 3:21 PMJúlia Mir
04/24/2025, 3:30 PMJúlia Mir
04/24/2025, 3:31 PM~/.cache/nfcore
~/.cache/nf-core
~/.config/nfcore
~/.config/nf-coreTong LI
04/24/2025, 3:35 PMnf-core --verbose subworkflows install fastq_align_mapad
DEBUG Popen(['git', 'version'], cwd=/lustre/scratch127/cellgen/cellgeni/tl10/temp/nf-modules_pipeline_test, stdin=None, cmd.py:1253
shell=False, universal_newlines=False)
DEBUG Popen(['git', 'clone', '-v', '--progress', '--', 'https://github.com/nf-core/modules.git', cmd.py:1253
'/home/ubuntu/.config/nfcore/nf-core/modules'],
cwd=/lustre/scratch127/cellgen/cellgeni/tl10/temp/nf-modules_pipeline_test, stdin=None, shell=False,
universal_newlines=True)
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, stdin=None, shell=False, cmd.py:1253
universal_newlines=False)
DEBUG Got '/home/ubuntu/.config/nfcore/nf-core/modules' as path utils.py:280
DEBUG Found a config cache, loading: /home/ubuntu/.nextflow/nf-core/wf-config-cache-9f4bce35ba6f9f759cbf54fa7.json utils.py:314
DEBUG Using config file: /home/ubuntu/.config/nfcore/nf-core/modules/.nf-core.yml utils.py:1385
DEBUG Got '.' as path utils.py:280
DEBUG No config cache found utils.py:321
DEBUG Running command: nextflow config -flat . utils.py:364
DEBUG Saving config cache: /home/ubuntu/.nextflow/nf-core/wf-config-cache-83d72a8175c14fcf070d5cd13.json utils.py:354
DEBUG Using config file: .nf-core.yml utils.py:1385
ERROR 'imaging_generatecompanion' commands_subworkflows.py:201Júlia Mir
04/24/2025, 3:36 PMTong LI
04/24/2025, 3:40 PMINFO Removed files for 'imaging/generatecompanion' and its dependencies 'imaging/generatecompanion'.
and still the same
Pulling from 'nf-core/modules' (https://github.com/nf-core/modules.git) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Waiting for responseDEBUG Popen(['git', 'version'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, stdin=None, shell=False, cmd.py:1253
universal_newlines=False)
DEBUG Popen(['git', 'fetch', '-v', '--progress', '--', 'origin'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, cmd.py:1253
stdin=None, shell=False, universal_newlines=False)
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, stdin=None, shell=False, cmd.py:1253
universal_newlines=False)
DEBUG Popen(['git', 'merge', 'origin/master'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, stdin=None, shell=False, cmd.py:1253
universal_newlines=False)
DEBUG Got '/home/ubuntu/.config/nfcore/nf-core/modules' as path utils.py:280
DEBUG Found a config cache, loading: /home/ubuntu/.nextflow/nf-core/wf-config-cache-9f4bce35ba6f9f759cbf54fa7.json utils.py:314
DEBUG Using config file: /home/ubuntu/.config/nfcore/nf-core/modules/.nf-core.yml utils.py:1385
DEBUG Got '.' as path utils.py:280
DEBUG Found a config cache, loading: /home/ubuntu/.nextflow/nf-core/wf-config-cache-83d72a8175c14fcf070d5cd13.json utils.py:314
DEBUG Using config file: .nf-core.yml utils.py:1385
ERROR 'imaging_generatecompanion'Tong LI
04/24/2025, 3:46 PMJúlia Mir
04/24/2025, 3:46 PMJúlia Mir
04/24/2025, 3:47 PMnf-core pieplines createJúlia Mir
04/24/2025, 3:47 PMTong LI
04/24/2025, 3:47 PMTong LI
04/24/2025, 3:52 PMnf-core --verbose subworkflows -g https://github.com/cellgeni/nf-modules.git install stitch
good and bad news: different error
it somehow thinks imaging_generatecompanion is a subworkflow. but it isn't.
,--./,-.
___ __ __ __ ___ /,-._.--~\
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'
nf-core/tools version 3.2.0 - https://nf-co.re
Pulling from 'cellgeni/nf-modules' (https://github.com/cellgeni/nf-modules.git) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Waiting for responseDEBUG Popen(['git', 'version'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False, cmd.py:1253
universal_newlines=False)
DEBUG Popen(['git', 'fetch', '-v', '--progress', '--', 'origin'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, cmd.py:1253
stdin=None, shell=False, universal_newlines=False)
DEBUG Popen(['git', 'checkout', 'main'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False, cmd.py:1253
universal_newlines=False)
DEBUG Popen(['git', 'merge', 'origin/main'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False, cmd.py:1253
universal_newlines=False)
DEBUG Got '/home/ubuntu/.config/nfcore/cellgeni/nf-modules' as path utils.py:280
DEBUG No config cache found utils.py:321
DEBUG Running command: nextflow config -flat /home/ubuntu/.config/nfcore/cellgeni/nf-modules utils.py:364
DEBUG Could not open /home/ubuntu/.config/nfcore/cellgeni/nf-modules/main.nf to look for parameter declarations - [Errno utils.py:346
2] No such file or directory: '/home/ubuntu/.config/nfcore/cellgeni/nf-modules/main.nf'
DEBUG Using config file: /home/ubuntu/.config/nfcore/cellgeni/nf-modules/.nf-core.yml utils.py:1385
DEBUG Could not find a config file in the directory '.' utils.py:1328
WARNING 'repository_type' not defined in .nf-core.yml components_utils.py:44
DEBUG Using selector: EpollSelector selector_events.py:54
? Is this repository a pipeline or a modules repository? Pipeline
INFO To avoid this prompt in the future, add the 'repository_type' key to your .nf-core.yml file. components_utils.py:55
? Would you like me to add this config now? [y/n] (y): y
INFO Config added to '.nf-core.yml' components_utils.py:59
WARNING Could not find a 'main.nf' or 'nextflow.config' file in '.' components_command.py:107
INFO The 'modules.json' file is not up to date. Recreating the 'modules.json' file. modules_json.py:628
DEBUG Got '.' as path utils.py:280
DEBUG No config cache found utils.py:321
DEBUG Running command: nextflow config -flat . utils.py:364
DEBUG Could not open main.nf to look for parameter declarations - [Errno 2] No such file or directory: 'main.nf' utils.py:346
? Can't find a ./modules directory. Would you like me to create one? [y/n] (y): y
INFO Creating ./modules directory in '.' modules_json.py:85
fatal: not a git repository (or any of the parent directories): .git
DEBUG Not in a git repository, skipping pre-commit hook. lint_utils.py:118
DEBUG Popen(['git', 'checkout', 'main'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False, cmd.py:1253
universal_newlines=False)
DEBUG Popen(['git', 'checkout', 'main'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False, cmd.py:1253
universal_newlines=False)
DEBUG Popen(['git', 'checkout', 'main'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False, cmd.py:1253
universal_newlines=False)
DEBUG Popen(['git', 'cat-file', '--batch-check'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=<valid cmd.py:1253
stream>, shell=False, universal_newlines=False)
DEBUG Popen(['git', 'rev-list', '--max-count=1', '432a2f50df2cc2f68c149df94606a2254be35fbb', '--', cmd.py:1253
'subworkflows/sanger/stitch'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False,
universal_newlines=False)
DEBUG Popen(['git', 'cat-file', '--batch'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=<valid stream>, cmd.py:1253
shell=False, universal_newlines=False)
INFO Installing 'stitch' install.py:131
DEBUG Installing subworkflows 'stitch' at modules hash 2ecc2cc028337abebe6b8938323ba845d9ac8a4d from install.py:132
https://github.com/cellgeni/nf-modules.git
DEBUG Popen(['git', 'checkout', '2ecc2cc028337abebe6b8938323ba845d9ac8a4d'], cmd.py:1253
cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False, universal_newlines=False)
DEBUG Popen(['git', 'checkout', 'main'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False, cmd.py:1253
universal_newlines=False)
WARNING Could not find a 'main.nf' or 'nextflow.config' file in '.' components_command.py:107
DEBUG Popen(['git', 'checkout', 'main'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False, cmd.py:1253
universal_newlines=False)
ERROR Subworkflow 'imaging_generatecompanion' not found in available subworkflows install.py:211
DEBUG entering code: StateBlock(line=0,level=0,tokens=0), 0, 1, False code.py:10
DEBUG entering fence: StateBlock(line=0,level=0,tokens=0), 0, 1, False fence.py:10
DEBUG entering blockquote: StateBlock(line=0,level=0,tokens=0), 0, 1, False blockquote.py:13
DEBUG entering hr: StateBlock(line=0,level=0,tokens=0), 0, 1, False hr.py:14
DEBUG entering list: StateBlock(line=0,level=0,tokens=0), 0, 1, False list.py:102
DEBUG entering reference: StateBlock(line=0,level=0,tokens=0), 0, 1, False reference.py:10
DEBUG entering html_block: StateBlock(line=0,level=0,tokens=0), 0, 1, False html_block.py:35
DEBUG entering heading: StateBlock(line=0,level=0,tokens=0), 0, 1, False heading.py:13
DEBUG entering lheading: StateBlock(line=0,level=0,tokens=0), 0, 1, False lheading.py:10
DEBUG entering paragraph: StateBlock(line=0,level=0,tokens=0), 0, 1, False paragraph.py:10
╭─ info ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ Use the command nf-core subworkflows list to view available subworkflows. │
│ │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ERROR __main__.py:132mashehu
04/24/2025, 3:53 PMTong LI
04/24/2025, 4:01 PMTong LI
04/24/2025, 4:04 PMERROR Subworkflow 'imaginggeneratecompanion' not found in available subworkflowsTong LI
04/24/2025, 4:19 PMimaginggeneratecompanion) in the subworkflows.
This is maybe not a caching issue but the tools issue?Tong LI
04/24/2025, 4:30 PMtools==2.14.0 as well.. slightly different error.
Pulling from 'cellgeni/nf-modules' (https://github.com/cellgeni/nf-modules.git) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Waiting for responseDEBUG Popen(['git', 'version'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'fetch', '-v', '--progress', '--', 'origin'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False, universal_newlines=True) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'main'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'merge', 'origin/main'], cwd=/home/ubuntu/.config/nfcore/cellgeni/nf-modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Using config file: /home/ubuntu/.config/nfcore/cellgeni/nf-modules/.nf-core.yml utils.py:1051
DEBUG Using config file: .nf-core.yml utils.py:1051
ERROR 'imaginggeneratecompanion' __main__.py:1711
I am now very confused💀🫠Tong LI
04/25/2025, 8:46 AMmodules repo?
I thought all we need is the .nf-core.yml + empty modules and subworkflows folders.
Has that changed?Tong LI
04/25/2025, 9:02 AMcaching issue now.
I've first cleaned up the known caching locations
rm -fr ~/.cache/nfcore
rm -fr ~/.config/nfcore
rm -fr ~/.cache/nf-core
rm -fr ~/.config/nf-core
rm -rf ~/.nextflow/nf-core
And I run nf-core --verbose modules list local, where it has nothing to do with our own repo (https://github.com/cellgeni/nf-modules/). And I always see the same error.
Pulling from 'nf-core/modules' (https://github.com/nf-core/modules.git) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Waiting for responseDEBUG Popen(['git', 'version'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'fetch', '-v', '--progress', '--', 'origin'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, stdin=None, cmd.py:1253
shell=False, universal_newlines=True)
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, stdin=None, shell=False, cmd.py:1253
universal_newlines=False)
DEBUG Popen(['git', 'merge', 'origin/master'], cwd=/home/ubuntu/.config/nfcore/nf-core/modules, stdin=None, shell=False, cmd.py:1253
universal_newlines=False)
DEBUG Got '/home/ubuntu/.config/nfcore/nf-core/modules' as path utils.py:280
DEBUG No config cache found utils.py:321
DEBUG Running command: nextflow config -flat /home/ubuntu/.config/nfcore/nf-core/modules utils.py:364
DEBUG Saving config cache: /home/ubuntu/.nextflow/nf-core/wf-config-cache-9f4bce35ba6f9f759cbf54fa7.json utils.py:354
DEBUG Using config file: /home/ubuntu/.config/nfcore/nf-core/modules/.nf-core.yml utils.py:1385
DEBUG Got '.' as path utils.py:280
DEBUG No config cache found utils.py:321
DEBUG Running command: nextflow config -flat . utils.py:364
DEBUG Saving config cache: /home/ubuntu/.nextflow/nf-core/wf-config-cache-af790a9274bf56bcfa3e1bd60.json utils.py:354
DEBUG Using config file: .nf-core.yml utils.py:1385
INFO Repository type: pipeline list.py:85
ERROR 'imaginggeneratecompanion' commands_modules.py:48Júlia Mir
04/25/2025, 9:04 AMJúlia Mir
04/25/2025, 9:05 AMJúlia Mir
04/25/2025, 9:05 AMTong LI
04/25/2025, 9:05 AMJúlia Mir
04/25/2025, 9:07 AMTong LI
04/25/2025, 9:08 AMJúlia Mir
04/25/2025, 9:10 AMJúlia Mir
04/25/2025, 9:10 AMTong LI
04/25/2025, 9:13 AM_ as you just pointed out.Tong LI
04/25/2025, 9:14 AM