Hello, I am adding a new module to <#C038XMTQWBT|g...
# tools
r
Hello, I am adding a new module to #genomeassembler which is a new assembly tool, I created the tool following the nf-core guidelines. When I try to lint I get an error
CRITICAL list index out of range. I would like to know what am I doing wrong
Slack conversation
j
moreinfo
s
To help we'd love a few more details please: 1. The Nextflow command you used to launch the pipeline 2. Any custom Nextflow config files 3. The full error message 4. The corresponding
.nextflow.log
file, where you are running the analysis. Remember to use triple backslashes on their own lines for code blocks. Also take a look at https://nf-co.re/usage/troubleshooting for more information on debugging failed pipelines. Thanks!
j
Do you have aP R someone can look at?
r
I haven't added a PR yet, I am trying to add a new assembly tooloption for #genomeassembler which is next denovo, the first step is to add the module to modules but before PR I wanted first to lint the module and got that error. I forked the project and I can add a PR if that's alright.
j
yes please - otherwise we have nothing to look at 😅
r
Yes you are right, I just thought it was a common error i will do that! thank you!
👍 1
j
Even when it's a common error it's good to do it anyway 🙂
r
It must be made to the dev branch right?
j
Yes
If on pipeline
r
modules/nf-core to later add it to the step ASSEMBLE with flye and hifisam
j
If it's to nf-core/modules you can open the PR directly to master
n
For clarification @Rayane ELMEDJADJI , the modules should go into nf-core/modules first, which is where all the tool-specific modules are stored for easy integration into various pipelines. Once it is in there, you can install them using
nf-core modules install <name>
into the pipeline you are working on, and then use them in there. The extra effort of going via nf-core/modules makes your modules more broadly available for other nextflow pipeline developers, and also means that any updates to the module files can easily be propagated across pipelines, since they can all pull from the same modules repo
this 1
r
I am sorry I did not know that, I will add the PR in that repo instead.
❤️ 1
n
sure, no problem, it can be kinda overwhelming in the beginning
r
At the end I could lint with no errors once i forked the nf-core/modules repo. I just did the PR so that you can check the commits. Note that this PR is more like a draft since the tests did not pass yet and there are still some modifications to be made.
I have an issue with a dependency :
Copy code
from paralleltask import Task                                                                                                                                                               │
│     ModuleNotFoundError: No module named 'paralleltask'
Here is the environment.yml file:
Copy code
channels:
  - conda-forge
  - bioconda
dependencies:
  - bioconda::nextdenovo=2.5.2
  - conda-forge::paralleltask=0.1.1
The module is available here: https://anaconda.org/conda-forge/paralleltask
n
Does your environment.yml have an empty line at the end?
r
Yes
I don't really understand this part for the containers:
Copy code
container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
        'https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/fa/fa1c1e961de38d24cf36c424a8f4a9920ddd07b63fdb4cfa51c9e3a593c3c979/data' :
        'community.wave.seqera.io/library/flye:2.9.5--d577924c8416ccd8' }"
I copied it from flye module
n
Is specifies singularity or docker containers that should be used to run the module
And the container should contain the tool you use
If you use
nf-core module create
it should ask you for the tool name and will try to find a suitable container
r
Amazing, I was looking in the web but was kind of hard to find. Thanks!
Hello I added a test that successfully ran, since for nextdenovo, the config is an input file itslef, I did a config that works for the test data i used. How can this test config file be accessible so we can retrieve it remotely?
Also when running on conda profile, the dependency doesn't seem to be resolved, when running on docker profile it does and when running singularity i get an error.
should i just resolve it with pip install (not sure if thats appropriate)
n
I think resolving via pip install is not great, just to clarify, you are trying this
conda install -c conda-forge -c bioconda nextdenovo==2.5.0--py39h20169af_0
and it does not work correctly?
I think that if you put the config file you need into the
tests
folder of your module it should be available for tests as
./<Filename>
r
nextdenovo is resolved normally, but its dependency parallel task couldn't be resolved with conda.
n
but that should be in conda-forge, no?
r
- conda-forge::paralleltask=0.1.1
I have this line in the envyml
environment.yml*
n
hm, I see, the container from biocontainers is also broken
r
Ah that's why it works when the profile is docker and not conda?
n
It does work for you with docker?
The container is pre-built, i dont really know the insides of biocontainers, but maybe @James Fellows Yates does
r
Yes with docker it works without the pip install command
n
Ok
and which container does that use?
r
the one suggested by nf-core
biocontainers/nextdenovo:2.5.2--py310h0ceaa1d_6
Also I did not specify anything for paralleltask for docker, I guess it is resolved automatically
When install it with
conda install conda-forge::paralleltask
in my terminal, it works
n
but when you have
conda-forge::paralleltask
in your env,yml it does not?
I have to admit that I am probably pretty useless at debugging conda problems, since I go through some effort to avoid touching conda as I think it is.. not a great experience
r
I don't understand either since, if I put it in the env, it doesn't work, if I put the install command inside the script in .nf it installs but then nextdenovo deosn't find it, but if I do pip install it works fine. Does it have to work without the pip install?
n
There seem to be a few modules that have pip in the env.yml so I geuss it is ok; e.g.: https://github.com/nf-core/modules/blob/358dc70db82e53dc379967d457c0705b6a81968b/modules/nf-core/checkqc/environment.yml#L9
r
when I define it that way, it passes the test but not the linting 😅
Copy code
// env yml
channels:
  - conda-forge
  - bioconda
dependencies:
  - bioconda::nextdenovo=2.5.2
  - pip
  - python=3.8
  - pip:
      - paralleltask==0.1.1
// error message
 The environment.yml of the module nextdenovo is not valid: 'pip' does not match '^.[^><]=[^><].$'.
n
Yes, that is a bug in the linter I think
r
yes It doesn't match the regex
is there access to the linter 's source code, maybe I can help debug the issue locally?
TypeError: '<' not supported between instances of 'dict' and 'str'
this might come from the fact that
sorted(env_yml["dependencies"])
is failing to sort the env_yml[dependencies], this step is done i think when the yml is valid.
Do you suggest that I keep this way while failing to lint, or do you prefer that I just put pip install in the main.nf script?
Copy code
channels:
  - conda-forge
  - bioconda
dependencies:
  - bioconda::nextdenovo=2.5.2
  - pip=23.3.1
  - pip:
      - paralleltask==0.1.1
  - python=3.8
by adding the version with =, the linter finds it valid and tries to sort it, but the
yaml.load(file.yml)
funciton will create a dict inside the
env_yml["dependencies"]
turning the
Copy code
- pip:
      - paralleltask==0.1.1
into
Copy code
{'pip': ['paralleltask==0.1.1']}
And when the
sorted()
is called on
env_yml["dependencies"]
it will compare strings with a dictionnary which is not defined. This is only a guess, i haven't really tested it but It might help you debug if you are looking to solve the bug.
Thank you very much for your help with the module creation.
Hello I made some modifications to the module and i pushed them. What would be the next step after the module being created to add in the #genomeassembler as another assembler.
n
Once the module is merged into nf-core/modules, you can use
nf-core modules install
in the pipeline directory to install the module into
modules/nf-core
. I guess for the details on how to add it then into the particular pipeline it would make sense to discuss in #genomeassembler
r
Thanks!
You think that I should start working on that new helper module? or is it best to wait for now?
n
Maybe wait until we heard something from james or maxime
I am not really sure what the usual process is
well, in any case, it probably wont hurt to create it
I just am not sure if it will be pipeline or module
r
Okay! Thank you!
@Niklas Schandry Hello, of what I understand the nextdenovo module would still take the config file as input and not create bases on user's set parameters, is there anything else that should be modified besides this?
n
Ok, I guess if you cover all aspects of https://nf-co.re/docs/guidelines/components/modules it should be ok
r
Alright, if there is anything that has to be modified just tell me and I will. The module passes the test but not the linting due to a nf-core's linter small bug with dependencies.
n
That is probably something that should be redirected to someone from the <!subteam^S044DU0NG68>, I dont know what the procedure is with the linting issue
m
Can I get a TLDR @Niklas Schandry?
Am I right into assuming it's a linting issue with pip in the conda environment dependencies?
n
Yes
m
ok, then if it's just that we can force merge
where is the PR for this module?
(Just noticed you did link this issue (https://github.com/nf-core/tools/issues/3461) in this thread already)
n
I think
r
n
If linting fails none of the CI tests start, correct?
Oh, it looks like they do
r
Don't we need to specify pip=<version> ?
m
no, if linting fails, only this test is failing, the rest of CI should work
👍 1
s
Could really do with this fixing in tools
party this 1
m
on it deal with infra
❤️ 1
m
🚀
r
I think the test with docker doesn't pass and I am pretty sure it does when I do it locally.
n
It seems to time out
r
ok, is there a reason for that? The test passes with the docker profile locallu