Pipeline linting of <#CE9MS66BS|mag> seems to eith...
# tools
j
Pipeline linting of #mag seems to either run EXTREMLY long or just hangs when I'm running it locally with:
Copy code
DEBUG    Running lint test: pipeline_if_empty_null                                                                                                        __init__.py:272
DEBUG    Could not open file split in pipeline_if_empty_null lint test                                                                       pipeline_if_empty_null.py:38
DEBUG    Could not open file split in pipeline_if_empty_null lint test                                                                       pipeline_if_empty_null.py:38
DEBUG    Could not open file split in pipeline_if_empty_null lint test                                                                       pipeline_if_empty_null.py:38
DEBUG    Could not open file split in pipeline_if_empty_null lint test                                                                       pipeline_if_empty_null.py:38
DEBUG    Could not open file split in pipeline_if_empty_null lint test                                                                       pipeline_if_empty_null.py:38
Any ideas why that is, and what else I could do to provide more information?
j
do you have a file called
split
in any of the pipeline directories? 🤔
j
There is
split_fasta.nf
?
j
I have tried to run it but I can't reproduce
where are you running it from? the pipeline root directory?
could you try
tree mag | grep split
and se what is the output?
j
Yes pipeline root directory
It doesn't happen when I run on gitpod.. 😕
Copy code
(nf-core) james@bionb103:~/git/nf-core/mag (merge-binqc-pydamage)$ tree | grep split
│   ├── split_fasta.py
│   │   ├── split_fasta.nf
│       │   │   ├── split -> /home/james/git/nf-core/mag/testing/work/90/4e9ab38e3115c8e1da675457199975/database/split
│       │       ├── split -> /home/james/git/nf-core/mag/testing/work/90/4e9ab38e3115c8e1da675457199975/database/split
│       │   │   ├── split -> /home/james/git/nf-core/mag/testing/work/90/4e9ab38e3115c8e1da675457199975/database/split
│       │   │   ├── split -> /home/james/git/nf-core/mag/testing/work/90/4e9ab38e3115c8e1da675457199975/database/split
│       │   │   │   ├── split
│       │   │   ├── split -> /home/james/git/nf-core/mag/testing/work/90/4e9ab38e3115c8e1da675457199975/database/split
│       │   │   │   ├── split
│       │   │   │   ├── split
I'll try removing the testing directory as a minimum...
j
ah when I do this in a fresh repo I only see
Copy code
❯ tree mag | grep split
│   ├── split_fasta.py
│   │   ├── split_fasta.nf
It's probably due to the links
what are they?
j
dtabase files
Copy code
$ tree | grep split
│   ├── split_fasta.py
│   │   ├── split_fasta.nf
OK I now have that
I'll tyr again
j
do these database files have to be there? if so, we could add an option to skip files through .nf-core.yml
j
I normally isolate my manual testing of the pipeline to a directory called
testing/
inside the github repo.
I guess you could have linting respect
.gitignore
?
work/
is already in there
But that's not helping
Actually
It's still stuck... unless there is some caching going on?
j
what is the log saying now?
j
oh wait that time it went through!
Let me run it again
with a time stamp and also redirecting it
Copy code
real    30m35.514s
user    60m31.855s
sys     1m19.104s
lint.log
j
nice! then it was due to the database files
we have the funcitonality to use .gitignore in place but we are not using it in this specific lint test 😄
j
I don't htink so
😱 1
Do I really ahve to wait 30m for linting on a pipeline?
That doesn't seem right?
Or?
j
ok that's a different problem then, since now it's not stuck with the database files
j
Yes 😞
j
are there any other files that you are ignoring in .gitignore and we should take into account?
j
I don't know if that is the issue now though
Because once I removed my
work
it didn't make a difference
(it was th same as you)
j
I am trying to lint again my clone of mag, it is taking a bit to lint all the modules, but I think this is expected
j
It definitely wa steh
if_empty
it sat a long time on...
But it would be good to know how long it takes for your run to get thruogh
j
43.15s user 142.67s system 59% cpu 5:13.17 total
j
Yeah that's quite different from 30m 😅
So you don't have any suggestion how to debug?
Or just always lint on gitpod or somehting
j
definitely way less than 30min 😅
if you clone the repo in a different directory and try to lint there, does it take as long?
j
Shall test
OK like 2 minutes
Soooooo it's just my original clone
Easy fix then 😅
Thank you for your patience @Júlia Mir!
j
anyway I think it would be good to ignore files in .gitignore for ALL linting tests, so it was a good thread 😄
🎉 2
j
I'm getting this again 😞