https://nf-co.re logo
Join Slack
Powered by
<@U03CKGEU3LZ> working on <https://github.com/nf-c...
# rnavar
m
@Nicolas Vannieuwkerke working on https://github.com/nf-core/rnavar/pull/215, any idea why schema is now complaining?
quick look 1
m
it’s a new nf-schema version, so maybe something change there?
m
yeah, that's where I was heading for
all my pipeline tests are now failing
but only on 24.10.5, not on latest-everything
n
Taking a look!
m
ERROR ~ Duplicate key "errorMessage" at 989 [character 31 line 22]
631
m
wait what is the minimum nextflow version for nf-schema 2.4.2?
n
It's probably a merge issue from the template
m
that sounds like a merge issue
m
that will fail on only 1 version
n
It also fails on 
latest-everything
if you look at the logs but it somehow isn't shown as failed
m
because we don’t fail latets-everything anymore 😉
n
Oh right! I forgot 😁
👀 1
m
Oh yeah, because we don't report that as failing
what did you forgot?
n
That we ignore those now 😛
🤣 1
m
I think i got it
🎉 1
in 
assets/schema_input.json
, there was a duplication for the error message
🙌 1
m
@Nicolas Vannieuwkerke could that error message point to the affected file otherwise the line number can be a bit confusing
m
yeah, the error message was totally un-helpful
n
I'll need to see how that error is thrown. It's an error from the package that handles the JSON schema 😛
Can you make an issue?
m
duplication was on line 22 indeed, but no idea which file
On it
🎉 1
n
Thanks!!!
m
I still have an issue with linting
n
that looks like a tools bug homer disappear
m
🎉 1
ok, I'll dig deeper then
given the error, I have some ideas
coming from:
Copy code
process {

    // UMITOOLS_EXTRACT
    withName: UMITOOLS_EXTRACT {
        ext.args   = { [
                params.umitools_extract_method                  ? "--extract-method=${params.umitools_extract_method}" : '',
                params.umitools_bc_pattern                      ? "--bc-pattern='${params.umitools_bc_pattern}'" : '',
                params.umitools_bc_pattern2 && !meta.single_end ? "--bc-pattern2='${params.umitools_bc_pattern2}'" : '',
                params.umitools_umi_separator                   ? "--umi-separator='${params.umitools_umi_separator}'" : ''
        ].join(' ').trim() }
    }
}
it complains about 
umitools_bc_pattern
but not 
umitools_bc_pattern2
interesting
ok, found a solution by ignoring the linting
🤣 1
m
that’s not a good solution…
m
it works, I'd call it a good enough solution, but I'd agree on saying it's not optimal
m
usually if linting complains it is right that there is something off, why do you think this isn’t the case here?
m
I don't think it's not the case, I just wanted a quick fix before properly fixing it
I know, not the right course of action
but that's why I haven't merge the PR yet
I am working on a proper fix
ok, definitively something is weird
I removed https://github.com/nf-core/rnavar/blob/dev/conf/modules.config#L133-L140 from my local copy, and yet, I still have the same linting error
I can clearly see this error with linting:
Copy code
│ schema_params: Param umitools_extract_method                  ? "--extract-method from nextflow config not found in nextflow_schema.json                                                 │
│ schema_params: Param umitools_bc_pattern                      ? "--bc-pattern from nextflow config not found in nextflow_schema.json                                                     │
│ schema_params: Param umitools_umi_separator                   ? "--umi-separator from nextflow config not found in nextflow_schema.json                                                  │
And I only have 3 occurrences of this params in the code, the 
nextflow.config
, the 
nextflow_schema.json
and the docs.
confused dog
Is the pipeline code stored somewhere else when being checked by linting?
@Nicolas Vannieuwkerke any idea there?
n
Not really no... 😕
m
can you run it in verbose mode 
nf-core --verbose pipelines lint
?
n
It looks like a bug to me where 
umitools_extract_method                  ? "--extract-method
is considered to be the parameter instead of 
umitools_extract_method
m
yep
m
yeah, but when I remove it I still have that
confused dog 1
ooh
interesting
I have a feeling, it's only testing what has been commited
the error message was updated due to the changes I just commited
Copy code
│ schema_params: Param umitools_extract_method ? "--extract-method from nextflow config not found in nextflow_schema.json                                                                  │
│ schema_params: Param umitools_bc_pattern ? "--bc-pattern from nextflow config not found in nextflow_schema.json                                                                          │
│ schema_params: Param umitools_umi_separator ? "--umi-separator from nextflow config not found in nextflow_schema.json                                                                    │
notice that all the spaces have disappeared
ok, so I made some good progress
when I delete the local cache, I can see this with verbose:
Copy code
DEBUG    Config key: process.'withName:STAR_ALIGN'.ext.args, value: { [                                                                                                         utils.py:339
                     '--outSAMtype BAM SortedByCoordinate',                                                                                                                                 
                     '--readFilesCommand zcat',                                                                                                                                             
                     params.save_unaligned ? '--outReadsUnmapped Fastx' : '',                                                                                                               
                     params.read_length ? "--sjdbOverhang ${params.read_length - 1}" : '',                                                                                                  
                     params.star_twopass ? '--twopassMode Basic' : '',                                                                                                                      
                     params.star_max_memory_bamsort > 0 ? "--limitBAMsortRAM ${params.star_max_memory_bamsort}" : '',                                                                       
                     params.star_bins_bamsort > 0 ? "--outBAMsortingBinsN ${params.star_bins_bamsort}" : '',                                                                                
                     params.star_max_collapsed_junc > 0 ? "--limitOutSJcollapsed ${params.star_max_collapsed_junc}" : '',                                                                   
                     params.star_max_intron_size ? "--alignIntronMax ${params.star_max_intron_size}" : ''                                                                                   
                 ].flatten().unique(false).join(' ').trim() }
so just 1 entry for 
STAR_ALIGN
but when I look at the entry for 
UMITOOLS_EXTRACT
, I can see:
Copy code
DEBUG    Config key: process.'withName:UMITOOLS_EXTRACT'.ext.args, value: { [                                                                                                   utils.py:339
DEBUG    Config key: params.umitools_extract_method ? "--extract-method, value: ${params.umitools_extract_method}" : '',                                                        utils.py:339
DEBUG    Config key: params.umitools_bc_pattern ? "--bc-pattern, value: ${params.umitools_bc_pattern}'" : '',                                                                   utils.py:339
DEBUG    Config key: params.umitools_bc_pattern2 && !meta.single_end ? "--bc-pattern2, value: ${params.umitools_bc_pattern2}'" : '',                                            utils.py:339
DEBUG    Config key: params.umitools_umi_separator ? "--umi-separator, value: ${params.umitools_umi_separator}'" : ''                                                           utils.py:339
                 ].join(' ').trim() }
so 1 entry per line
ok, I have no ideas why 
UMITOOLS_EXTRACT
does not register as one entry while 
STAR_ALIGN
does
and not sure if it's that or not
I'm going deep #C073GFZ68PR
n
Keep us updated 😛
m
From 
nextflow config -flat
Copy code
process.'withName:UMITOOLS_EXTRACT'.ext.args = { [
            params.umitools_extract_method ? "--extract-method=${params.umitools_extract_method}" : '',
            params.umitools_bc_pattern ? "--bc-pattern='${params.umitools_bc_pattern}'" : '',
            params.umitools_bc_pattern2 && !meta.single_end ? "--bc-pattern2='${params.umitools_bc_pattern2}'" : '',
            params.umitools_umi_separator ? "--umi-separator='${params.umitools_umi_separator}'" : ''
        ].join(' ').trim() }
process.'withName:STAR_ALIGN'.ext.args = { [
            '--outSAMtype BAM SortedByCoordinate',
            '--readFilesCommand zcat',
            params.save_unaligned ? '--outReadsUnmapped Fastx' : '',
            params.read_length ? "--sjdbOverhang ${params.read_length - 1}" : '',
            params.star_twopass ? '--twopassMode Basic' : '',
            params.star_max_memory_bamsort > 0 ? "--limitBAMsortRAM ${params.star_max_memory_bamsort}" : '',
            params.star_bins_bamsort > 0 ? "--outBAMsortingBinsN ${params.star_bins_bamsort}" : '',
            params.star_max_collapsed_junc > 0 ? "--limitOutSJcollapsed ${params.star_max_collapsed_junc}" : '',
            params.star_max_intron_size ? "--alignIntronMax ${params.star_max_intron_size}" : ''
        ].flatten().unique(false).join(' ').trim() }
so I'm wondering if it's not due to the way 
tools
is processing that
@Júlia Mir, could this be a 
tools
issue rather than a pipeline one?
j
can you summarize the error? 🙏🏻
m
I'm seeing this error with linting:
Copy code
│ schema_params: Param umitools_extract_method ? "--extract-method from nextflow config not found in nextflow_schema.json                                                                  │
│ schema_params: Param umitools_bc_pattern ? "--bc-pattern from nextflow config not found in nextflow_schema.json                                                                          │
│ schema_params: Param umitools_umi_separator ? "--umi-separator from nextflow config not found in nextflow_schema.json                                                                    │
I have no idea what's causing it
I have several other params in the 
modules.config
that use similar settings
using 
nf-core --verbose pipelines lint
, I can see that each args in 
UMITOOLS_EXTRACT
returns as a separate config key, while for any other args over multiple lines, it's just 1 entry for the array
I cannot see any weird things in my config files that would explain that
j
could be because we use a regex to find all params from the output of 
nextflow config
m
ok, so a tools issue then
j
looks like it, yes
we have to take this into account in our regex
m
yeah, but what is causing it?
j
I think the difference with the other is that it contains a 
=
👀 1
m
Removing 
&& !meta.single_end
trigger an error for 
umitools_bc_pattern2
And removing the 
=
removed all errors, you're right
ok, so I can re-instate my ignore linting until it's fixed in tools
Copy code
lint:
  schema_params: false
j
could you open an issue pleas? 🙏🏻
m
of course
thanks for helping me solve the issue
I did not notice the 
=
sign
@Nicolas Vannieuwkerke can I get a ✅ then for https://github.com/nf-core/rnavar/pull/215?
✅ 1
n
Looks good!!!
m
thanks
Open in Slack
PreviousNext
Powered by