https://nf-co.re logo
Join Slack
Powered by
hello! I have a question about why we are specifyi...
# nf-schema
r
hello! I have a question about why we are specifying the schema twice. I see it once here in the nextflow_schema.json: https://github.com/nf-core/sarek/blob/5fe5cdff171e3baed603b3990cab7f7fd3fcb992/nextflow_schema.json#L36 , but then we also set it when calling 
fromList
: https://github.com/nf-core/sarek/blob/5fe5cdff171e3baed603b3990cab7f7fd3fcb992/subworkflows/local/utils_nfcore_sarek_pipeline/main.nf#L125. What would happen if the two mismatch?
n
The one in the schema is used to validate the parameter and in the new version to show the help message of the file. The one in 
fromList
is the most important one because that one is used to validate the file and convert it to a channel. So basically the one in the schema is not strictly necessary
r
ok let's say I have set a samplesheet_schema.json in the nextflow.json that says everything is optional, but one in the 
fromList
function that says 
fastq
is required, then worst case a samplesheet without fastqs would pass the input validation but fail on the samplesheet reading?
n
yes correct!
r
would it make sense to converge the two to avoid any weird c&p errors?
n
That's not possible right now as the 
samplesheetToList
function is completely decoupled from the parameters and there is no way to know which parameter is linked to the samplesheet
It used to be like this in 
nf-validation
but that made the function very confusing to use so we decided to step away from it (also Ben advised it that way)
a
Maybe if we want to be fancy we can make a Samplesheet object?
Copy code
my_samplesheet = Samplesheet(params, schema)

my_samplesheet.validate() // raise error!
my_samplesheet.schema // ./assets/schema.json
my_samplesheet.toList(validate: true)

Channel.fromList(my_samplesheet.toList())
We could go further and make a big params object to use:
Copy code
parameters = Parameters(params, schema)

parameters.summaryLog() // print sumamry
<http://parameters.help|parameters.help> // print help string
parameters.toList()

// Validation object
validation = parameters.validation // Validation object
validation.status // bool for validation
validation.errors // array of errors
p
'cc @Ben Sherman as this could relate to future Nextflow syntax (I have a feeling I previously suggested the same 
.validate()
thing)
n
I'm not sure objects can be exposed in a plugin as of now? This looks very clean though I like this!
But this would probably require a 
3.0
update of nf-schema 😛, so let's not rush that for now
b
I think once we have record types, we can generate the samplesheet schema and its inclusion in the parameter schema, while in the pipeline we could use the record type to validate the samplesheet instead of the schema. So the schema would be a reflection of the 
params
block and 
record
defintions in the code but wouldn't be used directly by the code
🎉 1
n
Really looking forward to this!
a
@Nicolas Vannieuwkerke is there any way of accessing the samplesheet schema to use in the 
fromList
without accessing it twice?
Or just having it saved somewhere, I'm worried about divergence:
Copy code
"input_restart": {
    "type": "string",
    "description": "Automatic retrieval for restart",
    "fa_icon": "fas fa-file-csv",
    "format": "file-path",
    "exists": true,
    "mimetype": "text/csv",
    "pattern": "^\\S+\\.csv$",
    "hidden": true,
    "schema": "assets/schema_input.json"
},
Copy code
samplesheetToList(params.input, "$projectDir/assets/schema_inputs.json")
(note the small change)
n
I'm not entirely following what you mean?
a
I'm a little bit worried by divergence between the samplesheet schema in the nextflow_schema.json and the one we use in the 
samplesheetToList
method. Is there a way of enforcing they're the same? Or perhaps accessing the 
input_restart.schema
field in the above example?
n
I don't think you should be that worried about this? It's only two places and the schema location/names usually don't change. I've never heard of anyone having any issues with this 🙂. However, you could write some check in your pipeline for that if you want to be really sure, but there isn't much that can be done about this on the nf-schema side as of now
p
[ignore me, re-read and understand now]
Agree that 
samplesheetToList
should be able to take the path from the params schema, no?
n
That's what I thought when your message started dissapearing 🤣
p
Why does it need to be specified again in code?
n
Because the input of 
samplesheetToList
is not a parameter name, but a value. There is no way to know from what parameter this came
We used to have the parameter name as input, but a lot of people found that to be very confusing
p
Gotcha, I had a feeling I was getting deja-vu
n
I might add an option to specify a parameter name to 
samplesheetToList
but that would just create the same problem. how do you enforce that the value and the parameter name are coming from the same parameter
r
coming around to the object idea. It would be ideal if the schema and the samplesheet were somehow married to each other.
nod 1
n
That would be ideal indeed! And I will definitely do a brainstorm on this issue for nf-schema 3.0 😛. But it's not really possible in the current setup
a
I'd be happy to help with that
p
Back when we first introduced it, I did wonder about having the samplesheet schema as a nested schema object
it may also be possible to use JSON schema definitions for this, I think that you can refer to external files to be included when the schema is parsed
that way you end up with one big schema that's being parsed, even keeping separate files on disk
n
I'd be happy to help with that
Awesome! I'd love that
p
i can't really remember why I didn't do it that way 🤔
n
Yeah that's possible indeed using the 
$schema
key in JSON schema. and it would bring us closer to the standard syntax
👍🏻 1
8 Views
Open in Slack
PreviousNext
Powered by