hello, I have a pipeline of pipelines, each has its own nf-schema. I am wondering how is this envisioned to work when validating parameters. Right now i have a main that is importing the workflows, so my guess is that i run

validateParameters

at that level and not within the named workflows, and then i should probably merge all the nf-schema together inside the workflow of workflows? thank you for the help!

There is no specific support or guidance for this.

interesting, yeah, so we did the poor mans version of that, each workflow has a

<http://main.nf|main.nf>

and a

workflows/myWorkflow.nf

with a named workflow. 1)all workflows take a channel of inputs, and a map of parameters 2) all workflows have an

nf-schema

that validates the parameters, and then we create a dictionary that gets passed to the workflows. 3)all process are tagged with the name of the workflow 4)all workflows have

nextflow.config

This allow us to be able to run the pipelines stand alone, or compose them together. Making them installable sounds like the superior choice, right now we use git subtrees to synch everything up through github actions. Like this we can always just import the configs and chain them together quite easily, but I couldnt find a way to cleanly chain the all the schemas.

Because you can’t chain schemas. There are a few issues, primarily: • how should conflicting parameter names be handled? I can’t fully conceptualise your setup, it isn’t clear to me if these are actually separate worklfows or in reality a single workflow with many entrypoints.

it is a bit weird, there are cases where each individual pipeline is run alone, or a case where a combination of multiple ones are run sequentially. The conflicting names is a really good point, we have 2 ways to deal with this, either with nested maps, and we just pass the map associated to the pipeline, or we just need to be very careful. The main advantage is that as long as there are no overlapping keys, we can jsut pass the entire map, and users can reusue their

params.config

across workflows. We just started doing this, dictionaries was the only way also that i found to be able to pass keyworded arguements to pipelines, I wish this was available in processes and workflows.

As convenient as named arguments would be, I don’t see why they are necessary if you are wiring up the pipelines anyway, just pass the params positionally like one does in an nf-core pipeline?

yeah, that is an option, but our pipelines have a bit too many parameters , so it is a bit tedious, and have had cases where we mixed parameters. but I agree, we could do it positionally.

Also keep in mind that in the future parameters will only be supported in the top level workflow. So passing parameters as inputs is a good idea to make your pipelines future proof

we. are passing the parameters as a map, so that should be future proof, no?

yes that should keep working normally 😛