Hi! I'd like to have your point of view concerning...
# pipeline-maintainers
o
Hi! I'd like to have your point of view concerning the integration of Nextflow pipelines (nf-core or not) into other Nextflow pipelines. I'm currently developing the new #C0C6WPPDT0R pipeline, and ideally I'd like to integrate: • github.com/ebi-pf-team/interproscan6 • github.com/nf-core/ncrnannotator Is it possible to integrate such pipelines through git submodules (or an equivalent)? Using something like nf-cascade would be last resort solution, if something else works. Thanks a lot!
m
I think we're going to talk about such questions during the hacakthon tomorrow
fbwow 1
cc @Jonathan Manning
j
The new pipeline composition syntax (discussed in the hackathon tomorrow) will be the ideal way to do it, but it will be a while before lots of pipelines catch up with all the necessary syntax to make it work
ā¤ļø 1
So you might have to use other methods in the meantime
šŸ‘ 1
o
Super glad to see that things are moving forward on this aspect šŸ™‚ thank you very much guys for your work
m
I'm very interested in this, but can't attend the hackathon. Where can I follow the development or discussion about this?
šŸ‘ 1
e
I use seqera.io/blog/nf-cascade, which works nicely before the composition syntax arrives. It's some extra code to make it work as the pipelines are right now, but once added it works excellently.
šŸ‘ 1
m
The caveat is that nf-cascade hasn't been tested on Cloud so I have no idea if it even works there. Local and HPC works fine, and also works with Seqera Platform. But yeah, we're looking at Ben's new integration for pipeline composition at the Hackathon now.
šŸ‘ 2
j
Check out the dedicated channel for this: #C07ACNLELR2 We are currently exploring how the new Nextflow strict language features to achieve things like this
party this 2
Another project to describe and run pipeline chains in the meantime is bmds-tue.github.io/nf-meta/latest Disclaimer: This is a public beta. I am the author.
keanu thanks 1
party this 1
o
It looks really nice, thanks!
p
Caveat: no, I don't think we would be ok importing non-nf-core pipelines into an nf-core pipeline.
šŸ‘ 1
Also pipeline composition in order to make top-level nf-core pipelines is a major concept that needs a lot of discussion, and many updates to nf-core pipelines (it requires new syntax and record types). It's not going to be something we do any time in the near future.
šŸ‘ 1
m
So the component pipelines would themselves need to use the latest syntax, so for now nf-cascade is the simplest solution if you want to include it.
šŸ‘ 1