Hi <!channel>, thank you to everyone who joined the spontaneous discussion just now at the nf-core hackathon on this topic! I thought I'd try to put together a few basic notes for future me (maybe also for use in a wrap-up blog post / end of day slides!) • Pipeline chaining ◦ Pipeline chaining nearly always needs "glue" to massage outputs from one pipeline into the input of the next. ◦ Different approaches to this: Node-RED (eg. node-red-seqera) with custom JS nodes. @Julian Flesch interested in building tooling (Python? Groovy?) ◦ Data Lineage can help, by providing a way to get file paths from JSON namespaced object, rather than "fudging paths" (needs workflow outputs) ◦ Needs: More difficult use cases, discussion • Meta pipelines ◦ Most people don't want to have to maintain 2x (or more) sets of documentation and pipeline code ◦ Meta pipelines can have better concurrency, resumability - but are more work ◦ Stumbling blocks: global params (solved with workflow inputs?), config merging (solved with

withName

namespacing?) ◦ Needs: testing with above code changes, exploration • Short-term action items ◦ When converting to Workflow Outputs, try to consider related potentially-downstream pipelines, and generate compatible sample sheets ◦ [idea] Namespace all pipeline config

withName

configs under the top-level subworkflow name, so when importing multiple into a meta pipeline we don't get config scope clashes ◦ Join #C07ACNLELR2 ◦ Join the Nextflow 26.04 beta and play with record types Please add to / correct these notes in the S 🧵 🙂 nextflow scroll

FOMO

So every

withName:

needs to have the pipeline name in it?

Fear Of Missing Out

Is Node Red only on Seqera platform?

NodeRed is an opensource project from the realm of home automation. Its a sort of graphical programming language, where you can build event flows / chains and trigger scripts 👍 It's just a vessel and has nothing to do with Platform (afaik)

It does use tw commands/api calls to the platform at the moment to trigger the right pipelines . But yes it’s open and we can do whatever we ant

even raise a flag

🏴‍☠️

Ah, so the Seqera thing is based on https://nodered.org/

There isn't a single strategy I think - many people working on different approaches, some of which are complementary to one another.

Is there already an example for the "Short-term action items"-approach?

Fetchngs producing samplesheets for RNAseq, etc might be an example

For me using Node-Red, python, or anything to glue nextflow scripts together is better than nothing, but is a medium term solution. I understand how complex it is to do that at the nextflow level, but that is what I wish for at christmas

Here's what I had working at some point, but I made a mistake of not recording the version of which pipeline I was using and now things break: https://github.com/mahesh-panchal/nf-cascade/blob/main/README.md

And then we'd need on the nf-core side a way to pull the documentation

Maybe documentation needs to be a Nextflow module / subworkflow component.

🤔

Can a plugin be used to alter the behaviour of

nextflow pull

?

Yes, that’d be a nice step forward. The milestone for https://github.com/nextflow-io/nextflow/issues/2870 is now 26.04