https://nf-co.re logo
Join Slack
Powered by
So I answered a question about <running nextflow i...
# wg-meta-pipelines
m
So I answered a question about running nextflow in nextflow on nextflow slack, but I realise it has applications here too. I think people might prefer pipelines as subworkflows, but one could also do it this way, by running each workflow as a process.
Copy code
import java.nio.file.Paths
import java.nio.file.Files

workflow {
    RUN_NEXTFLOW().log
    | view
}

process RUN_NEXTFLOW {

    exec:
    def cache = "nxf-cache"
    Files.createDirectories(Paths.get(cache))
    def builder = new ProcessBuilder(
        'nextflow', 'run',
            'nf-core/demo',
            '-profile', 'test,singularity',
            '-ansi-log', 'false',
            '-resume',
            '--outdir', "$task.workDir/results"
    )
    builder.directory(new File(cache))
    def process = builder.start()
    process.waitFor()
    nflog = process.text

    output:
    path "results", emit: output
    val nflog     , emit: log
}
👀 1
Could even make a module out of this, with some inputs.
r
This only works if the nodes you submit can submit again, right? Does it respect things like max number of jobs you can submit as a user on a cluster?
m
It's an 
exec:
process. It runs locally, i.e. this runs where the 
nextflow
parent process is running
facepalm 1
r
nice! (sorry I can't read) 😄
m
I think I've almost got daisy chaining working: https://github.com/mahesh-panchal/nf-cascade I just need to get fetchngs to run through and produce the samplesheet ( currently errors because 
wget: unable to resolve host address '<http://ftp.sra.ebi.ac.uk|ftp.sra.ebi.ac.uk>'
), and I can pass it to the next process.
🤯 2
n
I've also done some testing with this and it works very well but it just feels hacky: https://github.com/nvnieuwk/pipeception
m
@Ben Sherman what do you think of this?
(sorry for calling Ben, but I really want to know if it's a good idea before we try more down that road)
m
How're you dealing with resuming a pipeline with 
script:
?
n
I moved to work directory to a location that's not recreated every time and used the 
scratch
config option to tell the process to use that directory instead
But yeah that's super hacky and I don't like it 😁
m
But isn't the database and everything then still in the work directory, and won't see the old stuff on a new run?
i.e. the 
.nextflow/
folder, etc
n
(It's also something that I put together in less than an hour so it's not perfect)
It's able to see old stuff (somehow) I'm not 100% sure anymore what I did since it's been a while
m
fair enough. It's nice to see some other ideas fly about
🎉 1
j
You've seen Ben's POC on this right? https://github.com/bentsherman/fetchngs2rnaseq (though it's not the Nextflow-as-process thing)
m
Yeah. I thought that was basically the end goal of this channel, to produce that with 
nf-core
tools.
j
No worries, just checking 🙂
r
yes exactly. Sorry I have been slacking on this. putting it on my todo list for today to set up a discussion round about this. Although I think all the different ideas is basically what we wanted so this worked out well by me not doing anything 😄
🤣 1
m
My POC is running through to rnaseq now, btw, although crashes on salmon because of the fetchngs 
test
profile data.
j
I used to run Nextflow-in-Nexflow all the time at the EBI (just as processes, our compute nodes could also submit). It worked just fine, but it was a pain to do things like control overall job numbers accounting for top-level and child-level jobs. We did it with LSF job group settings that capped things, but obviously that's not a widely applicable solution.
b
The exec process is a nice quick-n-dirty solution for chaining pipelines. Seems like the main downside is you lose the granularity of channels and you introduce some extra publishing overhead
m
So what are people asking for when it comes to chaining pipelines? Are people asking for finer control over outputs?
Does anyone have any other thoughts on my implementation? Does someone have the capacity to test it in a cloud environment, or does it break?
So another weakness of this method using a process to run Nextflow is that if one wants to use profiles, and in particular combine profiles, e.g. 
-profile demo[,singularity/docker]
and then propagate that via a params to the process, it's not straightforward for the average user. It requires exposing something not usually available, so you don't override params set by another profile. (The solution is on the 
with_profiles
branch for the curious)
Open in Slack
PreviousNext
Powered by