Hi everyone, I have been trying to create a plugin for nextflow and I would appreciate a little help. 🙂 Also, please let me know if this is not the right place for a question like this. Context: we would like to create a nextflow executor that uses the firecrest API as a backend to submit Slurm jobs to a cluster. The api provides access (through http requests) to the cluster's scheduler (submit/query/cancel jobs), filesystem utilities (ls/cp/mv/cp/stat etc) and download/upload files from/to the cluster. The use case that we have in mind is having: • nextflow running somewhere outside the cluster, not on the login node, as it would normally, and • all the input files are already on the cluster's filesystem. Ideally with not too many changes in the nextflow script. The only files that need to be uploaded to the cluster are the scripts that nextflow is creating. Maybe at a later stage we would like to download some files from the output, but it is not important for now. So with this use case in mind I would like some help to understand what needs to be implemented in this new plugin. I have gone through the

nf-hello

example and the relevant podcast. I understand that we need to have new

Executor

and

TaskHandler

classes for the scheduler calls. Possibly also a new class extending

BashWrapperBuilder

. I was hoping to use the one that is already used by the

SlurmExecutor

but I think

AbstractGridExecutor

is assuming a shared filesystem (between nextflow and the compute nodes). At the moment (with a very hacky first effort), I can submit the jobs to the cluster, but nextflow will complain after the job finishes with

Missing output file(s) ...

. What is the right way to ask nextflow to check on the files through the api instead of locally? And in general, I not sure what is the best way to handle the "remote" files (they will not be remote during the job). Maybe I am approaching the problem in a wrong way, so I would be interested in any feedback or - even better - if you have a plugin in mind that is similar enough, that could be helpful. Sorry for the long message and thanks in advance!

I think your idea is very similar to the nf-nomad plugin https://github.com/nextflow-io/nf-nomad

thanks for the suggestion, I will have a look!

Nomad is a kind of kubernetes simplified and we have created a task executor to submit tasks using their http api Basically is a thing wrapper connecting containers running in the nomad with the nextflow session

Since you need to access files through an HTTP API instead of a filesystem, you'll need to implement a Java NIO filesystem similar to the one we have for S3 in Nextflow (see

S3FileSystemProvider

). This would allow Nextflow to access files with a URL scheme like

<firecrest://foo/bar/file.txt>

, and your NIO filesystem would translate those URLs to API calls

Thanks! I was a bit confused because the files need to be seen as remote by nextflow but local from the compute job. In the case of copying some files for example, the copy would be implemented with firecrest if it's happening from nextflow directly, but with a simple

cp

command if it's happening on the job. In any case, the S3/Azure filesystem providers have been a good starting point 🙂

that's a good point, do you have a CLI that you can use to transfer files via firecrest API? for the cloud providers we typically mount their CLI into the container and generate the copy commands in the task script

hm there is a cli, but it's not completely stable at the moment. We could easily improve on it, if it turns out to be helpful in this case. I have to give a disclaimer, that I am a novice user of nextflow so maybe I am missing something. (writing the plugin while learning nextflow maybe wasn't the best idea 😅 ) Would the cli help compared to using a java/groovy client, since on the compute node (no matter if it's on a container or not) firecrest is not needed? The only files that need to be uploaded (for what we have in mind for start) are the nextflow scripts and this is done now during the

submit

function of the custom task handler. And the copy on any job script should be a simple

cp

. The user is expected to have the input files already on the cluster before running nextflow.

I see, so if the compute job has direct access to the work directory via filesystem, that makes things much simpler. You can just use

cp

as you say