https://nf-co.re logo
Join Slack
Powered by
I have an idea I want to try out for a plug-in, I ...
# nextflow-plugins
s
I have an idea I want to try out for a plug-in, I want to try extending the path function so I can (optionally) manage a cache for large reference files (e.g. sequence alignment index) outside the work directory. I want to modify path() to check the cache and only stage the path when it's not already present in the local cache, and do it in a thread safe way. Are there any example plugins that hook-in in similar ways or modify path() behavior that I can use for reference? The use case I have is creating a simple pattern to avoid repeatedly staging these files from aws s3.
I was browsing the nf-amazon source and came across s5cmd -- is this something that can be enabled? I was not aware of it, but reading the readme it looks like it should be way faster than aws-cli. Is it as simple as setting 
aws.s5cmdPath = /.../...
? https://github.com/nextflow-io/nextflow/blob/572f2116226bca4686c3540be9bd8b81bdded[…]ins/nf-amazon/src/main/nextflow/cloud/aws/util/S3BashLib.groovy
j
thinking loud, maybe you can create a PathFactory and manage a new schema "cache://" (for example)
s
Thank you, @Jorge!
I was hoping that I could somehow continue using the s3:// schema, with an optional parameter to path, like 
path('<s3://some/file>', cache: true)
And when rendering the .command.run script I would extend the usual staging functionality to do the caching -- that way the caching behavior could be enabled/disabled without requiring the inputs to change too much
j
not sure but following your approach I think maybe instead to create a new plugin you can improve current AwsPlugin
👍 1
(I mean, fork nextflow repo and create a PR over it)
s
Do you know where the staging script gets rendered?
@Jorge, following your suggestion I tried tweaking the core nf-amazon plugin and rebuilding nextflow using 
./compile.sh; make assemble
. I can see my changes in the files output under the build/ directory, but when I run nextflow the changes I made don't seem to be applied. I tried bumping 
modules/nextflow/src/main/resources/META-INF/plugins-info.txt
and 
plugins/nf-amazon/src/resources/META-INF/MANIFEST.MF
since it looks like that's involved in making a new release, but it didn't seem to help. I'm thinking that it is somehow getting linked to the previous version of nf-amazon. Is there anything else I need to do when changing one of the core plugins?
Ah, figured it out, 
make pack
seems to have done it
👍 1
Following your suggestion @Jorge I made a proof-of-concept implementation of the caching idea for the core nf-amazon plugin by forking and modifying the nextflow repo here . The minimal configuration is to specify that you want a host volume to be mounted into your container, and then to configure files to be cached in that location using nextflow.config:
Copy code
aws{
            batch {
                ...                
                volumes = ['/seqcmd/cache:/seqcmd/cache:rw']
                cache   = '/seqcmd/cache'
            }
        }
then if you have a process where you re-use a large input file across many task executions you can set 
cache:true
to tell nextflow to look for the file in the cache before resolving it from s3:
Copy code
process minimap2 {

    input:
        tuple(val(meta), path('in.fq.gz'))
        path('ref.mmi', cache: true)

    ...

    script:

    """
    minimap2 -a ref.mmi in.fq > out.sam
    """

}
Would love any pointers if you have any!
🤩 1
j
what about to "extend" the idea to other filesystems as Azure or Google ?
s
I haven't looked into it yet, just because I don't have accounts setup where I could test it. I think all the cloud executors rely on docker/bash, so I expect the code to do it would look basically the same for the other clouds. Are you offering to test? 🙂
😅 1
Open in Slack
PreviousNext
Powered by