By <@U04BMQHBG4U> &gt; Hi Guys, &gt; Is it possibl...
# nextflow-plugins
m
By @Andreas Schätzle
Hi Guys,
Is it possible to write / use a custom nextflow plugin without listing it in the plugin index(https://github.com/nextflow-io/plugins/)?
Yes, you can use a custom nextflow plugin that is not listed it in the plugin index. That’s how we do when it’s still under development and we want to try it you 🙂 But you won’t be able to load it with
plugins.id = ...
in
nextflow.config
.
a
If I can't load it, how can I use it in a workflow ?
m
Look at this plugin I’m working on currently: https://github.com/nextflow-io/nf-tweet
There are instructions there on the README.md file on how to use it (even though it’s not in the plugins repo you mentioned)
a
So NXF_PLUGINS_DEV pointing NXF_PLUGINS_DEV to the plugin location is enough? like this? export NXF_PLUGINS_DEV=$PWD/plugins
m
I’m not sure, but I don’t think so. You probably need more than simply this env variable (check https://github.com/nextflow-io/nf-hello/blob/master/launch.sh)
I will confirm and get back to you ASAP.
a
thank you
p
Stuff to add to your blog post @Marcel Ribeiro-Dantas 😅
m
Definitely. It’s already saved here 🤣 waiting a reply from Paolo regarding this topic
🚀 1
Just to confirm, @Andreas Schätzle: It’s indeed the only way, unless it’s in the plugins registry
p
@Andreas Schätzle - I'm curious about your use case. Can you share what your plugin is and why you don't want it listed centrally?
a
Gladly! Some activities in our pipeline produce data that should end up in another application (that might be temporarily down during pipeline execution). We'll probably use a message queue for this. We would like to avoid producing files that are then later parsed and then deleted. Also, we'd like to make it as easy as possible for the pipeline developers to push this data in our system. Also, the data from multiple activities might need to be consolidated before sending. So, we want to only send the final data at the end. This use case is pretty specific. So, we currently don't see the point in publishing it. And we have no experience writing plugins. so, we didn't even know that you could develop it without registering it beforehand. Maybe we are thinking to complicated. If there is a better way to achieve our goal, or if you think that this would be a worthwhile plugin, I'm happy to hear about it.
p
Any reason why this needs to be done via a plugin? Can it not just be a step in the pipeline?
m
I can imagine a situation in which a single pipeline would be tricky, but you can compose then. If I understood it right, it is expected that sometimes when the pipeline is finished, a service that is supposed to gather the results from this pipeline can be unavailable. Therefore, if we simply complete the pipeline, the data would never get to this service where we want it to be. You could have your pipeline always saving to a specific path X/Y/Z and have a second pipeline checking for availability of the service whenever new data is written to the path X/Y/Z (with
watchPath
), and even some timely triggering event with crontab. So that your pipelines finish just fine, and a second pipeline is periodically run to make sure the results are all delivered to service. What do you think, @Andreas Schätzle?
a
Thank you, both! I'll have a look into watchPath.
m
Just to be clear, you could have a single pipeline doing that, but if you had 100 runs, you’d have 100 Nextflow runs hanging waiting for service availability. With a second one, you would have 1 hanging waiting with
watchPath
😬
You’re welcome! 😉