Building Together: An Open Invitation to Collaborate on nf-shard • Nextflow, nf-core, and GitHub have made it easier than ever to collaborate on powerful, scalable pipelines. But while the core technology has grown tremendously, monitoring and managing these workflows often still requires command-line expertise and manual log checking. • Many teams find themselves wishing for something simpler — a visual, open-source interface to track workflows, collaborate across environments, and support diverse infrastructure setups. That's where nf-shard comes in. 🧬 Why nf-shard? As bioinformaticians working in cross-functional teams, we've consistently seen a few needs come up across labs and institutes: • Teams want to run pipelines on-premise, in the cloud, or both — without switching tools. • Scientists, analysts, and project managers need visibility into what's running, where, and why. • Many want something lightweight and OSS-friendly to deploy privately or in regulated environments. While some enterprise platforms address these needs at scale, there's still room — and real demand — for an open-source option that's flexible, approachable, and community-driven. We believe that is what nf-shard provides. 🧱 What's Been Built So Far (credits to @Augustinas Malinauskas (github) ) • nf-shard features a clean, web-based interface to monitor and inspect Nextflow workflows, with real-time status tracking, resource usage visibility, error reporting, and workspace-based organization. • Under the hood, a modern stack designed for both performance and developer accessibility: Next.js and React provide the responsive frontend experience, TypeScript ensures code reliability, while PostgreSQL with Prisma ORM handles data persistence efficiently. This combination was specifically selected to maximize open-source contributions from the community. • The architecture also includes a partially developed shard-worker service to enable workflow launching directly from the UI, keeping everything modular and open to feedback as real-world usage grows. 🔭 What's Coming Next We're now focusing on expanding nf-shard to support more execution environments and improve integration: • 🧪 HPC backend support, with help from Plant & Food Research (NZ) • ☁️ Cloud executor enhancements, supported by MemVerge and AWS • 🧰 Ongoing development and deployment testing by contributors at PeriMatrix IT All work will continue under the same MIT license and remain completely open-source. 🚀 Try It Now 🔗 **View Live Demo** or deploy locally with one command:

curl -s \
    <https://raw.githubusercontent.com/GallVp/nf-shard/refs/heads/main/nf-shard-deploy-local.sh> \
    | bash

This single command pulls the latest version, sets up the PostgreSQL database, and launches the web interface at

localhost:3000

. From there, simply add the provided tower configuration to your

nextflow.config

and start monitoring your workflows immediately.

🤝 How to Get Involved If you've ever wished for a GUI to make working with Nextflow more accessible — or if you're already building around HPC/cloud hybrids — we'd love your input. • 💻 Explore the code and roadmap on GitHub: https://github.com/GallVp/nf-shard • 💬 We're happy to start a dedicated public channel in the nf-core Slack for easier collaboration • 🧩 We welcome all forms of contribution — from bug reports and testing to development and feature ideas Whether you're an individual user, part of a core facility, or just curious about open science tools — this is a great moment to help shape the future of nf-shard. Thanks for reading — and if you're interested, let's connect and build together. The nf-shard community team

This is interesting!!

This has been around for a while now. We are now dedicating dev hours for making hybrid setups work via nf-shard

Yes we have tried it before 😛 I'm glad to see it gets some love

we had to stop @Jing Xie @ memverge from rolling out a similar tool and convince them and other stakeholders to invest in open-source nf-shard instead 💪

I can only support that! 😉

#C093NR8A5MW 🐇

Out of curiosity, was there no way to build on the open telemetry ecosystem? Seems that would be a natural fit.

nf-shard was the obvious choice for Plant&Food to jump on board as a community member (@Augustinas Malinauskas) had already spent quite a bit of time setting it up. Instead of starting something new or changing the tech stack, we thought it is better to add to the existing effort.

@moritz what do you mean by open telemetry ecosystem?

I mean sending telemetry data using the open telemetry standard. Having a collector and then using, for example, Grafana for visualization. https://opentelemetry.io/docs/

The observability part is already there in

nf-shard

and it if fully compatible with the Nextflow Tower API. I am currently adding session and workspace authentication (https://github.com/GallVp/nf-shard/issues/18). The next step is to get the current MM Cloud nf-shard-worker plugin (https://github.com/gluonfield/shard-worker) generally available. It was working for CFDX and most of the code is already there. It has to be generalised. With this plugin in place, we can use

nf-shard

to both observe and launch workflows. This plugin can then be ported to other executors such as AWS Batch and SLURM.

Don't worry about it. I think, it would be the right use-case for Open Telemetry and there is good tooling to support it. However, you have already created a lot, so it's too late to change course for nf-shard. And I at least don't have the time or energy to create another plugin. Just ignore my musings 😉

I like your thinking btw. 🙂 OTEL and prometheus endpoints would allow everything to be plugged into external monitoring systems. 🙂 For another day I suppose

Interesting - I built something somewhat similar lately that monitors nextflow jobs that are being executed with qsub/qstat. I found it hugely more productive being able to jump around between my different nextflow processes on the web and bring up the real time logging to see how things are progressing etc