Hi everyone! I've been working on a project <nf-sh...
# shameless-promo
a
Hi everyone! I've been working on a project nf-shard which is an open source alternative to nf-tower. It has nice UI which monitors running workflows, shows metrics, traces, errors, resource utilisation, estimates pricing, allows to filter old workflows. I've picked stack NextJS/React/Typescript/Postgres with large communities hoping to get some collaborators on board. Please let me know what kind of features you'd find useful! Video:

https://www.youtube.com/watch?v=Fzq9cqozwEU▾

Demo: https://main.d29s1tosoaghp.amplifyapp.com/runs Github: https://github.com/AugustDev/nf-shard Latest updates: •

Slack notifications▾

• Workspaces •

One click deployment with Vercel▾

👀 2
👍 1
🙌 6
❤️ 24
s
hey Im just curious as to why you had to undertake this project as the free version of the tower allows all the monitoring and shows metrics etc you mentioned 🤔
a
Hi @Sateesh_Peri! Mostly because it's been unmaintained for years and I want to add some features like executions logs, advanced filtering, better attachments etc.
🙌 2
s
got ya....here are my 2cents.....Firstly, I'd like to commend you on the effort and dedication you've put into nf-shard. It's evident from the video and demo that you've put a lot of thought into the project. However, I wanted to point out that the free version of the tower is actively maintained and already offers many of the features you've highlighted. While I understand and appreciate the desire to innovate and bring new features to the community, there could be a concern that creating parallel projects might inadvertently divide/confuse the community. It might be more beneficial to collaborate and contribute to the existing project, ensuring that the community remains united and benefits from collective efforts. That said, I genuinely appreciate the passion and drive behind your project. I just wanted to share this perspective for consideration. Cheers!
❤️ 1
j
@Sateesh_Peri I'm going to say the opposite and actually I'm really excited this has been developed! I love tower, however my department only uses it for monitoring and debugging, we won't use it for launching/datasets/cloud stuff etc. Unfortunately the pricing to have more than 5 members is waaaay too much for academic groups for just monitoring purposes. Of course Seqera needs income, which I'm sympathetic to, but we simply can't afford essentially enterprise/industry costs. Therefore I'm actually happy there is an alternative that may have just what we need, and I disagree that you should always contribute to an existing project.
❤️ 2
💯 8
Of course I would rather have a single project and everyone works together but an alternative option is sometimes good (even if simplified), and maybe the projects can take ideas from each other. So @Augustinas Malinauskas I'm very interested so will take a look. Dont know how much I can contribute in terms of code, but testing/ideas might be possible (currently on parental leave). @Thiseas C. Lamnidis might be too.
➕ 1
❤️ 1
(basically I was sad that the less advanced and less regularly updated community of version of tower was abandoned, as that had exactly what we needed and it I could keep people excited about using tower, it has become more difficult since)
❤️ 2
@Augustinas Malinauskas as far as I can see the two things I think are missing for us (or it might only be one): 1. Error reporting (does shard show the error of a process that has failed) 2. Ability to separate/isolate groups of runs (etc the workspace of tower)
❤️ 1
(2. For us is to separate automated runs versus user runs, so it's easier for us to find the user runs)
❤️ 1
m
I for one love this initiative! We can’t use any cloud based stuff in our hospital, so this is awesome to have!
💯 5
m
Unfortunately the pricing to have more than 5 members is waaaay too much for academic groups for just monitoring purposes
The Professional Tier is free for academics, @James Fellows Yates
❤️ 1
j
We weren't granted it unfortunately
❤️ 1
To be clear, yes there is an academic tier but for smaller groups it appears
❤️ 1
s
We perform all our computation on our HPC and our sysadmins are keen to keep it that way, so something that can we can host ourselves is usable for us (without paying out mega-bucks for the on-prem version of tower, which is never going to happen)
💯 5
❤️ 1
m
Very impressive work @Augustinas Malinauskas 🤩. Admittedly, I have just skimmed over the repo and watched the demo video, but I think this has a lot of potential. I am surprised that you opted for a relational database, since a document database like MongoDB or a mixed-model database like SurrealDB would probably accommodate the different run metrics and logs from Nextflow better, but evidently you got it to work nicely! Kudos! I think you might want to consider some UI redesign, so it is not too similar to nf-Tower, but it is clearly a promising project!
❤️ 1
a
However, I wanted to point out that the free version of the tower is actively maintained and already offers many of the features you've highlighted.
Hi @Sateesh_Peri, thank you for the feedback. Yes nf-tower community offers same features right now, however looking at the commit history of nf-tower, the last non-docs commit was 3 years ago. I'm with you, I would have never done this if I though project was well maintained! Enterprise version is great, but I could not afford self hosted version.
✅ 1
Thanks for feedback @James Fellows Yates! 1. Yes it will show error the same error you'd see in the console. Current examples do not show error message, because I just cancelled the runs. I should have nicer message when this happens - good idea! 2. Adding workspaces to group runs sounds like a great idea, fairly easy to support, will add to my list!
❤️ 1
Hi @Matthias Zepper thanks for feedback! I was considering Mongo, but chose Postgres due wider adoption. I use Prisma ORM to interact with database, interestingly it supports MongoDB too. Most likely I can support both Mongo and Postgres with little changes to schema. 🤔
j
That would be awesome @Augustinas Malinauskas and would cover everything we would need for our department 😄
m
Hi @Augustinas Malinauskas: No don't do that - you'll only get bogged down with supporting two databases. Your users will anyway not care what database it runs on and performance issues will hardly occur with the amount of data individual groups or users will store. So the only important criterion is your developer experience - how easy you can achieve what you want to build with the foundation your are building it on. You have chosen a solid ORM and in particular if you have previous experience with SQL databases and schemas, you will be more productive than trying to learn something new on the fly. Personally, I just happen to like the flexibility beyond schemes and strict tabular data structures that NoSQL databases offer, e.g. to save nested objects, vectors or time-series measurements easily, but I have essentially just been toying around with them. I didn't really think this through, but just remembered that Nextflow handles a lot of configuration/oputput as JSON and so thought of MongoDB or completely schema-less databases as a possible way of storing that output flexibly. But reading through the other responses in this thread, it seems there is one feature in particular demand - aggregating run metrics from isolated HPC systems that can not/are not allowed to freely connect to SaaS applications, but where a Tower on premise license is financially out of the question. Since such systems usually also don't have a running window manager, only a TUI would work on the system itself. But you could support that use case with nf-shard - either with tPRC (which you already have on your roadmap) or with a database architecture modeled after that of mobile applications. Unfortunately, I have very little knowledge about such architectures, but it is a common issue that mobile apps need to work with intermittent internet connection (in particular in Germany 😄), so there are likely many tutorials and open-source implementations available. As far as I know mobile apps often feature a lightweight database on the client that is then synced with the backend when possible. You could pack the relevant parts of your TS application into a standalone executable with bun and sync the collected data to nf-shard in batches, e.g. via an allowed proxy. I am aware of the combo RealmJS/MongoDB, but I guess there are more modern approaches in the meantime? But despite all those fancy ideas: KISS is very important, in particularly for early-stage minimal viable products. So once more: Really cool project, I will keep an eye on! 😊
🇩🇪 1
💯 2
j
nextflow pipeline run tracking from my phone 👀 👀 👀 👀 👀 👀 👀
😂 2
p
Just to note: Tower Pro academic license is not limited by group size. It's down to how we class places as academic or not.
this 1
👍 1
j
Yes sorry, as far as I know we were not granted it at the time due to the number of runs (from automated processing), not the size of the the research group.
p
Congratulation for the achievement! It looks like a nice project; however if you want to push it as OSS you should add a licence to it 😉
a
Thanks for insights @Matthias Zepper, much appreciated! I agree with KISS and making sure project is as easy to understand as possible. Database migrations with Prisma ORM are very pleasant. I like the idea of NoSQL as well these days, but didn't wanted to stem into too unfamiliar territory with this one. Regarding
aggregating run metrics from isolated HPC systems that can not/are not allowed to freely connect
Do you mean the network connection is not guaranteed during the time of execution, but is available eventually? That's an interesting one, let me think about it.
Good point @p - added MIT license!
👍 1
🚀 1
@James Fellows Yates I implemented basic Workspaces. What do you think? What could be improved? All runs can now be grouped into a workspace. You can filter by workspace, it also shows up in the run details. I indexed it on database for fast queries.
🤩 1
j
That looks perfect to me, thank you very much @Augustinas Malinauskas! That's pretty much everything Ive been looking for! I'm technically on parental leave at the moment but I'll see if I can get someone at work to try it out (ping @Thiseas C. Lamnidis)
a
Sounds good @James Fellows Yates! I'll write some notes over the weekend of how to deploy it with least effort as possible! I was running it on Vercel and AWS Amplify.
❤️ 1
j
That would be great, i would be on the same boat as @Matthias De Smet in that we would not be able to run on cloud
❤️ 1
a
Could you share whats possible for you, so I can tailor notes? Are you planning to run it on your own server?
j
Yes exactly. Something hosted on our local network, and a most likely a Ubuntu OS
If you have requirements let me know and I can answer questions accordingly
👍 1
a
Got it, should be fine! Requirement is only postgres database and running server.
👍 2
m
Isn't the docker compose setup already perfect for self hosting?
a
Hi @moritz, yes that could work too!
m
Though I guess Docker is a no-go on exactly those private clusters that can benefit most from nf-shard.
s
Yeah, Singularity/Apptainer is the alternative we use, for general use to run one command you can just use the docker container or (preferably) you can convert painlessly to a singularity container. Don't know about this use.
m
Docker containers is the way to go for us! We run all our web applications containerised on a nomad cluster
j
Docker probably could also work us as it would be a separate server ratchet then a cluster, but Moritz and Simon makes a good point if people want to run it themselves
s
Yeah, our sysadmin team uses docker for VMs like webapps, so it may be sufficient to use that and be permitted to talk to the cluster from the docker image.
👍 2
m
aggregating run metrics from isolated HPC systems that can not/are not allowed to freely connect
Do you mean the network connection is not guaranteed during the time of execution, but is available eventually? That's an interesting one, let me think about it.
What I meant is that nf-shard basically serves two functions: Live-monitoring of running workflows and posterior analysis and review with elegant visualizations. For the former functionality, a persistent connection is essentially inevitable. But for the latter, once could aggregate the metrics of many runs locally in e.g. a SQLite database with a lightweight deamon and then just retrospectively import them to nf-shard to review how efficiently the processes used the requested compute resources or to keep an easily accessible record of a run. But on a second thought, I feel that is so different from what you are building (it essentially more resembles MultiQC, just that is does not parse logs, but collects metrics at runtime via an API) that you should probably not further bother about it. The whole thing just sprang to my mind, because processing human genomic data on cloud infrastructure or on HPCs that can freely connect to the internet seem to be a legal no-go within the EU. For example, the NGI together with the SciLifeLab data center developed a data-delivery system that would allow us to deliver encrypted sequencing results via S3 compatible storage to our users. It took us a couple of months to develop, we had external security/encryption experts reviewing & approving the system and CLI and version 1 was ready in March 2022. Our initial timeline was, that latest in summer 2022 it would replace our legacy delivery infrastructure. Now, after more than one year, delivering human data via this system is still blocked by some legal shenanigans, while the legacy system causes mind-boggling costs. The key problem here is essentially responsibility. When you develop an own software or use a free and open-source software, somebody from the higher-ups in the department or university will have to take the flak when something goes wrong. When you licence a commercial product, you defer responsibility (at least to a large extent) to the company you are buying it from and that makes it very attractive to managers 🫣.
👀 1
😬 2
❤️ 1
a
But for the latter, once could aggregate the metrics of many runs locally in e.g. a SQLite database with a lightweight deamon and then just retrospectively import them to nf-shard to review how efficiently the processes used the requested compute resources or to keep an easily accessible record of a run
That's an interesting idea @Matthias Zepper. Alternative is to run workflows
-with-report
to produce
report.html
. These reports contain data about workflow, runs, progress in js datatypes. I could build ability to import such reports into
nf-shard
. This way any old reports can be imported as well as well as no need to host a daemon It's pretty wild how legals can slow down the innovations.
k
Incredible work @Augustinas Malinauskas, bravo! Is your vision to keep it as close to nf-tower as possible, or is there room to diverge? As useful as nf-tower is, there are a lot of UX improvements I would love to see in it, but like you said, it has been stagnant for a while. What are your thoughts on contributions to this project? Would you accept PRs and potentially grow it as a community project?
m
Alternative is to run workflows
-with-report
to produce
report.html
. These reports contain data about workflow, runs, progress in js datatypes. I could build ability to import such reports into
nf-shard
. This way any old reports can be imported as well as well as no need to host a daemon.
Thanks for pointing out this option. Importing those reports then is clearly the better approach to what I fantasized about! 😊 We have been using Tower for two years to review runs, I lost track of possible alternatives. I don't know why or for what reasons exactly, but somehow everyone in our team is convinced that using Tower gets you more details and better logs. This belief is reiterated in all relevant meetings, but I start to wonder how big the difference truly is? 🤔 I think, I should run a few comparisons, because we also work on the assumption that a pipeline started with
tw launch
will give you more insights than a
nextflow run --with-tower
. Since the latter would (for now) simplify our configuration, it would be annoying if the
tw launch
/
nextflow run
distinction was just a hoax.
j
@Augustinas Malinauskas did you get time for writing the deployment documentation?
a
Hi @James Fellows Yates, yes I updated Readme and proposed few ways to run it 1. Docker compose (to run both nf-shard + PostgreSQL or just nf-shard if you have PostgreSQL running elsewhere) 2. yarn package manager - this is most useful if developing nf-shard 3. One click deployment using Vercel. I made example video

here▾

Please let me know if this works for you. Very happy to brainstorm for other solutions!
❤️ 1
@Konrad Rokicki apologies for the delay! I think feature set will diverge. I'm quite interested in features like Slack notifications, ability to import old reports and anything else that can increase adoption. Contributions to the project always welcome! I intentionally picked one of the most popular technologies to maximise potential collaboration. 🙂
j
I tried the
docker-compose
option but it failed, I made an issue
a
Hi @James Fellows Yates apologies for this. I fixed it and replied to your issue
j
Bingo that worked 😄 thank you veyr much @Augustinas Malinauskas! I'll try asking for a small VM for our instite now 🙂
a
Jay! Let me know how it goes!
👍 1
a
Just released Slack notifications -

Video of how to set it up▾

m
Awesome! Do you reckon this would also work with MS teams hook URLs?
a
I can also support it for MS teams, very few changes needed @Matthias Zepper
❤️ 2