Augustinas Malinauskas
09/06/2023, 9:27 PMSateesh_Peri
09/06/2023, 10:58 PMAugustinas Malinauskas
09/06/2023, 11:24 PMSateesh_Peri
09/07/2023, 12:24 AMJames Fellows Yates
09/07/2023, 4:07 AMJames Fellows Yates
09/07/2023, 4:10 AMJames Fellows Yates
09/07/2023, 4:12 AMJames Fellows Yates
09/07/2023, 4:18 AMJames Fellows Yates
09/07/2023, 4:19 AMMatthias De Smet
09/07/2023, 4:59 AMMarcel Ribeiro-Dantas
09/07/2023, 5:37 AMUnfortunately the pricing to have more than 5 members is waaaay too much for academic groups for just monitoring purposesThe Professional Tier is free for academics, @James Fellows Yates
James Fellows Yates
09/07/2023, 6:39 AMJames Fellows Yates
09/07/2023, 6:39 AMSimon Pearce
09/07/2023, 6:48 AMMatthias Zepper
09/07/2023, 7:01 AMAugustinas Malinauskas
09/07/2023, 9:52 AMHowever, I wanted to point out that the free version of the tower is actively maintained and already offers many of the features you've highlighted.Hi @Sateesh_Peri, thank you for the feedback. Yes nf-tower community offers same features right now, however looking at the commit history of nf-tower, the last non-docs commit was 3 years ago. I'm with you, I would have never done this if I though project was well maintained! Enterprise version is great, but I could not afford self hosted version.
Augustinas Malinauskas
09/07/2023, 9:57 AMAugustinas Malinauskas
09/07/2023, 10:04 AMJames Fellows Yates
09/07/2023, 10:20 AMMatthias Zepper
09/07/2023, 12:05 PMJames Fellows Yates
09/07/2023, 12:33 PMPhil Ewels
09/07/2023, 2:04 PMJames Fellows Yates
09/07/2023, 2:41 PMp
09/07/2023, 2:45 PMAugustinas Malinauskas
09/07/2023, 4:42 PMaggregating run metrics from isolated HPC systems that can not/are not allowed to freely connectDo you mean the network connection is not guaranteed during the time of execution, but is available eventually? That's an interesting one, let me think about it.
Augustinas Malinauskas
09/07/2023, 4:42 PMAugustinas Malinauskas
09/07/2023, 4:43 PMJames Fellows Yates
09/07/2023, 7:09 PMAugustinas Malinauskas
09/07/2023, 7:19 PMJames Fellows Yates
09/07/2023, 7:21 PMAugustinas Malinauskas
09/07/2023, 7:23 PMJames Fellows Yates
09/07/2023, 7:26 PMJames Fellows Yates
09/07/2023, 7:27 PMAugustinas Malinauskas
09/07/2023, 7:28 PMmoritz
09/07/2023, 8:34 PMAugustinas Malinauskas
09/07/2023, 9:52 PMmoritz
09/07/2023, 10:12 PMSimon Pearce
09/07/2023, 10:28 PMMatthias De Smet
09/08/2023, 4:31 AMJames Fellows Yates
09/08/2023, 6:39 AMSimon Pearce
09/08/2023, 7:28 AMMatthias Zepper
09/08/2023, 8:24 AMaggregating run metrics from isolated HPC systems that can not/are not allowed to freely connect
Do you mean the network connection is not guaranteed during the time of execution, but is available eventually? That's an interesting one, let me think about it.What I meant is that nf-shard basically serves two functions: Live-monitoring of running workflows and posterior analysis and review with elegant visualizations. For the former functionality, a persistent connection is essentially inevitable. But for the latter, once could aggregate the metrics of many runs locally in e.g. a SQLite database with a lightweight deamon and then just retrospectively import them to nf-shard to review how efficiently the processes used the requested compute resources or to keep an easily accessible record of a run. But on a second thought, I feel that is so different from what you are building (it essentially more resembles MultiQC, just that is does not parse logs, but collects metrics at runtime via an API) that you should probably not further bother about it. The whole thing just sprang to my mind, because processing human genomic data on cloud infrastructure or on HPCs that can freely connect to the internet seem to be a legal no-go within the EU. For example, the NGI together with the SciLifeLab data center developed a data-delivery system that would allow us to deliver encrypted sequencing results via S3 compatible storage to our users. It took us a couple of months to develop, we had external security/encryption experts reviewing & approving the system and CLI and version 1 was ready in March 2022. Our initial timeline was, that latest in summer 2022 it would replace our legacy delivery infrastructure. Now, after more than one year, delivering human data via this system is still blocked by some legal shenanigans, while the legacy system causes mind-boggling costs. The key problem here is essentially responsibility. When you develop an own software or use a free and open-source software, somebody from the higher-ups in the department or university will have to take the flak when something goes wrong. When you licence a commercial product, you defer responsibility (at least to a large extent) to the company you are buying it from and that makes it very attractive to managers 🫣.
Augustinas Malinauskas
09/08/2023, 3:08 PMBut for the latter, once could aggregate the metrics of many runs locally in e.g. a SQLite database with a lightweight deamon and then just retrospectively import them to nf-shard to review how efficiently the processes used the requested compute resources or to keep an easily accessible record of a runThat's an interesting idea @Matthias Zepper. Alternative is to run workflows
-with-report to produce report.html . These reports contain data about workflow, runs, progress in js datatypes. I could build ability to import such reports into nf-shard. This way any old reports can be imported as well as well as no need to host a daemon
It's pretty wild how legals can slow down the innovations.Konrad Rokicki
09/11/2023, 5:45 AMMatthias Zepper
09/11/2023, 7:27 AMAlternative is to run workflowsThanks for pointing out this option. Importing those reports then is clearly the better approach to what I fantasized about! 😊 We have been using Tower for two years to review runs, I lost track of possible alternatives. I don't know why or for what reasons exactly, but somehow everyone in our team is convinced that using Tower gets you more details and better logs. This belief is reiterated in all relevant meetings, but I start to wonder how big the difference truly is? 🤔 I think, I should run a few comparisons, because we also work on the assumption that a pipeline started withto produce-with-report. These reports contain data about workflow, runs, progress in js datatypes. I could build ability to import such reports intoreport.html. This way any old reports can be imported as well as well as no need to host a daemon.nf-shard
tw launch will give you more insights than a nextflow run --with-tower. Since the latter would (for now) simplify our configuration, it would be annoying if the tw launch / nextflow run distinction was just a hoax.James Fellows Yates
09/25/2023, 5:05 PMAugustinas Malinauskas
09/25/2023, 5:19 PMAugustinas Malinauskas
09/25/2023, 5:22 PMJames Fellows Yates
09/26/2023, 8:56 AMdocker-compose option but it failed, I made an issueAugustinas Malinauskas
09/26/2023, 9:03 AMJames Fellows Yates
09/26/2023, 9:17 AMAugustinas Malinauskas
09/26/2023, 9:18 AMMatthias De Smet
09/27/2023, 7:50 AMAugustinas Malinauskas
09/27/2023, 9:38 AM