So a confession first, I spent two months building...
# nostupidquestions
q
So a confession first, I spent two months building a tool and only now thought to ask whether anyone needs it in bioinformatics. Classic engineer right? Measure never, cut twice. Background. I come from regulated software industry, where traceability is not a hobby, it is a compliance requirement. Every change points to a requirement, every requirement points to a test, and when a bug is found you must say exactly which product type it reached. Auditors visit. They do not laugh istg! So I assumed bioinformatics was the same, and built github.com/QuietFlare/clew. It reads the Nextflow lineage store and answers for ex: these two samples were swapped at intake, what did they touch, and what do I do with each output, and can seal that answer as a report. Now the stupid question. Is that a real need? When a reference input or container turns out faulty after a run, does anyone in clinical or regulated settings have to produce an impact record an auditor would ask and accept? Or does everyone just type -resume, say a small prayer, and move on? I would honestly like to know which world I built this for.
m
We have a channel for that! -> #C05V42SU8DP
party this 6
a
I'm trying to understand what the tool actually does, it sounds interesting and something I could use (also in a regulatory setting but not as strict as you've described).
q
Thanks Ammar! so the tool reads the data lineage record nextflow writes, every task, what files went in, what came out. clew just reads that, touches nothing in the pipeline. then you tell it what went wrong (an incident), like “seqtk container was bad” or “sample 3 got mislabelled”, and it walks the graph and tells you per task: rerun this, quarantine that, this one was never touched so leave it. with the file chain as proof. if it cant prove something from the record it says so instead of guessing. and you can also seal the whole answer into a bundle which someone can verify later, thats the auditor part. I’ve put a codespace setup in the readme to quickly check the behaviour, two demo runs already in there. If you’re not on latest Nextflow and have no lineage store, it reads the work directories instead. I’m curious what your setting is and how this could fit in your setup? happy to do a quick demo if you’re interested
a
Ahhh I follow, it's an quality auditor very cool.
👍 1