We'd like to collect enough workflow execution meta data that we can reproduce the same results running outside of seqera, or on a different seqera environment. This would include but not limited to gitrepo, githash, dataset (Let's assume we have the datasets backed up/replicated on whatever platform we're running on), nexftlow version, input params etc... Is there already a way to produce such a file (json preferably)?

This is a bit of an open-ended question, as there are so many ways to run pipelines. The git side is easiest, as running specific commits of pipelines is already baked into how Nextflow and Platform work, so that should be as simple as selecting something specific when adding to the launchpad in Platform, or using

-r

if running Nextflow on the CLI.

I activated the lineage plugin. Looks like I'm getting most of the data I need, although there's some stitching to be had. Looks like I'm lacking 2 things 1. a "lineage event" for the publish. The current lineage only points to work/scratch that could be considered ephemeral. The published artifacts are what would be retained and what would be referenced from other workflows, hence, that's the uri I want to associate in the lineage. 2. I'm currently supplying datasets through the launcher form. They show up in lineage as urls related my seqera workspace. In a non-seqera context these means nothing. It would be nice if the lineage would reference the absolute s3 uri for the dataset. I think it's possible and not that hard to consume lineage from an executed workflow and augment it with these things (maybe given some conventions) but it doesn't sound impossible to add these features to the lineage functionality I'm having a chat later today with people from Seqera. Lets see what comes out of that

Yup, you've pretty much hit the nail on the head with our intended direction of travel with lineage

I though the datasets that's uploaded to seqera was actually stored on S3?

If you're talking about https://docs.seqera.io/platform-enterprise/data/datasets then no, they're stored in the Platform database

Aha... my bad!

No problem! It's a bit confusing with many concepts flying around sometimes 😅

I don't think this is a blocker. We'd just have to make sure the datasets are available and possible to crossreference outside of seqera