• I heard about a proposal to keep using tuples for process inputs/outputs with static typing. Don't do this. It is not the way. I'm happy to discuss this in detail, and I will add more in 🧵 , but here I just want to make clear that I do not think this is a viable path for nf-core.

Thanks for the overview! I don't think anyone disagrees about the technical benefits. But at the crux of the matter is indeed the 'picking your poison' bit: I definitely would pick arguing over tuple ordering over arguing about naming any day. And this is sort of the impasse we are reaching: if we want to retain a certain of level of nice compatibly between nf-core modules (to allow as much plug-and-play as possible, or to keep modules in line as possible so you _dont_have to do too much renaming) we will need to have a standard vocabulary across nf-core modules including decision makers, infrastructure for maintaining such a standard vocab, etc. A regular contention point in the past has been people wanting to hardcode meta fields in modules. But then arguments about the names of the fields should be would then inevitably breakout. While one could argue while technically the code might cleaner in reduced number of LoC/shorter code lines in some areas, as @Jim Downie points out, if you are constantly having to rename your record fields all the time away from their domain specific 'purpose' or 'meaning' within your pipeline, it becomes much harder to understand what the pipeline is doing with these fields. For example what a 'fasta' record field refers to may change meaning 10 times, for example whether they are all reads, a reference genome, assembly, bin etc. is then lost. When we were discussing this standardisation of field names at a module level, and another issue that Mahesh pointed out as we also have experienced even in the current system of modules, where someone designs a module for a very specific/narrow use case, but then later someone wants to use the module for another use case - having to redesign these have been painful. Having suboptimal record names would exacerbate this particularly if we incorporate standard naming. This is less bad in very highly standardised fields like cancer genomics (or people who are using samtools regularly), but this will become extremely painful for many others: which comes back to the concept of whether a module should serve a pipeline or whether pipelines should serve modules. Which ultimate is related to how much freedom/flexibility we should be offering users - and I think this has served us very well so far.

Yeah, we're all on board with the technical advantages. It's more about the people here and how quickly they should have to adopt to the new features and syntax. It would be great if the pipeline maintainers all had time to keep everything up to date but we see they don't. LLM's etc aren't a solution since it's not just a matter of getting things done faster. Keeping tuples for now reduces the burden on the maintainers to learn all the new stuff that gets dropped. The change will have to come eventually, but for many this is a very large ask to convert entire pipelines or keep patching until they can find the time to keep up.

Yeah, it would give us more time to address the question of record field standardisation (or not)

Obviously I'm team Ben here, and I want all the cool type checking etc, but I'm also a pipeline maintainer of a few pipelines, and I'd MUCH rather do one comprehensive migration than a death-by-a-thousand-cuts set of micro-migrations. Also, I want consistency. We pretty much HAVE to have record types at pipeline boundaries, and being forced to keep tuples internally, preventing me from accessing the nice type safety and forcing me to retain a Frankenstein's monster of standards is going to make me very sad. Also, as a commercial team providing services to customers (and I appreciate that's the not majority of nf-core right now), we need to be able to provide the strong features that make customers happy. I'm worried that nf-core sticking with outdated practices will force a parallel ecosystem that does implement those features, which won't be good for nf-core in the long run.

What are peoples stances on standardised vocabulary?

I'm not convinced it's feasible given our ecosystem

Although note that I am slowly being ground down and may capitulate. E.g. I guess with the Nextflow module registry I can also imagine we may end up allowing non nf-core modules in nf-core pipelines. And at that point any form of naming standardised structure/system is out of our hands.

I want standardized stuff like params, fields in samplesheets...

Yeah we discussed that yesterday: I think at a pipeline level having such standardisation is much easier because you can come up 'within-domain' standards amongst related pipelines. Modules is a competely different ball game

My thoughts: 1. My suggestion for a migration static typing with tuples is merely that it should be a stopgap. We auto-port all our modules to the new typed format, keeping tuples, and THEN we can think about how we are going to use records. This facilitates transitions to statically typed workflows without significantly reinventing the wheel. Otherwise we're going to be very slowly converting all modules to records, as we have done with topic versions. 2. As I've said before, I think the real advantage of records is at the pipeline level, to represent pipeline-specific or domain-specific constructs, like samples, references, or particular analyses, and this conflicts heavily with processes requiring generic-enough input names, where the names of record fields are in flux in order to accommodate what the modules require. Really processes should be functions that take arguments with an

arg=value

format, where you /map/ your record fields directly onto the input arguments of a process, and you don't permanently modify a record by renaming fields. This also sidesteps the problem of structure entirely - processes take flat inputs and how a pipeline gets them to a module is entirely up to the pipeline. 3. On the topic of a standardised vocabulary, I think such a thing would be hard for the domain-specific things discussed already. But also, I think we would have to have a system of hyper-generic input names ("fasta", "alignment", "tsv", etc.) with a system of super-specific output names (e.g. "minimap2_align_bam", "samtools_faidx_fai") so that when we join records together, there's no risk of overwriting fields accidentally (than if all modules that write BAM have an output "bam" field, for example).

I am starting to agree with this as well

I think the other thing I'm really starting to feel is that the drip-feeding of these new features has made this more complicated. I appreciate you didn't want to straight drop a DSL3 on us, but it might be better if we just had a target NF version where we said, the old stuff is out and the new stuff is in. And we could hard-branch our modules or whatever and switch once to the new mode when a pipeline is ready rather than awkwardly transitioning between them, chasing each new feature

Yes, for me as @Jim Downie and @Mahesh Binzer-Panchal point out, the main issue is about people having the resources to migrate and I got some comments of people getting tired of all these changes happening very often in nf-core and that is impossible to keep the path. The naming issue I think is not so problematic as, even if it is not the most optimal, people can always rename the records as they want

Ok then I think the debate is not necessary 😆 ???? I guess everyone is OK with records in principle - arguing about how/when Nextflow is rolling out stuff and how/when nf-core implements this is not really relevant to this specific discussion at this point. It's always going to painful regardless which way we go (one very painful stabbing pain [one difficult change which breaks a everything] vs a long running low-level bruise pain [too many small gradual migrations] 😉 ). If I understand it correctly, people don't see the naming of record fields thing as much of an issue(?), or it's just a minor inconvience? If that is indeed the case -> we go all in on records; wait to see how much people complain about the lack of standardisation of names and hardcoded names in nf-coremodules etc? And if it is a BIG problem then we try and address that challenge then

I think it would be still good to get a consensus from the room.

Also, I want consistency. We pretty much HAVE to have record types at pipeline boundaries, and being forced to keep tuples internally, preventing me from accessing the nice type safety

If you have type-safety at the boundaries, do the internals - which you control as a pipeline dev - really matter?

I think the other thing I’m really starting to feel is that the drip-feeding of these new features has made this more complicated

100% agree On the naming issue, named inputs are both great and awful. You now have the ability with named record fields to be order-agnostic and clearer to a reader about what the input is. However if you just end up having to rename half your fields between modules then that is going to be no better and possibly even a considerably worse user experience, than having to occasionally re-shape channels. In that sense I think

nf-core

should absolutely aim for generic and standardised field names.

If you have type-safety at the boundaries, do the internals - which you control as a pipeline dev - really matter?

Yeah @Julian Flesch proposed this - I liked this too. But @Jonathan Manning and @Nicolas Vannieuwkerke didn't want to rompcromise

I think ultimately typed records throughout is the ideal, but realistically if I control the internals it’s less of a concern. I think that some of the push to have modules use record types is presumably because that supports the standalone

module run

command stuff, which is neat, but as a pipeline dev not really important to me.

For me personally, I'd prefer to have named inputs because it's just clearer for developers since you can annotate inputs now instead of just using the order of inputs (which gets nasty for big tuples)

I think the other thing I’m really starting to feel is that the drip-feeding of these new features has made this more complicated

I would also much prefer the shorter stabbing pain. As for the naming, I think we can meet halfway be defining naming conventions for frequently used inputs (eg.

reference_fasta, reference_fai

) while also allowing custom naming for less frequently used or domain specific stuff. Output naming should be process specific along the lines of

module-name_output-name

Well I think generally it would be good to have inputs and outputs that are standardised names e.g.

vcf

vcf_index

etc because then I have to do fewer channel manipulations and renaming of record fields no?

Are you guys willing to arbitrate the choices of these names?

But what if you want to join two records together that both have a

vcf_index

from different processes? e.g. SVs and indels for one sample

The module call might be clearer with named inputs, but that is little benefit if I now have channel manipulations either side of every process. And what is the compute/memory overhead of renaming of record fields?

I do think we would have to attach module names to the output fields to facilitate joining

Having join hygiene is something the pipeline developer needs to do, not the module developer

> But what if you want to join two records together that both have a

vcf_index

from different processes? Well in that case you rename them - but everyone who doesn’t need to, doesn’t have to?

there is that

pulling smth out of my behind here

BCFTOOLS_VIEW(vcf: ..., vcf_index: ..)
BCFTOOLS_INDEX(vcf: BCFTOOLS_VIEW.out.bcftoolsview_vcf)

so standard inputs, custom out

Yeah standardization will just be hard, it's never going to be perfect, but I still prefer that over big tuples

I repeat my question though:

Are you guys willing to arbitrate/curate the choices of these names?

new nf-core team? #team-pedantic

Are you guys willing to arbitrate/curate the choices of these names?

Are you asking if I want to wield the hammer of power? 😂 But I think generally this not much different from out current situation in nf-core, most places that output a vcf call it

vcf

?

#C0C9L0J9V35 channel created for you @Matthias De Smet

most places that output VCF might call it

vcf

but I don't think that's the same for FASTA

I’ve seen

fasta, reference, fa

all pointing to a reference genome

I think I can think of some modules that needs or output different fasta files

Are you sure: https://nfcore.slack.com/archives/C07ACNLELR2/p1792069604226669

hence

reference_fasta

🤷🏻 as an example

Yes, for alignment - fasta, reference, genome, assembly might all be valid names. But FASTA could also represent amplicons, primers, or other short sequences, ORFs, etc. One presumes we would want to be specific about what a module wants -

orf_fasta

,

reference_fasta

etc.

I think if generic names work for say 75% of modules though, then that is beneficial imo

That is true

defining naming conventions for frequently used inputs (eg.

reference_fasta, reference_fai

) while also allowing custom naming for less frequently used or domain specific stuff

That’s what I meant here

Though sometimes domain-specific tools (or perhaps more commonly, tools implemented in a domain-specific way) need to be changed

we have robots for that IMO. the barriers between a feature request and a feature PR have never been smaller for tools.

@here We'll have a hackathon discussion at 11 AM CET, we'll open a Slack huddle for any online people that want to join

OK, maybe I'm being convinced that /naming/ per-se is not such a problem. But I do think that constant renaming makes having "semantic" records - that represent e.g. a sample, an analysis, etc. less easy. Though again, perhaps these are most important at the boundaries - inputs and outputs of subworkflows, for example.

I read about 2/3rds of this thread before my brain started to dribble out of my ears, but I don't think we're far away from agreeing on records?

Really processes should be functions that take arguments with an

arg=value

format, where you /map/ your record fields directly onto the input arguments of a process, and you don't permanently modify a record by renaming fields.

This seems like a technical thing that we may be able to work towards on the Nextflow side if folks really want it?

it might be better if we just had a target NF version where we said, the old stuff is out and the new stuff is in.

We did exactly that, it's 26.10 - I thought that came across but maybe not so clear. But yes since the start of 2025 we've been saying that the final language features will be in by 26.10

Having tuples will take at least as much work to reorder files in them + you can't have any other files in the tuple inputs

I really don't see the benefit of half-migrations, or tuples

Yeah and I don't actually want standardized names anymore tbh, I want the names to make it super clear what the file is actually doing in the process

For me that's a huuuuge boost for nf-core functionality

Also maybe I missed it, but don't people want to run modules directly?? nextflow module run

I've never seen the utility really, cool idea - but I might as well just run the tool stadnalone

no not at all

The wrapping helps. https://nfcore.slack.com/archives/CPHN84NQ4/p1791203816532119?thread_ts=1791198763.914719&cid=CPHN84NQ4

but.. why?

CLI tools can be used for many things but we don't complain about having named CLI flags?

I mean that's not quite the point right, it's more like what if every named CLI flag needed to be pointed at a file with the same name?

Also maybe I missed it, but don't people want to run modules directly??

nextflow module run

I really do!

but.. why?

We're agreeing. People do want

nextflow module run

So now that we have reach an agreement, we may need to prolong our hackahton group during the weekend to have some stuff done 😆

Summary: We're adopting records, and imposing a few other standards: https://nfcore.slack.com/archives/C07ACNLELR2/p1792145775463469

YEAAAH

Working on it 😅

You're good

FWIW

nextflow module run

is orthogonal to the record/tuple question. You get the same functionality either way

You summarised exactly what we ended up concluding in person a couple of hours ago 😎

Good. Fear not, for I (and

nextflow lint

) will be with you 😄