Hi <!here>, I just added the last small changes to the document explaining the new guidelines. Could some of you proofread this? hackmd.io/YmVPL55sTomtFIpNKlRKBA. (Thank you all for the input and help on the document 🎉)

There are some small bugs like

meta.id

still being there. Going through and fixing

Ah i thought I replaced them all, my bad 😅

Looks good to me. Very excited for all these changes!

Thanks Jim 🤣

What to do with 5 + 6? Ideally we want proper naming, but what to do with this?

The common consensus was that this would be a problem to solve for another time 😅

should work. in fact I think you could (1) load the samplesheet records without creating a meta map and (2) set

sample += [meta: sample]

(set the meta record to the sample record itself) since meta is typed as

Record

in the process, nextflow won't try to stage any files in it. it's a bit silly but it works 🤷

That's good to know! Thanks for confirming 😛

What collection type should multiple paths be?

List<Path>

,

Set<Path>

, or

Bag<Path>

etc?

any of those are valid. just depends on whether you care about ordering. Set vs Bag are usually interchangeable for files

Thanks. So then we say

List<Path>

or

Bag<Path>

for convenience/reference

Does the order of files impact the resumability of the process?

I would assume yes

“The optional meta record can be used to supply sample-level metadata beyond the identifier ‘id’. It must always be emitted in the output record.”

Feel free to change the wording if you know a better way to explain it 😅 (english is not my first language 😛)

It's mine, but I still suck at it

That made me feel a bit better 🤣

Does the order of files impact the resumability of the process?

if you use a List then yes, for Bag and Set no

Okay thanks!

If we define an input field as

Set

but the input record has a

List

, is there an implicit type conversion ? And is caching done on the list or the set ? (thinking about the resumability)

• you can convert List/Bag to Set with

toSet()

• you can convert Set/Bag to List with

toSorted()

Are we forcing just 1 input record? I know that's good practice but it feels very strict and the occasional record + value may be required.

Yes, I believe so

Yes this will also make the module way more flexible at almost zero cost. It will need a bit of a change in people's mindset on writing pipelines as everything will become sample-based (but that's a good thing IMO)

Are we allowed to use named records in the inputs?

input:
MySuperSpecialRecord // instead of record()

We decided against it because of naming collisions with records

My feeling regarding collection types is that we should stick with Lists for the most part, as I would expect most tools have order-dependent outputs that users might want to control

We decided against it because of naming collisions with records

How so? Because SampleRecord may clash? Note unless you import it, a record is local to the file so I don't know how likely this is

How are

env

and

stdout

types covered?

typed process has

env()

and

stdout()

in the output block

Ah I must have missremembered that records could only occur once in a pipeline. What advantages would having named records have for you @Adam Talbot?

Looks much cleaner, less record boiler plate in the process. Named inputs, so it's easier to know what you're referring to (although if you force

input.size == 1

that's less of a concern).

// private record type
record MySampleRecord {
    id: String
    // stuff
}

process USES_MYSAMPLERECORD {
    input:
        samples: MySampleRecord

    script:
    """
    echo 'Hello ${samples.id}!'
    """
}

the common consensus seemed to be that an anonymous record in the input block was cleaner and easier to read than a record declaration somewhere else in the process. another downside is the input variable names are longer (

samples.id

in your case,

id

if you used anonymous records)

I hate embedding meta data in IDs

I have a question actually, because this so far only touches on typed processes but not subworkflows. Can you type subworkflows anonymously with records, or do we need to make named types?

Can you type subworkflows anonymously with records, or do we need to make named types?

subworkflows should use record types

Not read the chat yet but while reading through: > (this includes the id field or any other required meta fields like single_end if needed) > I would use this chance to drop all meta fields except

id

. As in `id`is the only standard/hardcoded/required one. Everything else is up to pipeline dev. Everything to control the module should come via as an 'input'.

subworkflows should use record types

Yes, but do they need to take a named type in order to enforce contents, or can we pass an anonymous function like we do with process inputs?

I would use this chance to drop all meta fields

that's exactly what we are doing, even

id

is technically no longer required in meta and it's become fully optional

I would use this chance to drop all meta fields

that's exactly what we are doing, even id is technically no longer required in meta and it's become fully optional Right. But the first part of my message was a quote from the text. Can I remove 'any other required meta fields like single_end' then?

yes! they should be required input values as they are required for the process to run

This bit then is also not valid right? Essentially meta is just for passing through the module (it's only used for pipeline logic, essentially)? Everything for module logic should come via top level inputs, Correct?

Good point, that must have been a leftover from a previous version. i'll update it

> How else would you do this in modules without breaking the flexibility?

record SampleInfo {
    id: String
    treatment: Treatment
    replicate: Int?
}

process DO_STUFF {
    input:
    record(
        meta: SampleInfo,
        bam: File,
        bai: File
    )

    script:
    def prefix = ext.prefix ?: meta.findAll { it?.trim() }.join('_')
    // etc
}

Yeah... 😅

Yes, but do they need to take a named type in order to enforce contents, or can we pass an anonymous function like we do with process inputs?

you can pass an anonymous record. the type checker will verify that it satisfies the record type

Right, that's also a new guideline that slipped my mind. i'll rephrase

that also seems to be the only remaining purpose of the meta map, to allow for runtime override via

ext

Yes! And also to help pipeline developers with more complex joins

It's also for passing metadata through a pipeline, without having to do a separate -> process -> join loop at every step

It's also for passing metadata through a pipeline, without having to do a separate -> process -> join loop at every step

Which could also be avoided if something like this was implemented in Nextflow, which would remove the need for

meta

entirely https://nfcore.slack.com/archives/C07ACNLELR2/p1792161665296959

I missed this mega-thread in my catchup yesterday! FWIW @Adam Talbot my initial prototype had all the named record type setup (I still find that nicer), but we stripped it in favour of anonymous inline definitions so as not to scare the horses more than we needed.

Thank you! But it wasn't a one-man show. Thanks to all of you for participating and sharing opinions on this subject. Our approach wouldn't have been nearly as good otherwise 😛

am I allowed to use named records inside the input record at least?

input:
record(
    id: String,
    thingy: Thingy, // special thingy type
    bam: Path,
    bai: Path?
)

In what case would you use that? The new guidelines state that all required values should in the top-level of the record

One of the advantages of records is the ability to model your data properly. Flattening it every time removes a big part of that advantage. Here's a more obvious example:

input:
record(
    id: String,
    thingy: List[Thingy], // list of thingy
    bam: Path,
    bai: Path?
)

Where thingy might have multiple useful things. If we flatten, we get:

input:
record(
    id: String,
    part_1_of_thingy: List[String],
    part_2_of_thingy: List[Int],
    part_3_of_thingy: List[Path],    // Hope these guys are ordered correctly!
    bam: Path,
    bai: Path?
)

Yes, but how would we ever agree on such a structure in nf-core? 😅

a realistic example might be:

record Vcf {
    id: String,
    vcf: Path,
    tbi: Path?
}

...
process COMBINE_VCFS {

    input:
    record(
        id: String,
        vcfs: List[Vcf]
    )
}

I agree that's a nice approach, and it works well if you have tight integration between the pipeline and the modules. But nf-core modules need to be quite generic, which decouples them from the pipeline context. So such data structures can only really be the domain of a pipeline. And it adds questions about how and where Vcf is defined, does it need to be imported if it's shared among modules, or are there as many Vcf implementations as there are modules that use it?

where Vcf is defined

In the same file, above the process

does it need to be imported

No

if it's shared among modules

No, they're just compatibility checks. They're closer to interfaces not types.

are there as many Vcf implementations as there are modules that use it

yep. that's fine. doesn't impact anything really.

doesn't impact anything really

That's fine until for one reason or another a particular module uses different field names, or extra field names, etc., right? And then what seemed clean is a bit more complicated and less predictable - you need to read the manual every time

File 1:

record Vcf {
    id: String,
    vcf: Path,
    tbi: Path?
}

...
process COMBINE_VCFS {

    input:
    record(
        id: String,
        vcfs: List[Vcf]
    )
}

File 2:

record CompletelyDifferentVcfType {
    id: String,
    vcf: Path,
    tbi: Path?
}

...
process OTHER_VCF_PROCESS {

    input:
    record(
        id: String,
        vcf: CompletelyDifferentVcfType
    )
}

workflow:

include { COMBINE_VCFS      } from '<http://combine_vcfs.nf|combine_vcfs.nf>'
include { OTHER_VCF_PROCESS } from '<http://other_vcf_process.nf|other_vcf_process.nf>' // No type imports!

workflow {
   COMBINE_VCFS( ch_vcf.collect() ) // I know this doesn't work but I'm being quick!
   OTHER_VCF_PROCESS( ch_vcf )
}

I mean extra required fields

If one can settle on the record names, it means we can also settle @James Fellows Yates variable naming issue too

Also, does the above actually work? It's not clear from what Ben said here if the files in

Vcf

would be staged?

yes. Ben is referring to a bare record where nothing is declared as a

Path

when it comes to file staging: • nested

Record

is ignored because nextflow can't see the types in it • nested record type like

Vcf

will have its path fields staged (

Path

and

Path

collections) • a collection of record types like

List<Vcf>

is not staged automatically. you need an explicit stageAs:

input:
record(
    id: String,
    vcfs: List<Vcf>
)

stage:
stageAs vcfs.collect { v -> v.vcf }, '*'
stageAs vcfs.collect { v -> v.tbi }.findAll { tbi -> tbi != null }, '*'

example process? github.com/nf-core/modules/pull/10336/changes

that PR was merged, so you will be able to do paths / path collections / records / record collections and nest these to your heart's desire. nextflow will stage every declared Path it finds from these