https://nf-co.re logo
Join Slack
Powered by
I'm working on a first implementation of the `samp...
# nf-schema
n
I'm working on a first implementation of the 
samplesheetToRecords
function which will convert a samplesheet to a list of Record types. However while doing this I'm also finding myself at the point where I need to decide how we want to use record types (especially with the meta map). So my question is: What would you prefer? (please react with the emoji) Option 1️⃣ : All values will be available as a key-value pair in the Record, meaning that meta field and input field will be on the same level. (TBH this is my preferred way of working with records as this allow directly joining on specific meta keys using the 
join
operator without having to do any preprocessing of the channel structure). Example:
Copy code
[[id:'test', single_end:false], test_R1.fastq.gz, test_R2.fastq.gz]
Would become:
Copy code
record(id: 'test', single_end:false, fastq_1:test_R1.fastq.gz, fastq_2:test_R2.fastq.gz)
Option 2️⃣ : We keep a 
meta
key in the record that contains all meta values. (This would keep our channel structures relatively the same as before) Example:
Copy code
[[id:'test', single_end:false], test_R1.fastq.gz, test_R2.fastq.gz]
Would become:
Copy code
record(meta:[id:'test', single_end:false], fastq_1:test_R1.fastq.gz, fastq_2:test_R2.fastq.gz)
Option 3️⃣ : If you have a better idea please post it in this thread!
2️⃣ 5
3️⃣ 1
1️⃣ 14
🍿 2
<!subteam^S044DU0NG68> <!subteam^S016E36N7TQ> sorry for the ping but I think this will have quite an impact on how we will define best practices for record types in the future so your opinions are greatly appreciated 🥺
f
Would 1️⃣ also support other custom meta fields?
s
The issue is what happens when they collide with the input
j
What are the cons of option 2?
n
Would 1️⃣ also support other custom meta fields?
Could you elaborate a bit more what you mean here?
s
1️⃣ looks really nice, and has some good properties. But you can't use 
bed
, 
bam
, 
index
, if you would collide with anything
in the "meta" map
(or the record keys, or something, IDK)
n
The issue is what happens when they collide with the input
Field names in samplesheets should be unique so there should never be any overlap between field names
We would need to create a set of standard names for option 1️⃣ to work but that would be a nice thing IMO
😱 3
f
Hm ok because I use a lot of custom meta map fields that would then be more complicated I guess?
a
Option 1 also doubles as an interface, i.e. a contract that says "this process will accept the input if it includes fields x, y and z".
party this 2
n
You need to explicitely do a 
.join
to merge channels with record types so you are in control of when an overlap happens. It's up to the pipeline developer to decide how to handle that IMO
Option 1 also doubles as an interface, i.e. a contract that says "this process will accept the input if it has fields x, y and z"
Yes this would basically remove the distinction between meta field and value inputs of a process
🙌 1
j
Without having thought extra hard about it, I feel a bit like 
meta
was a hack we were forced into because 1 wasn't possible. Would be nice to kill it.
party this 5
a
The answer is that 
samplesheetToRecords
should make flat record and that then the pipeline developers can choose to group them in a 
map
of the channel if the so choose (effectively convert one record type to another)
n
Yes 
samplesheetToRecords
will just create a standard record, it's up to the pipeline developer to cast to the correct record type.
m
I think I prefer 1️⃣ , since with the 
meta
key you essentially remove the whole idea of typing your input by creating a dumping ground for whatever you want in there
2️⃣ forces devs to give their structs a little more thought
a
There are just too many issues with flat records for my liking - unless anything has changed recently, you couldn't access arbitrary (not-specified in the process input) fields in eg 
ext.args
closures in config
n
You can access it if you use a record type as input.
even fields that are not defined by that type
BTW I have a draft PR here for a full static types conversion of a pipeline that also fully uses record types. I based my opinions on the things I discovered when developing that PR: github.com/nf-cmgg/wisecondorx/pull/25
Another bonus of option 1️⃣ is that this would make the implementation a lot easier and less error prone on the nf-schema side
j
On my phone so can't really look at PR, but how does grouping on a subset of records work? E.g. I semi commonly use a pattern (and I know other pipelines do it) of pushing a couple of meta elements into a temporary secondary 'key' meta map, operate on that, and push then back after
(or rather remove the temp key)
n
You can do something like this if both channels are of type `Channel<Record>`: 
ch1.join(ch2, by:id)
ATM the 
by
option only takes one value, but I think we could make a suggestion for it to also allow a list of keys
j
I think that would be important
n
The records in the first channel will take precedence if there is any overlap
👍 1
j
Otherwise flattening would be too destructive
n
Yes, I'll open an issue on the Nextflow repo for this!
j
But also I'm very scared by 'we would need to create a standard set of names'
100 rainbow 2
Because
Names and definitions are going to be extremely hard to get a consensus
n
It's not entirely necessary but that would reduce the amount of name remapping we have to do
i
Merge with closure would be nice
🙌 1
n
Some fields are easy (like 
fastq_1
and 
fasq_2
) but indeed stuff like bam, cram, sam will need a common name which will be a bit more tricky
j
Ok what extent do we need to go with those? What you have described is the curse of samtools
🤣 1
What about other areas which are not super standarised
n
I think we can let this grow a bit organically and intervene if we see too many different names for the same files?
j
Then we will have @Simon Pearce raging about having a second test-datasets 🤣🤣🤣
To be clear I like the idea but if you want to go that route we are going to need a huge load of extra infrastructure
(imo)
i
Some fields are easy (like 
fastq_1
and 
fasq_2
) but indeed stuff like bam, cram, sam will need a common name which will be a bit more tricky
That is a discussion independent of records, they are porting Sarek subworkflows to the modules repo and it's already coming up
🫠 1
🤣 2
n
m
sarek, of course
shakefist 1
🤣 1
a
Not sure if I understand you correctly James, but I think you mean you nest maps within the meta map? In which case, you could 'just' flatten the first layer;
Copy code
sample_info = record(foo: "x", bar: "y", baz: "z")

sample_record = record(id: "test", sample_info: sample_info, fastq_1:test_R1.fastq.gz, fastq_2:test_R2.fastq.gz)
n
Yes record types are going to be a huge change indeed, hence why I wanted to start this discussion (thanks all for responding). And I'm not sure this will need a huge load of extra infrastructure, just some list with agreed upon names for specific file types?
a
and then join on 
id
& 
sample_info
i
@Nicolas Vannieuwkerke what about join with closure? e.g. 
ch_samples.join(ch_genomes, by: { sample, genome -> sample.genome == genome.id })
j
Yes record types are going to be a huge change indeed, hence why I wanted to start this discussion (thanks all for responding). And I'm not sure this will need a huge load of extra infrastructure, just some list with agreed upon names for specific file types?
Infrastructure I mean community infrastructure for agreeing to this. You say a list but would we need linting? Who decides on the definition? What if someone wants to use another name for the same thing etc.?
@Adam Talbot no not nesting. But if I understand your example that may still work I think (still not that familiar with records yet)
m
But aren't supposed to be able to update things one bit at a time? like modules first, then subworkflows, then pipelines, then meta pipelines, then the world?
maxime 1
n
what about join with closure? e.g. 
ch_samples.join(ch_genomes, by: { sample, genome -> sample.genome == genome.id })
Yeah feel free to comment on my issue with it, but I don't think this will be entirely necessary since you most likely will only join channels originating from the same channel
a
Sorry I'm confused...why do we need a strict list for inputs record keys?
n
Infrastructure I mean community infrastructure for agreeing to this. You say a list but would we need linting? Who decides on the definition? What if someone wants to use another name for the same thing etc.?
We could keep this quite flexible though, I'm not set on being super strict on this. It would just be nice to have some common names for files that often are used (so we don't need to rename all the time)
a
Some fields are easy (like 
fastq_1
and 
fasq_2
)
As someone who worked on libraries with 3-5 FASTQ files this was a nightmare
🤣 3
n
But aren't supposed to be able to update things one bit at a time
Yeah but it's all connected, if you use a record type input for a process you also need to update your workflow to work with it
a
(well a mild inconvenience)
😁 1
n
That's something we can figure out once we start updating those modules 😁
a
Personal feeling: start with modules, see how similar the inputs are. See what's consistent and should become a rule and what should be guidance.
party this 3
a
Sorry I'm confused...why do we need a strict list for inputs record keys?
You don't need one but for my money nf-core modules ceases to be that useful if I'm having to wrangle my records between each one so that I have the set of required keys. The current input situation has the advantage that the processes don't care what you refer to the items as outside of it (except keys within the meta map - where we have a very strict policy on hardcoded value use)
this 1
party this 2
j
This is why I've personally resisted custom meta fields in all modules etc
💯 1
I'm on Team @Arthur so far
a
Right now there are so many different names used for channels which are effectively VCF indexes for example: 
vcf_index
, 
index
, 
tbi
, 
csi
and I'm sure some others
👍 2
n
Yeah but in the current system you often have to do a 
.map
to set the right structure which is kinda the same as having to rename fields
j
True but I think this would add a LOT more
Particularly if modules can choose whatever they want
My philosphy remains nf-core modules should work for pipeline developers, not the other waya round
a
Oh I see, we need to bring the hammer down on naming consistency...but also understand that different tools will need different names for the same thing. Tricky balance.
this 1
i
That will be a huge pain no matter what we do
m
how about, 
var
, 
var2
, etc. worked for 
meta
😜
😬 1
this 2
🙅 2
🤣 2
🙈 1
i
shared a GIF
j
Then we need the system for naming things with definitions etc ;)
It's between wildwest test-datasets and clinically abstract meta, meta2, meta3 stuff!
🤠 🤖
a
I'm starting to think that the issue here is that records as inputs are a bad design. They violate the idea the the process internals don't need to know about the externals
i
I feel like we've completely changed the question at this point
🫠 1
f
> I feel like we've completely changed the question at this point Classic nf-core 👌
💯 4
They violate the idea the the process internals don't need to know about the externals
@Arthur Curious why you think that this is a feature / requirement
a
I'm not following that either?
a
Well in eg python, r, etc the positional or named arguments of a function do not rely on the names of the variables being passed in, and that is generally a good thing. What we've basically concocted here is functions taking dictionaries with arbitrarily required keys - something which is typically a bad design. Which is to say the idea of named inputs is nice; but record type inputs arent named inputs, they're effectively unnamed dictionaries with required keys
party this 2
👍 1
f
Ahh ok I got your concern. I can see why that would not be a great design. I guess from my understanding record types should just ensure that the actual TYPE of the file is known / passed correctly...
But yes, the way we are describing it here, we would force the actual key naming to be uniform.
j
To make sure I am also on the same lines Arthur: if a module says it calls a record field by name, e.g. 
input.fastq_1
, then the entire pipeline must also use 
input.fastq_1
throughout, if it doesn't want to have to rename record fields each time it passes a record to a new module?
confused dog 1
Maybe j should not try and follow these discussions on my ohone while travelling, sorry
😁 2
🧳 1
n
Also pinging @Ben Sherman here 👀
f
shared a GIF
🤣 4
n
But yeah the discussion has completely diverted from the initial question, I think everyone agrees that it's better to drop the meta map here. (Please shout at me angrily if assumed wrong here 😉)
j
Uh not necessarily
I think Arthur raised a point about it
n
you mean the 
ext.args
thing?
s
I don't necessarily agree either
j
Possibly, scroll is failing ATM 😅
f
I don't necessarily agree either
Simon not agreeing really threw me for a loop 😉 😄
😆 3
shared a GIF
s
What happens if your (currently) meta variables are also inputs or outputs of the module code?
shared a GIF
❤️ 1
f
shared a GIF
s
Now maybe that won't happen as much as I was thinking, because I guess schema inputs that are file paths get treated separately
f
What happens if your (currently) meta variables are also inputs or outputs of the module code?
Why would this be potentially a problem? It's just other fields now pointing to files isn't it?
s
Say your process currently takes 
tuple (meta, file)
and outputs 
tuple (meta, foo)
If we had 
meta.foo
on the input record, then I think foo gets overwritten
n
What happens if your (currently) meta variables are also inputs or outputs of the module code?
Can you give an example where this is the problem? I can't remember a single moment where my meta fields and process input names overlapped and where it would cause a problem
f
@Simon Pearce that generally sounds like bad variable naming to me if you have a meta input value with the same name as an output channel and the only way you are distinguishing them is with 
meta.
to be fair, you can still do 
meta_foo
as the record type. I guess this brings a good question though. Do most people like the fact that having 
meta.
identifies meta values as metadata, in which case, keeping it more for naming sake than functionality might make sense...
party this 2
n
And so we return to the discussion about naming best practices 😉
😬 1
j
I like the metadata concept personally
I know it sucks @Nicolas Vannieuwkerke but I think it's a key point here which influences the implementation...
n
You could also still add a record in a record that consists of metadata
Yeah I'm just joking @James Fellows Yates , no worries 😉
j
I think that would be a softer landing for a lot of people with this concept
(just checking because we are up to 111 comments in this thread now ;))
😁 1
nf core deal with it 1
n
On the other hand, if we plan on ever dropping the meta map, the record migration might be the best time for it
f
I actually have to say that I also like the meta convention because it makes it easy to know what's metadata and what's actual files etc
this 1
n
I kinda agree and kinda don't. Since we allow value inputs to processes the line between metadata and process inputs has blurred a bit
s
Can you give an example where this is the problem? I can't remember a single moment where my meta fields and process input names overlapped and where it would cause a problem
I don't have an example to hand. And there might not be one
f
I actually think this is totally a vibe question at this point 😅 . We need @Ben Sherman to tell us whether there are actual functional implications we haven't thought of I think!
Ben's out of office though, so that will have to wait till next week!
n
Okay I'll put this on hold until next week then 😛 This was very interesting though thanks all for the pointers
a
What we've basically concocted here is functions taking dictionaries with arbitrarily required keys - something which is typically a bad design.
As opposed to an array of tuples only accessed by index 😉
party this 1
n
I agree that it's not 100% ideal but it's quite an improvement over the tuple structure we currently have
m
Can you give an example where this is the problem? I can’t remember a single moment where my meta fields and process input names overlapped and where it would cause a problem
Yeah I can 😓 Check our own preproc pipeline, where there’s 
fasta
in the meta (so we can have a genome per sample) and then the fasta inputs for the modules
n
but those would be the same file so not a problem?
this would remove that duplication
m
also true
🥳 1
n
I would prefer to have a map that I can use for all of my pipeline related structure (i.e. 
meta
) and not enforce keys on any other fields. E.g. What if my pipeline takes two different types of short reads (e.g. genomic, HiC), I cannot put both of them into 
fastq_1,2
? There is also the situation where I might store stuff in meta, that is not used as an input, but comes in via 
ext.args
through config, how can this be done with scheme 1?
n
The record types allow other keys to be given to it. You can still reference those keys in the 
ext.args
block. having two types of fastqs is something completely different and would return us back to the naming discussion 😁
n
Ok, so if there is a module that expects certain keys as inputs additional keys present in the record will just be passed through ?
n
This is definitely a possibility by merging the input record with the output record (either in the process or using the join operator)
We still need to figure out what's the best option for that
n
Joining after will block though, unless records also solve that problem
n
Joining after would be done on an ID given to the input record and passed to the output record so nothing will block that join
a
I honestly kind of think we should consider just keeping 
meta
as a nested record. A more specific record type will always match a generic one I think, so pipelines can effectively just then define their own meta record type with any of their required keys there. The process itself obviously wouldn’t be doing any type-checking of it, but presumably the typing of the channel itself would effectively gate it anyway? And then we can declare 
meta
as a reserved key within nf-core, which is way simpler than worrying about arbitrary name clashes
Can still promote 
id
to a first-class record field too
n
I kinda agree that keeping the current system would be easier to migrate. On the other hand using record types to the fullest will make our pipeline development significantly easier. I think we all just need to play around with the record type system a bit more and have a discussion in some upcoming meeting about what everyone thinks
j
Lots of little reprex examples of pros and cons of each approach might be useful
🙌 1
n
I'll try and whip some up inbetween my other work 😛
❤️ 1
s
On the other hand using record types to the fullest will make our pipeline development significantly easier.
I remain to be convinced by this, so some examples would be helpful
party this 1
a
I think Simon also hits on a really key point here - lots of the community will simply not adopt any of this until nf-core does. And then once we adopt it, the standard will likely become the prevalent use form. So I really think there is a benefit in being a) sceptical and b) thorough before launching with a very clear standard/spec
party this 2
n
I completely agree! This will definitely not be a decision taken lightly.
s
I generally feel sceptical on most of the recent changes to the language, such as the strict types as well, that it just makes the language less intelligible at the benefit of being a more "pure" language
n
How does it make the language less intelligible to you? I actually found it to be way more clear with strict types
s
eh, it is my feels
I don't claim to be rational on this point
and I stopped bothering with trying it as it was just frustrating
a
I am pretty keen for some of the updates, but I would say that I have found the transition period to be a real pain. I know nobody wants a python 2 --> 3 style debacle, but the slow-trickle of related-but-technically-standalone features has led to me mostly keeping pipelines on 25.10 until everything I want is in a stable version
100 rainbow 1
i
Most of the changes are useful, even though some of them are annoying. Types will be the biggest pin to migrate to, but they should finally make channel shapes more manageable (and hopefully finally give us more useful errors), so I am optimistic.
a
My best real-world example would probably be the workflow outputs, which with deep nesting of subworkflows and modules would be a real pain to enable selective publishing of any arbitrary process until you can combine record types (all related files in a single nice named-field-access object) and topic channels, so that data can propagate up without needing to wire up every single process output to bubble up through layers of nesting
🙌 2
s
Yes, workflow outputs are great, until you try and use them in exactly that manner
and for some reason we can't just get some way to pass everything through to the next level
n
Yeah really looking forward to seeing that in action, I agree with you that the system definitely is not perfect yet (which they aren't claiming too), but it will get better if we vent our frustrations to the team IMO
s
Eh, doesn't feel like we get listened too
n
Yes and no for that in my experience. all problems get properly resolved if I make proper issues or communicate directly on the beta channels. We also gotta keep in mind that so many people ask things from them that they probably can't implement all requests in time? But I get your frustrations
b
Nico is right, it's all connected. Even though the original question is about mapping the samplesheet to records, we have to think about how those records are used throughout the pipeline With tuples and records you have to pick your poison: either you must align channels / process inputs by tuple index, or you must align them by record field name. There will be glue logic either way But records are ultimately better because they are more flexible -- a record input can accept additional fields and pass them through, a tuple cannot. This one difference eliminates a lot of the branch / multiMap glue logic that is currently required with tuples Records also allow you to model your data in a sane manner versus tuples, which as a benefit is hard to quantify but I think it's clearly an improvement
🎉 3
As for the meta-map question, I think nf-core should ultimately use whichever approach they think is best. Option 1 is the ideal that I am striving for, and i will continue to try to make it the obvious best choice. But the meta-record also seems to have some practical benefits, especially for nf-core modules where generality and reusability are important Robert used a meta-record for bactopia (example: prokka), I think because he also needs to maintain many modules in a generic way
I finally had some time to explore this issue further and made a PR with my current proposal: github.com/nf-core/methylseq/pull/625 It builds on my methylseq static types migration as a separate PR so we can focus on this issue The tl;dr is that ext and meta maps are interconnected: to remove one you must remove the other In order to achieve the ideal situation where each samplesheet row is just a record without a nested meta-map, I had to replace ext config with process inputs and params. A single 
args
param provides per-pipeline overrides for tool args. Per-sample args overrides are supplied as samplesheet columns See the PR description for more details. Feel free to comment there with your feedback I don't need you to trust me, but I do need you to approach this issue with an open mind. The current patterns are shaped by the old constraints of DSL2. With static typing those constraints have been lifted and new approaches are possible. But we have to be willing to explore them fully, not just speculate
n
Does this require users to actually create the args map in the samplesheet, or how would this interface work?
m
I just asked the same question on the PR. How do dynamic args work here on an actual sample level, e.g. using filename in prefix or the id in the args, etc
👀 1
a
I still come back to the feeling that this makes nextflow inherently less "powerful" for advanced users. Maybe that ultimately doesn't matter, and with AI writing boilerplate for arbitrary args you want to wire up will be somewhat simple, but I think that this removal of 
ext
means an off-the-shelf pipeline is likely to be less flexible or require a much greater amount of boilerplate?
👆 3
m
My first take on this is that you’re proposing nf-core adopt a base record structure for all modules with: 
id
, 
prefix
, 
suffix
, and 
args
. So in other words, explicitly moving all the accepted meta keys from the nf-core guidelines into explicit universal record attributes
n
Echoing arthur, I fell that this makes it more opaque for users, because, as I understand the user would need to know which name the pipeline dev uses internally to refer to one of the possibly many subworkflow-nested samtools_sort process specific argnames, whereas the withName selector is more obvious? I am also not a fan of increasing boilerplate / making things more opaque justified with "AI will do it"; AI is not free
100 rainbow 3
I have to admit though that I actually do not follow why records and 
ext
are not an option, is this incompatibility ideological, to achieve the perfectly shaped input row, or is it a language limitation? I guess what I am asking, would something like: 
def args = args && task.ext.args ? args.join(task.ext.args) : args ?: task.ext.args ?: ''
be an option?
i
I think we're all fine with losing 
meta
, since it was a hack anyway, but I don't think we want to lose 
ext
with all its options. Being able to "break open" a pipeline when needed is one of the most powerful aspects of our setup and it would be a shame to get rid of it. For the more advanced user, acting directly on the specific module is the natural thing to do and it eliminates all the layers of complexity in between.
👍🏻 1
➕ 4
p
Emphasis on 
user
, and not 
developer
this 5
j
I left a big comment on the PR, but I wanted to mention here too - I am very much in favour of moving args/prefix/etc. into the module inputs, as I think that's much more declarative (set everything in one place and invoke the process! - you don't have to read multiple files to understand what's happening). I think that's a huge win for readability, as well as maintainability of pipelines. But I do agree that we can and should maintain a 
task.ext.args
override for power users!
n
I agree with your point re. process as functions. A pipeline is just a bunch of transformations of the input samplesheet
a
Did we decide where to put the meta map in the end? 😛
😂 4
j
As proposed, I don't see any way to easily ensure that all metadata that goes /in/ to a process comes out the other end in the emitted record. Fields will be silently dropped. So I think we need the meta map. (but for now we should stick with tuples!)
a
errr I think that's straightforward...hold please...
Fields will be silently dropped. So I think we need the meta map.
Confirmed, this isn't true
extra values will be passed through if you use a 
meta
item: gist.github.com/adamrtalbot/…
Oh is that what you meant facepalm
j
Perhaps I'm missing how that's the case, but in the example PR modules always output a brand-new record:
Copy code
input:
    record(
        id: String,
        reads: List<Path>,
        bwa_index: Path,
        fasta: Path?,
        args: String?,
        args2: String?,
        prefix: String?
    )
    sort_bam: Boolean

    output:
    record(
        id   : id,
        bam  : file("*.bam", optional: true),
        cram : file("*.cram", optional: true),
        sam  : file("*.sam", optional: true),
        csi  : file("*.csi", optional: true),
        crai : file("*.crai", optional: true)
    )
If the input record has some other field, a piece of important and identifying sample metadata, it won't be included in the output.
Yes
a
you can but it's manual: gist.github.com/adamrtalbot/… Which I think answers our question, we do 2️⃣
party this 1
j
You could do:
Copy code
input:
mysample: record(
        id: String,
        reads: List<Path>,
        bwa_index: Path,
        fasta: Path?,
        args: String?,
        args2: String?,
        prefix: String?
    )

output:
mysample + record(...)
But that way lies ever-growing-sized records and name collisions. Really we only want to passage metadata.
a
yep, it works but this is way prettier:
Copy code
record Meta {
    id: String
    tumour: bool
    other: String
}

record Sample {
    meta: Meta
    bam: Path
}
Copy code
input:
sample: Sample

output:
record(meta: sample.meta, results: file('output.txt'))
❤️ 1
j
Exactly!
a
(fyi I just tested and you don't need to rename or coerce stuff, it's basically the same behaviour as before with the map)
But none of this helps 
task.ext.args
b
Thank you all for your feedback so far I think my core problem is not 
ext
in general but the fact that 
ext
is the only way to set things like args / prefix / suffix: • pipelines must spill their guts into config files in order to set pipeline-level defaults. this makes pipeline composition more difficult because module config doesn't travel with the process/workflow it's trying to configure • same for workflow modules, if an nf-core subworkflow has some module config, I have to remember to include that config when I use the module. and I have to deal with process selectors, etc • if I run a process/workflow directly via 
nextflow module run
, I can't set the tool args via param because it isn't declared as a process input. I have to prepare a config file before every command, again dealing with process selectors etc
Fortunately, @Jim Downie might have saved the day with this simple trick:
Copy code
args ?: task.ext.args ?: <default>
so we declare every ext setting as a process input but we still allow a runtime override from config. the process input means we can move the pipeline's ext config into pipeline code, which is all I really care about that also implies we have to keep 
meta
. it's the only way for a user to access metadata fields from config. for me this is suboptimal but not really a problem. you might be able to eliminate it by writing the process inputs/outputs differently but I don't think it's worthwhile so, here is take 2: github.com/nf-core/methylseq/pull/626
👀 1
so @Nicolas Vannieuwkerke hopefully that is finally approaching an answer to your original question 😅
🤣 1
n
I expected this to be quite a discussion but this surpassed my expectations 🤣 Still a bit confused since the conclusion here seems to be that we keep the meta map but the votes indicate that people prefer flat records 😅
Will we still be able to easily join on specific meta keys if we add a meta map?
b
A flat record without meta is still ideal IMO. And I think non nf-core pipelines will be able to do that. But nf-core's particular need for flexibility and reuse basically necessitates the use of meta maps You can join on specific keys with some extra 
map
operations. I'm also looking into joining on a closure predicate, but that won't be in 26.10
j
Joining on a closure would be an excellent feature
Also some control over the type of join - left, right, inner, etc..
🔥 1
n
allright if joining on a closure will be possible than I'm all for having a 
meta
map in the record 😛
party this 1
nf core party 1
9 Views
Open in Slack
PreviousNext
Powered by