I have a question regarding the use of

meta

: Am I allowed to stuff whatever into meta in my pipeline, or are there guidelines that prohibit me from doing that? The background is that I am currently doing quite some

join

s to recover information stored in the channel, which is lost when I prepare the inputs for a process, but if I could store all of that valuable information in meta (e.g.

meta.sample_data

), I could avoid joins, and thus blocking?

Yes that's exactly what the meta map is for! The only reserved keys are

id

and

single_end

, all other keys can be completely custom

In your pipeline, anything you like. In nf-core/modules, we like to restrict it to only

meta.id

and

meta.single_end

, to not impose restrictions on use

Ok, that is kind of what I assumed and that will allow me to remove a bunch of ugly crap from my pipeline

yah, but in your pipeline it is exactly what to use it for

Here's one of my horrendous meta objects. You really can put anything in them!

[id:baUndUnlc1.pri, assembler:hifiasm, long_read_dataset:baUndUnlc1, long_read_platform:pacbio_hifi, long_read_1n_coverage:[], illumina_hic_dataset:baUndUnlc1, illumina_10x_dataset:[], maternal_illumina_dataset:[], paternal_illumina_dataset:[], phased_assembly:false, trio_assembly:false, purge:true, polish:false, scaffold:true, assemble_mito:true, assemble_plastid:true, hifiasm_arguments:--primary -f0, purging_cutoffs:[], purge_middle:false, yahs_arguments:, busco_lineage:auto_euk, mitohifi_reference_species:Caradrina clavipalpis, mitohifi_mito_genetic_code:5, mitohifi_plastid_genetic_code:11, mitohifi_reads_arguments:, oatk_kmer_size:1000, oatk_coverage_cutoff:[], oatk_arguments:, oatk_mito_hmm:[], oatk_plastid_hmm:[], coverage:59.68]

Nice, I am striving to create something equally, potentially even more, horrendous

Nice! I am very excited for record types to potentially clean a lot of this up

This is so much better than doing joins all the time to carry along sample-data, although I agree that the meta map becomes a bit disgusting

just remember that by default joining on the meta map will only join if all the elements are the same.

I no longer need to join on the meta map

Without knowing more, I can't say for sure what the error is, but I do recall that we should not use

.clone()

as it's not guaranteed to return a deep copy?

I also do not really know what is causing the error, the logs do not really reveal where that happens.. Since i went and refactored the whole pipeline in one go, i am only observing this now after i edited 10 swfs

You can always do the thing where you comment all the pipeline out except the first step, run it, etc.. and step through manually to find the error

Yeah, but not on a Friday afternoon 😉

Indeed not...

I assume it is related to constructs like this:

.flatMap { it -> // it looks like [meta, output_path]
            it[0].metas
                  .collect { meta -> [ meta: meta + [hifireads: it[1]] ] }
        }

potentially it is just dumb to use

meta

as key and value

Do we agree that once the Record structure (as introduced in last November by Ben Sherman) is available, it should replace little by little the meta maps?

I think that's kind of the idea, yes. I don't think stuffing everything into meta is great, it seems like a crutch to do something that should be handled properly, but it is the only way i currently see to carry sample information through processes.

Absolutely, there's no other way for now

For completeness: the

ConcurrentModificationException

I complained about was caused by

dump

I'd say single_end is only useful if you actually have that kind of data. For me the only one we share across all nf-core is id

I think that's kind of the idea, yes. I don't think stuffing everything into meta is great, it seems like a crutch to do something that should be handled properly, but it is the only way i currently see to carry sample information through processes.

From what I've seen so far of the Record type proposals in the 26.04 beta, I think we're still going to have to have

meta

objects inside our nf-core modules 🙈

Well, at least that means if I refactor now towards stuffing everything into meta, I may not have to refactor again until.. october?

8 months of stability!