https://nf-co.re logo
Join Slack
Powered by
Hi, I would like to use the vcf_gather_bcftools s...
# modules
l
Hi, I would like to use the vcf_gather_bcftools subworkflow for our phaseimpute pipeline. The issue I currently have, is that this subworkflow reassign the meta id and does not support the presence of other information in the meta map expect the id and the grouping key. I would like to make it more polyvalent, but this would induce some changes that my cause trouble in other pipeline using it. How should I approach this issue ? Code in following comment.
j
IMO a subworkflow should only ever add keys to a meta map (and my preference is to prefix the keys with an 
_
), not overwrite or remove them. It should be the responsibility of the subworkflow user to stage the input metas appropriately for joining/grouping.
party this 1
l
Here is the previous code:
Copy code
take:
    ch_vcfs // channel: [ meta, vcf, tbi ]
    ch_scatter_output // channel: [ meta, bed, gather_count ] => output from the scatter subworkflow, if you didn't use this subworkflow you can just use `[]` as bed since it isn't used
    val_common_meta // string:  The name of the meta field that should become the new id
    val_sort // boolean: Whether or not the output file should be sorted !! Add the config when using sort !!

    main:
    ch_concat_input = ch_vcfs
        .join(ch_scatter_output)
        .map { meta, vcf, tbi, _bed, gather_count ->
            meta = val_common_meta ? meta + [id: meta[val_common_meta]] : meta
            [groupKey(meta, gather_count), vcf, tbi]
        }
        .groupTuple()
        .map { meta, vcf, tbi -> [meta.target, vcf, tbi] }

    BCFTOOLS_CONCAT(ch_concat_input)
and her is my proposition
Copy code
take:
    ch_vcfs           // channel: [ meta, vcf, index, count ]
    val_common_meta   // string : The name of the meta field that should become the new id
    val_sort          // boolean: Whether or not the output file should be sorted !! Add the config when using sort !!

    main:
    ch_concat_input = ch_vcfs
        .map { meta, vcf, index, count ->
            newMeta = val_common_meta ? meta[val_common_meta] : meta
            [groupKey(newMeta, count), meta, vcf, index]
        }
        .groupTuple()
        .ifEmpty {
            error("ERROR: grouping operation resulted in an empty channel.")
        }
        .map { key, meta, vcf, index ->
            cleanedMetas = meta.collect { m -> 
                    m.findAll { k, v -> k != val_common_meta }
            }
            newMeta = [(val_common_meta): key.target, metas: cleanedMetas]
            [
                newMeta, vcf, index, vcf.size()
            ]
        } // Compute number of records
        .branch{ it ->
            one: it[3] == 1
            more: it[3] > 1
        }

    // Concatenate vcf with more than one record
    BCFTOOLS_CONCAT(ch_concat_input.more.map{ it -> it[0..2] } )
IMO a subworkflow should only ever add keys to a meta map (and my preference is to prefix the keys with an 
_
), not overwrite or remove them. It should be the responsibility of the subworkflow user to stage the input metas appropriately for joining/grouping.
I agree ! However in this particular workflow there is a grouping operation. Therefore my way of handling things would be to get the grouping key as the main key and register all the others into a 'metas' key to keep track of all file processed.
This produce something like
Copy code
{
  "sample": "test",
  "metas": [
    {
      "id": "test_1",
      "other": "annotated"
    },
    {
      "id": "test_2",
      "other": "not_annotated"
    }
  ]
}
j
Ah, that makes sense too. Can val_common_meta be a list that you use to subset the meta with subMap? That might be a bit cleaner, specify the keys which define a group
l
This is something that I would like to try !
With the following
Copy code
ch_concat_input = ch_vcfs
        .map { meta, vcf, index, count ->
            newMeta = val_common_meta ?
                val_common_meta.collectEntries { key -> [(key): meta[key]] } :
                meta
            [groupKey(newMeta, count), meta, vcf, index]
        }
        .view()
        .groupTuple()
        .ifEmpty {
            error("ERROR: grouping operation resulted in an empty channel.")
        }
        .map { key, meta, vcf, index ->
            cleanedMetas = meta.collect { m -> 
                m.findAll { k, v -> !(k in val_common_meta) }
            }
            newMeta = key.target + [metas: cleanedMetas]
            [
                newMeta, vcf, index, vcf.size()
            ]
        } // Compute number of records
        .branch{ it ->
            one: it[3] == 1
            more: it[3] > 1
        }

    // Concatenate vcf with more than one record
    BCFTOOLS_CONCAT(ch_concat_input.more.map{ it -> it[0..2] } )
I obtain for a grouping array 
sample, key
Copy code
{
  "sample": "test",
  "key": "1",
  "metas": [
    {
      "id": "test_1",
      "other": "annotated"
    },
    {
      "id": "test_2",
      "other": "not_annotated"
    }
  ]
}
j
Copy code
.map { key, meta, vcf, index ->
            cleanedMetas = meta.collect { m -> 
                m.findAll { k, v -> !(k in val_common_meta) }
            }
            newMeta = key.target + [metas: cleanedMetas]
            [
                newMeta, vcf, index, vcf.size()
            ]
        }
The idea is to store the unused meta components as a list of maps inside a new key?
j
Jumping in very briefly: I'm pretty sure we've already handled this in one of the 
bwa
subworkflows that you could also look for inspiration
n
You can blame my younger self for creating this mess, sorry about it 😁. Feel free to fix whatever is needed for it!
l
The idea is to store the unused meta components as a list of maps inside a new key?
Yes, indeed!
j
It might be worth including meta.id with each meta remainder so it can be re-identified?
l
I've created the PR with the update: https://github.com/nf-core/modules/pull/9856 Could I have a review ?
Open in Slack
PreviousNext
Powered by