https://nf-co.re logo
Join Slack
Powered by
Does anyone have any Intuition as to why this bit ...
# help
j
Does anyone have any Intuition as to why this bit of code Nextflow code
Copy code
channel.of('1,2\nalpha,one', '1,2\nbeta,two', '1,2\ngamma,three')
    .collectFile(
        name: 'sample.txt',
        storeDir: '/home/james/git/jfy133/NextflowSandbox/collectfile_manualheader',
        newLine: true,
        keepHeader: true,
     )
    .subscribe { file ->
        println "Entries are saved to file: $file"
        println "File content is:\n${file.text}"
    }
Results in an empty line between the 'kept' header, and the rest of the rows? I'm guessing it's due to the 
\n
, but I don't see how to otherwise specify the header based on the string 🤔
Copy code
1,2

alpha,one
gamma,three
beta,two
If you're free @Mahesh Binzer-Panchal, I see from GitHub issues on Nextflow you've explored odd behaviour from 
collectFile
before
n
Have you tried 
channel.of('1,2', 'alpha,one', 'beta,two', 'gamma,three')
?
j
Yes, but it doesn't help me as 1) it's not how my 'real' use case is foramtted and b) I want to use 
sort: true
to ensure stable output files, and if I do it like you have suggested the header won't be at the top
Copy code
gamma,three
alpha,one
beta,two
1,2
OK, if I hard code a new line, and turn off the Nextflow 'newLine' thingy itworks:
Copy code
// This works but weirdly produces an empty line between the header and the rest of the rows
// Specify the header for each element in the map, split with a new line
// In collect file make sure to specify newLine to ensure each element is a new row, and keepHeader to keep the first half of the element and share across all
channel.of('1,2\nalpha,one\n', '1,2\nbeta,two\n', '1,2\ngamma,three\n')
    .collectFile(
        name: 'sample.txt',
        storeDir: '/home/james/git/jfy133/NextflowSandbox/collectfile_manualheader',
        keepHeader: true,
     )
    .subscribe { file ->
        println "Entries are saved to file: $file"
        println "File content is:\n${file.text}"
    }
n
I've done it like this in some pipelines: 1. Define a channel with the header 2. Collect all entries, sort them and flatten them again so they have the same order everytime 3. mix the channel with entries into the channel with the header and then pass it to collectFile
👍 1
j
Copy code
1,2
alpha,one
beta,two
gamma,three
Yeah, that's what Mahesh has done as well in the past but it never felt satsifyhing to me (having to make a separate channel just for the header 😕
But maybe it's the more nextflowy way... 🤔
For context this is ACTUALLY what I'm trying to do:
Copy code
ch_bin_contig_names = ch_binning_results_gunzipped
        .transpose()
        .map { meta, binfile -> [meta + [bin_id: binfile.name], binfile] }
        .splitFasta(record: [header: true], elem: 1)
        .map { meta, contig_header ->
            "bin_id\tassembly_id\tbinner\tcontig_id\n${meta['bin_id']}\t${meta['assembler']}-${meta['id']}\t${meta['binner']}\t${contig_header['header']}\n"
        }
        .collectFile(
            name: 'contig_to_bin_map.tsv',
            storeDir: params.outdir + '/GenomeBinning/QC',
            keepHeader: true,
            sort: true,
        )
n
I think it's the easieast way to do it like that, (still could be easier if the function was written in another way but oh well)
And you're not actually creating a channel just for the header, you're initialising a channel for collecting that starts with a header 😉
a
I'm fairly convinced this same issue came up somewhat recently in another lengthy thread, but I can't find it right now. Could well have been on the 
nextflow
slack too
👍 1
j
I opted for my appraoch as the separate cahnnel system started almost doubled the lines of codes
a
j
Aha, so I indepdndently used the same approach from @Maxime Garcia basically..
👀 1
m
What did I do again?
m
Is my input still needed?
j
Opinions still welcome of maybe any cons to my method (which I've implemented now), and/or any suggestions why I was getting the random 'empty' line below the header - but otherwise no, don't need any active fixes 🙂
m
Interestingly, if you use:
Copy code
channel.of(
"""\
1,2
alpha,one
""",
"""\
1,2
beta,two
""",
"""\
1,2
gamma,three
"""
        )
    .collectFile(
        name: 'sample.txt',
        // storeDir: '/home/james/git/jfy133/NextflowSandbox/collectfile_manualheader',
        // newLine: true,
        keepHeader: true,
        // skip: 1
    )
    .subscribe { file ->
        println "Entries are saved to file: $file"
        println "File content is:\n${file.text}"
    }
it works as expected.
Copy code
File content is:
1,2
alpha,one
beta,two
gamma,three
Edit: Never mind. I see you discovered the ending in new line fixes it already.
What's going on here?
Copy code
.map { meta, binfile -> [meta + [bin_id: binfile.name], binfile] }
        .splitFasta(record: [header: true], elem: 1)
Since when did 
splitFasta
go into a tuple to find the file?
and since adding a new line is needed to fix it, you can try something like:
Copy code
channel.of('1,2\nalpha,one', '1,2\nbeta,two', '1,2\ngamma,three')
    .collectFile(
        keepHeader: true,
    ) { txt ->
        ["sample.txt", txt + '\n']
    }
    .subscribe { file ->
        println "Entries are saved to file: $file"
        println "File content is:\n${file.text}"
    }
and just to make sure it works with a file too
Copy code
channel.fromPath('data/*.txt')
    .collectFile(
        keepHeader: true,
    ) { infile ->
        ["sample.txt", infile.text + '\n']
    }
    .subscribe { file ->
        println "Entries are saved to file: $file"
        println "File content is:\n${file.text}"
    }
👍 1
j
AH the closure is what I was looking for 🙃 , I still didn't fully understand the 
["sample.txt"]
but which is why I don't think ti was working for me...
m
Yeah. First element is filename, second is content, which is then filtered through the stuff like keepHeader, etc.
👍 1
j
Thank you ver ymuch!
🍻 1
2 Views
Open in Slack
PreviousNext
Powered by