Do you know any smart (and low mem) solution to li...
# pipeline-maintainers
j
Do you know any smart (and low mem) solution to linecounting on the fly? A quick solution I’m experimenting would be something like this:
Copy code
import java.io.BufferedReader
import java.io.FileReader

// Define a function to count lines in a file using BufferedReader
def countLines(filePath) {
    def lineCount = 0
    new BufferedReader(new FileReader(filePath)).withCloseable { reader ->
        while (reader.readLine() != null) {
            lineCount++
        }
    }
    return lineCount
}

// Use the filter channel operator with a closure to check the line count
OPENMS_IDFILTER_FOR_SPECLIB(ch_fdrfilter_comet_idxml)
    .filtered
    .filter { meta, idxml -> countLines(idxml.toString()) >= 130 }
    .set { ch_filtered_idxml_for_speclib }
j
Is there a reason the built-in
countLines
function won't do? https://www.nextflow.io/docs/latest/reference/stdlib.html#splitting-files
this 1
nf core duck 2
j
Thanks! That one I was looking for facepalm
j
Tip that I learned the other day: it will decompress a gzipped file if it has a
.gz
extension or you pass the argument
decompress: true
fbwow 1
p
Warning: If you call this on input data and someone throws a bunch of 500GB
.fastq.gz
files into your pipeline, the Nextflow head job is gonna go sloooow. Also those files will need to be staged to the tmp directory if they're not local, which can fill up local disk space.
🫢 1
šŸ‘€ 1
If you're counting big files, it might be better to throw into a process so that it runs on a compute node in parallel (though that comes with downsides, such as needing to schedule jobs and stage the data, so it's swings and roundabouts)
j
I learned my lesson some time ago when I was running countFastq on hundreds of input files the head node...
šŸ’Æ 1
Though it would be nice to be able to cache this sort of thing - is there a plan to incorporate something like the nf-boost inline exec operator in nextflow?
p
Feel free to make an issue with a use case example šŸ‘šŸ»
šŸ‘ 1