Good morning, NF community! I am writing because I am having trouble creating a channel from an S3 file. I would like to create a unique list of chromosomes from a bed.gz file (1.9Mb) and then make a groupTuple with the size parameter based on the length of this list. When I try to use the command with a local file, the list comes out perfectly. However, when I use a local path from s3, the length of the list and its elements vary. When I run it on AWS batch, the variable is not even defined in the workflow. Do you have any advise?

#!/usr/bin/env nextflow
workflow {
chrlist = Channel.fromPath("<s3://path/to/the/file/bed.bed.gz>")
          .splitText(decompress: true) { line -> line.split().first() }
          .filter { chr -> chr.matches('^chr([1-9]|1[0-9]|2[0-2]|X|Y|M|MT)$') }
          .unique()
          .toList()
          .view()

chrlist.size().view()
}

I have no idea why this fails, but this is more computational work than simply parsing some small sample sheet, so I would recommend doing that work in a

process

(e.g. with

zgrep -oP

, you already have the regex, just replace the

$

with

\t

and pipe into

uniq

. (If your

grep

doesn't have

-P

, use a literal TAB and

-E

.)