Anybody have any recommendations regarding best practices for a mixed-input of gzipped/uncompressed files? E.g. take an arbitrary channel of mix of .fastq and .fastq.gz, and you want to send this channel to a downstream process that requires gzipped input. My predicament is I have a client that wants a mixed-input samplesheet, but I don't want to 1) Add a complicated ad-hoc module to normalize it and 2) I prefer not to modify nf-core's fastp module. I am wondering if anybody has some fancy groovy/channel magic that would simply sift out the uncompressed inputs, compress them, and mix them back into the channel with correct suffixes. My rational for not wanting another module is I don't necessarily want to spawn off jobs for this, and would prefer to do this with nextflow/groovy if possible. I want to avoid changing the fastp module simply because I don't want to maintain my own. Thanks!

You can branch your channel based on filename and then merge after you compressed them

Thanks for the syntax reference that will help a lot, do you know of any way I could get around making a module for it or is that just what has to be done?

There's already a module for compressing data, you just need the correct channel logic in your pipeline using the branch operator

@Matthias De Smet I'm aware of the existing module, I was trying to see if there was any way around me using a module for it, as I am on HPC and job submission can be a bottleneck. I can work around this by specifying that single process to run locally using withName, but that felt a little janky. If there is no groovy + Nextflow channel operator magic that would do this without having to use a module directly in the workflow, I'll just stick to convention. I was just curious if anyone else has attempted this

Offloading processes to the head job is generally a bad idea, due to overhead and staging etc

Generally, I agree. The use-case at my institution is a bit bespoke, which is why I assume others aren't doing this. It's better to offload relatively low resource tasks like file compression, text parsing, etc. to avoid the I/O load from creating all the work files and mainly avoid the job submission bottleneck that is caused by us having a very small compute cluster

There are some things that can be done in a Nextflow channel directly (I think counting records in a FASTQ is one of them), but compression needs potentially specialized tools (depending on compression algorithm and indexing, etc.), so I don't think that's included, i.e. you have to use a process. The way I see this is that you either have a preprocessing script that compresses any uncompressed input and "fixes" the samplesheet, or you put up with the costs of having an extra process. I don't think the job submission per-se is costly (for most schedulers), just the wait for an open slot is. If you have a queue that accepts (only) small jobs (and lots of them, since they run in parallel and share a node), you can make a config that routes those compression jobs to that queue and the wait should not be that long. (There is also the

process.beforeScript

directive, but I'm not sure the files would be staged when that is invoked, and it runs outside the container, with only OS tools available, so I don't think you can use that to e.g. compress the staged files.) Another option (abandoning your premise of not touching existing modules) might be to "patch" the nf-core fastp module so either accepts uncompressed input or runs the

gzip

command on the uncompressed file(s) (with the decision based on file name extension or output from

file

command). If you are developing a pipeline, that might be the most efficient way of dealing with this (at least until changes to the module break your patch ...). If your patch works well, could be universally useful (I think uncompressed input would be), and does not significantly increase runtime or long-term maintenance cost, you could propose to incorporate it into the module upstream.