I am making a module that downloads data packages into a local directory. the input has no meta, as one is not needed. Ideally, I would like the output files to have a meta map where the id is the dirname of each output file. Is there a way to do this at all? e.g.

output:
my_data_package/
    file1
    file2

So that the channel output is not simply:

channel1 = [file1]
channel2 = [file2]

but

channel1 = [ [id:"my_data_package"], file1]
channel2 = [ [id:"my_data_package"], file2]

You can let Nextflow download files itself using staging (which is way easier to use than a module IMO). You can then do something like this to get the dirname of the file:

file(<path-to-remote-file>).parent.name

this particular tool is built for data download, from its own archives, and then validates the data etc, so I wouldn’t use nextflow’s inbuilt staging in this case. Would the

file(<path-to-remote-file>).parent.name

still work when multiple files go into each channel tho? i.e. there’s multiple files being downloaded that would end up in the same output channel because they share a file extension

Ah that's understandable 😛 and yes you can for example get the dirname of the first file if they are all located in the same directory.

out_channel.map { files -> [ [ id: files[0].parent.name ], files ]

sadly not. if multiple data packages get downloaded, then one file from each package would end up in each channel 😅 I would have preferred to have one channel with all the files of a package in it, but I think that clashes with the nf-core module guidelines:

Give each file type its own `emit`entry, usually one per format.

It depends a bit on the use case. Do you need to provide the full package to another downstream process?

generally, yes. downstream tools can use only some parts of a package, but for most cases we would pass the entire package downstream

Yeah just output the whole package at once then I'd say 😛

thanks for the input! That would at least make the channel logic downstream to add a meta a lot simpler.

no problem! glad I could help

You can also capture the dirname in an

env

, and then transform that into a metamap