HI
@a.br.ravichandran, thanks a lot for your question.
I think you're missing the annotation-cache s3 bucket from your list.
I am also unsure you need to whitelist all these domains/endpoints.
As replied previously,
We have a dedicated resource for this particular topic on our website:
https://nf-co.re/usage/offline
The nf-core tools CLI has a dedicated subcommand to download most dependencies as Singularity/Apptainer images:
https://nf-co.re/docs/nf-core-tools/cli/pipelines/download
For the necessary data, I recommend pre-downloading the files from ngi-igenomes.
Follow the genome key to get the list of files from
https://github.com/nf-core/sarek/blob/master/conf/igenomes.config
For annotation, I recommend
https://annotation-cache.github.io/ that has built-in support in Sarek. Similarly, you can pre-download these files.
From what I understand you do not run in a offline setup similarly to how we would run things in a typical academic setting.
In that case, what we would do would be download all necessary dependencies and then transfer all to a closed environment.
For me, for an offline usage, I would.
Pre-download nextflow and all necessary plugins.
(versions can be found in the root nextflow.config).
If you plan to use docker over singularity/apptainer, I would recommend having a custom registry, and overriding via a custom config file all container declarations and pointing them all to your own containers.
I would recommend to mirror the s3 buckets locally (no need to have all files, just the one you need for the analysis).
so in sarek case, annotation-cache on top of igenomes.
I would love to take this as an opportunity to improve the documentation we have on running offline.