Hi, I am looking for suggestions how to include a ...
# configs
h
Hi, I am looking for suggestions how to include a custom genome config (similar to igenomes but internal) in our institute profile so that users can directly use genome files that are made available genome on our HPC, any input is highly appreciated... Currently, I have added this line to the profile config but the lint test is failing during the pull request since the config file is not found in the configs repo.
includeConfig '/internal/path/readable/by/all/genomes.config'
PS: I did ask a similar question earlier but didn't get to any conclusions so I am trying my luck again! thanks for your patience
j
<!subteam^S044DU0NG68> ant idea?
j
Might be farfetched, set a `default_path`:
Copy code
def CUSTOM_GENOMES_PATH = System.getenv("CUSTOM_GENOMES_PATH") ?: "/default/path/to/public/repo"
In the docs mention that they need to patch this ENV variable to the shared config file.
👀 1
p
Does this configuration need to be secret? Otherwise you can either include it in the configuration, or publish it somewhere accessible (I don't see any problem with considering this a special "pipeline" and putting it under https://github.com/nf-core/configs/tree/master/conf/pipeline, but it wouldn't need to be there).
a
@Maxime Garcia eyebrows
Simplest way is to override the
params.genomes
in a nextflow.config file:
Copy code
params {
    genomes {
        "mygenome" {
            fasta = "path.fasta"
            ...
For importing to all workflows, add it to somewhere all Nextflow users can access it: https://www.nextflow.io/docs/latest/config.html
m
I'd even go for pipelines specific configs as genome params varies from 1 pipe to another
a
Currently, I have added this line to the profile config but the lint test is failing during the pull request since the config file is not found in the configs repo.
includeConfig '/internal/path/readable/by/all/genomes.config'
Thing is, this makes sense because your pipeline can't run on Github. Indeed, no-one can run your pipeline outside of your home institution. Rather than making thinking about how to get around the genomes config, think about making it run everywhere. Can you include a custom trimmed down genome within the repo itself for testing? How about including your
genomes.config
file in the repo?
p
Although I think this is for an institutional config rather than a pipeline,. But it's probably still a good idea not to try to get rid/workaround of the tests on github.
this 1
h
the
genomes.config
file is changing frequently by adding/updating genome files to it therefore I thought to just keep it somewhere local instead of pushing to the
configs
repo all the time!
Since it is a general config file, my aim is to use it for multiple pipelines (rnaseq, chipseq, etc) just to make it easier to provide required reference files for each pipeline.
j
I think they are referring to a repo that you control, different from the
nf-core/configs
.
h
aha Ok, then how do I refer to the file in that new repo within my inistitute profile config ?
j
includeConfig "<https://raw.githubusercontent.com/husensofteng/nf-kaust-config/path/to/custom/config>"
think it can be done like that. Nothing to fancy :D
☝️ 1
☝🏻 1
🎉 1
h
wow didn't know that works! thanks I will try it
🎉 2
🎉 1
p
Note that quite a long time ago we also had the intention that folks would use RefGenie for this: https://nf-co.re/docs/usage/reference_genomes#using-refgenie-for-genome-management
but in practice I'm not sure how many people do this.
Either way, agree that the above includeConfig approach is probably the best 👍🏻
h
Actually, I started with
RefGenie
first, then moved to
genomepy
, and finally it seemed that just having a script to download
fasta
and
gtf
files from
Ensembl
and
NCBI
for a given species, then running the
reference
workflow that @Maxime Garcia is working on to generate indices, etc would be the best option to generate an internally maintained genome resource. Finally, a script scans the genome directory and generates this config to make the genome files accessible by the workflows 🫠
p
Yup, having automatic config file generation would definitely be nice once #C073ZDLNC92 picks up a bit more 👍🏻
this 2
👍 1