I am running into issues setting up the config fil...
# mag
j
I am running into issues setting up the config files. Would anyone be able to assist me? FYI, I am trying to run a huge dataset of co-assemblies in hybrid mode on a HPC at the Helmholtz Center for Environmental Research - UFZ. So the configuration of resources is somewhat confusing me since I have to submit a slurm submission script
j
Happy to help (I'm on the other side of Leipzig even 😉 ) In the meantime, have you tried following this tutorail: https://nf-co.re/docs/tutorials/use_nf-core_pipelines/writing_institutional_profiles If by 'configuring' the pipeline you mean getting it to work on the HPC vs. setting the write parameters
j
Hi James, Thanks a bunch for the quick reply. Would it be possible to set up a zoom meeting? Maybe it would be quicker to clarify some issues. There is a lot of documentation to go through 🙂
j
I'm very busy this week unfortunately, lots of meetings and running a workshop thurs/friday
You could maybe use our #C06KSS9L8M7 on Thursday though if you can wait until then, I can let the person hosting that you would be coming Otherwise I'm happy to answer questions here in betwee nmeetings
j
No worries James (I also have to give a phage workflow tutorial for the next 3 days) 🙂. Depending on the time I might be able to join the weekly helpdesk. In the meantime would it be possible to send you personally my scripts and config files for you to have a look at once you have time? I know I'm being a bit annoying but would love to have this workflow up and running.
j
Sure, send me the scripts/config file, and the problem you're having
You can just post it here, unless teh scripts/configs have something sensitive in which case just send via DM
j
Also, I think I might need to rethink my strategy for this specific project. I have 36 samples of both short and long reads. They are quite big files and I would like to have a hybrid co-assembly but the resources required would be humongous. Short read folder amounts to 74Gb and long reads = 103Gb
j
OK, That I can't really comment on
But is is your issue with teh main configuration?
I see a couple of things, but let me know your main issu efirst
j
I sometimes get resource errors. So I guess it's mostly about setting up the config file and making sure it doesn'T conflict with the actual submission script to the slurm HPC
j
I think reading the documnetation page I linked above will help you a lot with that 😉
I thnk that's one of things you are missing
j
I have to admit that I'm not versed on nextflow so it takes me while 🙂
j
But we should really make a separate config for the HPC specifically https://github.com/nf-core/configs/
and then have a custo mone for the pipleine/your projec titself
👍 1
I have to admit that I'm not versed on nextflow so it takes me while 🙂
That's fine! That's what teh tutorial is about to help people 🙂
j
Hi James. First of all happy new year!
I am still trying to get things running smoothly on the pipeline. I do have a question regarding the use of GPUs
is it possible to run the mag pipeline with them?
I'm thinking it would speed up the process of a lot of tools
j
Not that I'm aware of. But I think other than maybe a couple of binners none of the tools re designed for GPU/do not have GPU modes anyway...