Following up, it looks like my pipeline finished b...
# taxprofiler
m
Following up, it looks like my pipeline finished but most of the reads are 'Unclassified'. I'm not sure why this would be. I suspect it is because I was using the databases provided by this command: curl -O https://raw.githubusercontent.com/nf-core/test-datasets/taxprofiler/data/database/kraken2/testdb-kraken2.tar.gz curl -O https://raw.githubusercontent.com/nf-core/test-datasets/taxprofiler/data/database/centrifuge/test-db-centrifuge.tar.gz curl -O https://raw.githubusercontent.com/nf-core/test-datasets/taxprofiler/data/database/kaiju/kaiju.tar.gz Are these incomplete versions? Thanks, again!
j
If you're just running the test profile, yes it's expected most of the reads are unclassified - for our tests we just need the tool to run as fast as possible without 'erroring' , it doesn't need to make sense. But if you are using those 3 databases on your own data, this is definitely not advised as they are not real databases 😅. Taxprofiler doesn't come with it's own pre built databases as this is another area in metagenomics where there is no gold standard. I built a whole sister pipeline #C062H82UCRG for this purpose (building custom databases). That said all the tools you mention above do provide their own pre built databases in various locations you can use
(sorry if you are aware of these points, just not sure exactly what you were trying to run here, so trying to cover all bases)
m
James, Thanks so much!! I was definitely trying to run them with my real data haha, my bad! I downloaded a prebuilt Kraken2 database and kicked off the pipeline again. It's (naturally) taking much longer now. I did add the db_param --memory_mapping as Kraken2 recommends. Any other tips for helping it run faster / smoother? Finally, even though I picked the Standard Kraken2 database, there are still a pretty large number of unclassified reads. Is there a way to investigate further into what these are? Thanks for your help! Truly!
m
What is a pretty large number to you and what kind of samples are we talking about? Anywhere from 5-70% unclassified can be expected depending on what was sampled.
this 1
j
(and what is in the database etc)
m
Gotcha!! The samples that have completed so far have between 35-62% unclassified. Because of our depth, that still gave us 700k to 1M assigned reads.
I was using the 'Standard' database from this link: https://benlangmead.github.io/aws-indexes/k2
m
Okay, so your sequencing is fairly shallow. And what are you sampling? There may be more tailored reference databases.
👍 1
m
These are cecal contents from mouse. It's possible that some reads are host.
Also, thanks so much for your guidance! I thought we had sequenced to a sufficient depth so it is good to know that's shallow as well.
(It's our first shotgun experiment so we're learning a lot!)
j
I think the recommended minimum was around 2million
m
Host removal is straight forward with this pipeline. You just need to provide the mouse genome.
j
We chose 0.5 million for evaluation here because it is the highest depth at which the sequencing cost is still less than approximately half of the total cost of generating data, and yet it is a depth at which one still obtains reasonable sensitivity for species-level recovery; however, we do encourage readers to increase depth toward 1 million or 2 million reads when budget allows, as this will continue to increase sensitivity for detecting rare species.
(anyway back to nfcore retreat now!)
m
When I switched from kraken standard database to one based on MGnify for human fecal samples, I was able to classify many more reads.
🙌 1
this 1
I don't know how similar human and mouse gut microbiome are.
m
Thanks, James! I'm guessing they mean 2M assigned reads? For most samples we hit > 2M reads just about half are unclassified
And thank you Moritz!!
Lastly, any tips on improving my processing speed? I turned on the --memory-mapping flag for Kraken
m
Did you copy the database to an in memory directory beforehand? Otherwise the option will not have an effect or even make it worse.
m
I'm not sure what you mean, sorry! I have the database in the .gz format in a top level directory and am working within a subdirectory. When I didn't have the --memory-mapping flag, the pipeline kept crashing saying it couldn't allocate enough memory so I wasn't sure how to handle that.
I was looking for one of the MGnify databases like you menioned. I found this site: https://www.ebi.ac.uk/metagenomics/browse/genomes Which resource did you download for your more successful analysis?
m
Have a look at the kraken2 manual regarding memory mapping. I'm on my phone right now.
👍 1
This one https://www.ebi.ac.uk/metagenomics/genome-catalogues/human-gut-v2-0-2 but as mentioned, I don't know if you can expect similar species in mice.
m
Thanks for all the tips! I've made some edits to my script and am kicking off the pipeline again. The computational environment I'm working in has 32 CPUs and 60 GB memory. Do you think this is why things are going so slowly?
m
I worked in an environment similar to that, but I had samples with up to 100 M reads. Took between 5-20 minutes per sample.
m
Hmmmmm. Okay! Were you multithreading? Mine is going substantially slower than that.
m
6 or 8 threads yes
I suspect that when you properly set up the memory mapping, it will go much faster.
m
I've been trying to do more reading on what I am doing wrong regarding the memory-mapping, but can't find any solutions. In both copies of the manual I've found it only says that you can turn that feature on but doesn't indicate any specifications beyond that. When you say I have it improperly set up, do you mean I'm running it from the wrong place? Or maybe my database is in the wrong place? So sorry, I've been trying to dig on that but no luck.
m
You need to create an in memory file system like tmpfs and unpack the database there. Only then will the memory mapping option help you, as kraken2 will not load into memory once more.
m
Okay, so create a temp folder, untar the database there, then point the pipeline towards that location for the database instead?
So sorry. I am really trying to follow
m
I'm still on my phone, hence I'm rather brief. I can send more info tomorrow.
👍 1
m
So sorry for bugging y'all again. Was that strategy I outlined valid - in which I would copy the database into /tmp and untar it, then point the pipeline towards that unpacked database? I was trying that but my system keeps crashing when I try to copy it into /tmp
m
I dont know if you /tmp is in memory (it often is) and I don't know how much space is allocated to it. Do you have someone administrating your workstation? Perhaps you can ask them to create a tmpfs and mount it for you? Otherwise your steps are the right ones.
m
Unfortunately I don't have anyone administrating. I had tried to do the following:
sudo mount -o remount,size=1000G /tmp
To give plenty of space for copying it over and unpacking. Maybe it is in memory. If I do df -hT I can see my file systems and do have a few /tmpfs options.
image.png
Is it possible / okay to delete things from within the work folder to try to free space? It is > 0.5 Tb right now but I don't want to delete anything if it will mess up the cache
m
You said that your system has 64 GB memory. So you cannot have a tmpfs filesystem that is larger than that. You must also leave some memory for the system to operate. Your /tmp mount has 32 GB, that's probably a good start. Although I vaguely remember some issues with nextflow and the /tmp directory.
You can definitely clean up the work dir but you'd have to go through nextflow log to decide what is safe to delete.
j
Two things on cleaning to expand a bit on what Moritz said: You can have nextflow to clean the work directory on successful runs by setting
cleanup = true
in a nextflow config And then using the info from
nextflow log
to get run names, and other info, you can use
nextflow clean
with that info to do the cleaning. Otherwise it you just want to start completely from scratch, it's no problem to just
rm -r
the work directory, if nothing is running and you don't want to resume
m
Thank you guys! I went ahead and untarred the database within my directory to try to alleviate the issue. Which file do I point to now for the database? I see taxo.k2d or ktaxonomy.tsv?
Or just the whole root directory
j
Most tools in taxprofiler require the directory
Including kraken
m
Thanks! It's running! Very slow but running!
🎉 1
The pipeline is still running but none of the samples have passed through Kraken2 yet (the counter is at 0 of 48).
j
Are you concerned it's got stuck?
m
I'm not sure. I can check the logs to see when the last file was written
It looks to be still running.
👍 1
Sorry to ask so much, just trying to be mindful of resources as we use AWS so we pay per hour haha
j
You can also change into the working directory of the process and run
tail .command.log
to see if it's doing something
No worries!
Are you using seqera platform? It might be able to help with a lot of these things, and it's main focus is on Aws actually
m
I'm not, but I do need to look into that.
I have an account on it and have used it at previous workplaces but haven't implemented here yet
👍 1
m
Are you renting a single EC2 instance or using something like AWS batch?
m
I have a single EC2 instance that is quite large that I'm using
👍 1
m
Then I'm not sure why it's so slow with your rather small sequencing runs.
m
I'm actually not sure it is running I suppose. In htop I see the classify commands but the status is D for all of them. It says files are being written to in /work so I suppose it is still going?
Update - the first sample finally finished running through Kraken2
m
Hmm, what kind of disk did you choose for your instance? Network storage will be rather slow. A local NVM drive performs best.
m
The root device type in the instance is EBS
Not sure if that's the info you needed
m
If I'm not mistaken, that is network attached storage, yes. So it will be quite a bit slower than a disk physically present on the node.
m
As of this morning, 3 of 48 samples have finished. Do you think I should just spin up a larger EC2 instance?
m
I would ask folks more experienced with AWS. The instance size should be fine, but maybe you need either a fast, large enough, local disk or really embrace horizontal scaling with many instances via AWS Batch.
👍 1
m
Thank you! I'll reach out to someone at AWS to see if I can get some help
m
There are also folks here in the community who specifically know about using nextflow on AWS.
m
Just one last question for you guys here. Since I unpacked the database, should I remove the --memory-mapping flag now? Is that potentially slowing things down?
m
No, that will be required to speed things up.
👍 1
m
All, This run had been going for ~6.5 days and had 22 of 48 samples passed through Kraken2. Despite indicating I have 32 CPUs available and indicating --threads 6 as a command for Kraken2, I only saw one thread running using htop. I decided to stop the run and tweak the config file to try to force multithreading, but when the pipeline resumed, it shows 0 of 48 samples for Kraken2 now (i.e. none cached). When I look in the results folder, the output files from the 22 samples that finished before are still there so I'm not sure why it is not detecting the cache. Can I get some help on what is going on?
Update: I tried resetting everything to how it was (using the original nf.config) and running again but it still isn't recognizing the cache.
I think I just lost >$1,000 of compute time 😞 Please help if possible
j
Changing the threads outside of the native method will change the command and retrigger the process as it's no longer the same, so that's to be expected unfortunately:(
You should still be able to recover the original run though to resume, if you get the run name (you should be able to find it with `nextflow log`if you don't remember But you need to find out exactly what was causing it run so slowly. Unfortunately I don't have much experience on AWS, so I don't know how to check it :(. Did you check the `.commsnd.log`in the working directory of the running process? If you still have the work directory you should still be able to look in it. What was the command that was running on one CPU on htop?
Can you send your config file as well? It's odd it's taking the 6 CPUs and not 32 too
And how exactly are you submitting the pipeline? On a single node?
m
There are significant portions in kraken2 that are single threaded. If those are also related to file writing and that's very slow on EBS, you will see it running on one core for long periods of time.
👍 2
j
Good to know
m
@Megan Justice were you able to compare with a node that has SSD or better NVMe storage on the node?
m
I have not been able to try the pipeline on a different EC2 instance as I'd have to spin one up and am trying to be mindful of cost. If you guys think it will be helpful to create a new instance and set things up differently I can definitely attempt that. How much storage space would you allot in the new instance? It looks like the work directory alone is > 0.5 Tb and I'm only working in a 1 Tb space. Is the work directory always so large during runs? As for resuming, I had been resuming using the job ID. Should I use the run name instead? Will that do anything differently? Thanks for all your help
m
I'd set up a smaller instance just to try for one or two samples.
Regarding space, it depends on your batch size. I worked with a 4 TB disk in the past and it filled up easily for a few hundred samples.
m
I only have 48 samples that as you said are not terribly deeply sequenced, so I hoped the space I had set up was sufficient. I can look into smaller instances with SSD
m
You probably only need it for two hours. Shouldn't cost much.
It seems that you will need a Nitro system for NVMe support. I haven't done this on AWS myself, so I can't recommend a setup for you, sorry. Try to figure it out here https://github.com/awscloudvendor1/amazon-ec2-user-guide/blob/main/instance-types.md#ec2-nitro-instances
m
Thanks! I'm also meeting with some folks from AWS to try to get their take on these issues tomorrow so they may have a suggestion
I tried to do -resume (run_name) on the run that had completed 22 samples in Kraken2. It still shows no cache for Kraken2. I was hoping to salvage what I had already done. Is there a way to help point it towards the files that have already been made?
(doing this in addition to trying another instance as I need to get the data analyzed)
I also tried doing a cleanup of everything after the run that had 22/48 and resuming again but no luck
m
If you have the result files already, you can also remove those 22 from your sample sheet.
m
I have the Kraken2 output from those. I wasn't going to use any other databases. Were there more output files I'd be missing for those 22 if I omit them from the sample sheet and restart ?
m
I think, you're good then.
🙌 1
m
Looking back at the nf-core/taxprofiler page I don't actually think the Kraken2 output is complete for the 22 that had finished. The only file is the sample.kraken2.report.txt for each file. I tried to have it output the classified and unclassifed reads in fastq format but I don't see those or a classifiedreads.txt document either.