https://nf-co.re logo
Join Slack
Powered by
Hello everyone, I've asked the following on the he...
# rnasplice
r
Hello everyone, I've asked the following on the help channel but I figured it is probably better to ask here. I am trying to run the rnasplice pipeline offline (on a cluster with offline compute nodes), and I have two newbie questions. First question: I have downloaded nf-core-rnasplice_1.0.4 which has three directories: 1_0_4, configs, and singularity-images. When I run _nextflow run nf-core-rnasplice_1.0.4/1_0_4 -name this_run_wont_work -profile singularity -resume -params-file nf-params.json_ I get Failed to pull singularity image command: singularity pull --name depot.galaxyproject.org-singularity-mulled-v2-cf0123ef83b3c38c13e3b0696a3f285d3f20f15b-64aad4a4e144878400649e71f42105311be7ed87-0.img.pulling.1716823765865 https://depot.galaxyproject.org/singularity/mulled-v2-cf0123ef83b3c38c13e3b0696a3f285d3f20f15b:64aad4a4e144878400649e71f42105311be7ed87-0 > /dev/null status : 255 hint : Try and increase singularity.pullTimeout in the config (current is "20m") message: [31mFATAL: While pulling from image from http(s): Get https://depot.galaxyproject.org/singularity/mulled-v2-cf0123ef83b3c38c13e3b0696a3f285d3f20f15b:64aad4a4e144878400649e71f42105311be7ed87-0: dial tcp 128.118.250.14443 connect: no route to host So it seems that the pipeline tries to download singularity images although it should access them in the singularity-images directory. How can I solve this? NB: the above command runs on the online login node (though it fails due to insufficient computing resources). Also, if I use conda instead of singularity, I can first run on the login node so that the process creates the required environments in the ./work directory, and I can run it offline afterwards. Second question: I am trying to run the rMATS analysis on 6 .bam files (two conditions in triplicate) of ~13GB each. How do I estimate how much memory will the pipeline require? In case anyone knows about the above, please let me know! Thanks! :)
t
To your second question: You will need a profile for your cluster. If one in the 
configs
directory comes close, you can just make a copy (
mycluster.config
) and edit it, so it meets your needs. The most important things it should contain are information about the 
executor
(SLURM, LSF, etc.) and the container engine (
singularity
in your case). See some more info here and here. You can then edit 
configs/nfcore_custom.config
to add your 
mycluster.config
with the name 
mycluster
, which will allow you to run with 
-profile mycluster
(if the singularity info is in your config file, you don't need the 
singularity
profile). The Nextflow process itself needs minimal resources. You can submit it as a job (1-2 CPUs, 2-4 GB RAM is more than enough) or even run it in a 
screen
session on the head node. (If you put it in the background or a compute node, redirect stdout and stderr to files.) This process will deploy a bunch of jobs, which will run on the cluster and do the actual work (alignment, rMATS, QC, etc.). The pipeline already has estimates of the resources these jobs need and will submit them accordingly with correct parameters. Your config file should set 
max_cpus
, 
max_memory
, and 
maxtime
in a 
params { ... }
section to the maxima your cluster can handle (ask your sysadmins, if in doubt). This ensures that the pipeline never asks for more (which would prevent a job from being scheduled). In short, let the pipeline worry about how much memory it needs.
To your container problem: If the install worked correctly, your containers should all be in 
singularity-images
in your install directory. Make sure that is the case. If not, repeat the install procedure. (If you run multiple pipelines, you might save some space by having them share a container cache directory. You can put that anywhere.) The next step is to help Nextflow find your containers. A simple way would be with 
export NXF_SINGULARITY_CACHEDIR=/path/to/singularity-images
. (I actually use that during install, so the containers end up in the correct place.) The superior way is of course to to set 
cacheDir
in the 
singularity
scope of your config file, which you are using as profile. Once you have done that, the pipeline should not try to download anything again, and it should work. I assume you have seen https://nf-co.re/docs/usage/getting_started/offline for any other tips.
r
Hi Thomas, this is very useful. Thank you very much for your help!
šŸ‘ 1
Hi! I have a follow up question, unfortunately. I created a profile for my hpc, specifying the directory for the singularity images, and the pipeline starts without errors now. However, it gets super slow at the SAMTOOLS_SORT step because of a memory related issue. Here is the pipeline log so far: ######################################################################## nextflow run ./1_0_4 -profile sissa_ulysses -resume -params-file nf-params.json N E X T F L O W ~ version 24.04.1 Launching 
./1_0_4/main.nf
[cheesy_wozniak] DSL2 - revision: 93d808d3e7 ------------------------------------------------------ ,--./,-. _ _ _ __ _ /,-._.--~' |\ | |__ _ / ` / \ |_) |__ } { | \| | \__, \__/ | \ |___ \`-._,-
Copy code
-,
.,.,' nf-core/rnasplice v1.0.4 ------------------------------------------------------ Core Nextflow options runName : cheesy_wozniak containerEngine : singularity launchDir : /scratch/rimargi/project_CRO/data_may2023/splicing_analysis/nf-core-rnasplice_1.0.4 workDir : /scratch/rimargi/project_CRO/data_may2023/splicing_analysis/nf-core-rnasplice_1.0.4/work projectDir : /scratch/rimargi/project_CRO/data_may2023/splicing_analysis/nf-core-rnasplice_1.0.4/1_0_4 userName : rimargi profile : sissa_ulysses configFiles : Input/output options input : /scratch/rimargi/project_CRO/data_may2023/splicing_analysis/samplesheets/OVCAR8samplesheet_PTresU_PTresSR_genome_bam.csv contrasts : /scratch/rimargi/project_CRO/data_may2023/splicing_analysis/contrasts/PTResSRvsPTResU_contrast.csv source : genome_bam outdir : /home/rimargi/project_CRO/data_may2023/splicing_analysis/rMATS_nfcore_rnasplice/OVCAR8/PTResSRvsPTResU email : rimargi@sissa.it Reference genome options fasta : /scratch/rimargi/project_CRO/data_may2023/genome_annotations/GRCh38.p14.genome.fa gtf : /scratch/rimargi/project_CRO/data_may2023/genome_annotations/gencode.v44.annotation.gtf igenomes_base : s3://ngi-igenomes/igenomes/ Institutional config options custom_config_base : /scratch/rimargi/project_CRO/data_may2023/splicing_analysis/nf-core-rnasplice_1.0.4/1_0_4/../configs/ config_profile_description: SISSA Ulysses cluster config. Max job request options max_cpus : 40 max_memory : 160 GB max_time : 12h Read trimming options skip_trimgalore_fastqc : true QC options skip_fastqc : true Alignment options aligner : star_salmon skip_alignment : true rMATS options rmats : true rmats_paired_stats : false rmats_read_len : 100 DEXSeq DTU options min_samps_gene_expr : 4 min_samps_feature_expr : 2 min_samps_feature_prop : 2 Miso miso_genes : ENSG00000004961, ENSG00000005302, ENSG00000147403 miso_genes_file : null SUPPA options suppa : false !! Only displaying parameters that differ from the pipeline defaults !! ------------------------------------------------------ If you use nf-core/rnasplice for your analysis please cite: * The pipeline https://doi.org/10.5281/zenodo.8424632 * The nf-core framework https://doi.org/10.1038/s41587-020-0439-x * Software dependencies https://github.com/nf-core/rnasplice/blob/master/CITATIONS.md WARN: ============================================================================= '--skip_alignment' parameter has been provided. Skipping alignment, genome-based quantification and all downstream QC processes. =================================================================================== executor > slurm (13) [0e/db3974] process > NFCORE_RNASPLICERNASPLICEPREPARE_GENOME:GTF_GENE_FILTER (GRCh38.p14.genome.fa) [100%] 1 of 1, cached: 1 āœ”ļø [9f/aaec44] process > NFCORE_RNASPLICERNASPLICEPREPARE_GENOME:MAKE_TRANSCRIPTS_FASTA (rsem/GRCh38.p14.genome.fa) [100%] 1 of 1, cached: 1 āœ”ļø [3c/698493] process > NFCORE_RNASPLICERNASPLICEPREPARE_GENOME:CUSTOM_GETCHROMSIZES (GRCh38.p14.genome.fa) [100%] 1 of 1, cached: 1 āœ”ļø [d4/d7d0d3] process > NFCORE_RNASPLICERNASPLICEINPUT_CHECK:SAMPLESHEET_CHECK (OVCAR8samplesheet_PTresU_PTresSR_genome_bam.csv) [100%] 1 of 1, cached: 1 āœ”ļø [f8/5a55c6] process > NFCORE_RNASPLICERNASPLICECONTRASTS_CHECK:CONTRASTSHEET_CHECK (PTResSRvsPTResU_contrast.csv) [100%] 1 of 1, cached: 1 āœ”ļø [ab/8c4816] process > NFCORE_RNASPLICERNASPLICEBAM_SORT_STATS_SAMTOOLS:SAMTOOLS_SORT (PTres_SR_2_c8) [ 63%] 7 of 11, failed: 5, retries: 5 [d3/54a1cd] process > NFCORE_RNASPLICERNASPLICEBAM_SORT_STATS_SAMTOOLS:SAMTOOLS_INDEX (PTres_SR_2_c8) [100%] 2 of 2 [- ] process > NFCORE_RNASPLICERNASPLICEBAM_SORT_STATS_SAMTOOLSBAM STATS SAMTOOLSSAMTOOLS_STATS - [- ] process > NFCORE_RNASPLICERNASPLICEBAM_SORT_STATS_SAMTOOLSBAM STATS SAMTOOLSSAMTOOLS_FLAGSTAT - [- ] process > NFCORE_RNASPLICERNASPLICEBAM_SORT_STATS_SAMTOOLSBAM STATS SAMTOOLSSAMTOOLS_IDXSTATS - [- ] process > NFCORE_RNASPLICERNASPLICERMATS:CREATE_BAMLIST - [- ] process > NFCORE_RNASPLICERNASPLICERMATS:RMATS_PREP - [- ] process > NFCORE_RNASPLICERNASPLICERMATS:RMATS_POST - [- ] process > NFCORE_RNASPLICERNASPLICECUSTOM_DUMPSOFTWAREVERSIONS - [- ] process > NFCORE_RNASPLICERNASPLICEMULTIQC - ######################################################################## I also had a few messages like the following: [c0/c416e7] NOTE: Process 
NFCORE_RNASPLICE:RNASPLICE:BAM_SORT_STATS_SAMTOOLS:SAMTOOLS_SORT (PTres_U_2_c8)
terminated with an error exit status (137) -- Execution is retried (1) I wonder if I interpret this log correctly: so far, there were 5 runs, 5 retries, and 2 completed jobs. The jobs status is: JOBID NAME USER ST TIME CPUS MIN_MEMORY ALLOCATED_CPUS CPU_USAGE 13058933 nf-NFCORE_RNASPLICE_RNASP rimargi R 33840 12 72G 1 13058929 nf-NFCORE_RNASPLICE_RNASP rimargi R 34624 6 36G 1 13058930 nf-NFCORE_RNASPLICE_RNASP rimargi R 34624 6 36G 1 13058932 nf-NFCORE_RNASPLICE_RNASP rimargi R 34624 6 36G 1 Among these, one of them failed and was retried with twice as much memory (72G). The two completed jobs must have been retried twice, requiring more than 72G of memory and are now completed. For example, from the log (.nextflow.log) I see: Jun-03 120027.621 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 13059186; id: 17; name: NFCORE_RNASPLICERNASPLICEBAM_SORT_STATS_SAMTOOLS:SAMTOOLS_SORT (PTres_SR_2_c8); status: COMPLETED; and running sacct -j 13059186 --format=JobID,JobName,AllocCPUS,State,Elapsed,MaxRSS I get JobID JobName AllocCPUS State Elapsed MaxRSS ------------ ---------- ---------- ---------- ---------- ---------- 13059186 nf-NFCORE+ 18 COMPLETED 000320 13059186.ba+ batch 18 COMPLETED 000320 81489072K 13059186.ex+ extern 18 COMPLETED 000320 956K So the subprocess 13059186.ba+ required ~81G of RAM, which is compatible with the two retries (allocating 36x3G of RAM). Now the questions are: 1. Is my interpretation correct? 2. These processes are idle on the compute nodes for a very long time, then they run in just 3 minutes. How could I improve this? 3. Why do the processes start with only 36G if they need much more RAM? Can I change the starting point so that I don't wait very long for each process to fail twice before being run with enough memory to complete? By the way, these bam files are already sorted and indexed, so these steps could be skipped, I think. Is there a way to do so? (though I would really like to have the entire pipeline running smoothly for future analyses).
t
Exit code 137 means SLURM killed the job, because it used more memory than it requested. The memory requirements for sorting are not fixed, but depends in the on the sample (more reads -> more RAM), so the pipeline has to guess what a reasonable amount to request would be. Clearly, that turned out not to be enough for your samples. To see how much RAM had been requested for a particular job, you can add 
ReqMem
to your 
sacct
query, or you can look at the 
#SBATCH --mem ...
line in the 
.command.run
file in the work directory of the process. Since the pipeline reruns on OOM errors (exit code 137), I see three options: ā€¢ Make 
samtools
use less RAM. You could maybe pass some option to it via 
ext.args
in a config file that makes it write to disk more and lkeep fewer records in RAM (I think that exists). This will likely slow down the process. ā€¢ Set 
process.maxRetries
in a config file to a number that enables enough retries, so you eventually get enough RAM. That seems inefficient. ā€¢ Set the memory request for this particular type of process manually. Note that this will not increase on reruns, as you are overriding the mechanism that facilitates that.
Copy code
process {
    withName: 'SAMTOOLS_SORT' {
        memory = '120 GB'  // adjust as needed
        maxRetries = 0     // or 1 at most
    }
}
Put this into a file and pass it to the pipeline with 
-c samtools_sort.config
. I hope this answers most of your questions. You say the BAM files are already sorted. If these are the same BAMs that are being precessed by the 
SAMTOOLS_SORT
process, then that does not make sense and should not happen. If you can verify that this is a bug, maybe start a new thread here to confirm that this is the case, and then open an issue on GitHub.
These processes are idle on the compute nodes for a very long time, then they run in just 3 minutes.
You mean there is no activity (how do you know?) after the job has started on a compute node? If so, I suspect you might be misinterpreting what is happening. Maybe you are looking maybe at files generated, etc., and those may be either in a 
/tmp
directory you are not checking, or the data is being sorted in RAM for a long time until it is being written to disk. I don't think there is much you can do, though, unless you can find a better way to run 
samtools sort
. (If you mean instead the jobs are waiting a long time to start on a node, that is likely a function of the load on the cluster. Not much you can do, but you can check with your sysadmin if you have concerns.)
r
Hi Thomas. Thank you very much again! Adding the memory requirement for SAMTOOLS_SORT solved the problem šŸ™
If you mean instead the jobs are waiting a long time to start on a node, that is likely a function of the load on the cluster.
Some of the jobs were waiting on the node in uninterruptible sleep. This didn't happen after adding the memory requirement.
You say the BAM files are already sorted. If these are the same BAMs that are being precessed by the 
SAMTOOLS_SORT
process, then that does not make sense and should not happen.
The BAMs are indeed sorted by coordinate, and I have the associated BAIs. The pipeline supposedly should test this before proceeding with the SAMTOOLS_SORT process (?). Here are the parameters that I am using to run the pipeline: { "input": "...\/samplesheet_PTResU_PTResSR_genome_bam.csv", "contrasts": "...\/PTResSRvsPTResU_contrast.csv", "source": "genome_bam", "outdir": "...\/PTResSRvsPTResU", "fasta": "...\/GRCh38.p14.genome.fa", "transcript_fasta": "...\/gencode.v44.transcripts.fa", "gtf": "...\/gencode.v44.annotation.gtf", "skip_trimgalore_fastqc": true, "skip_fastqc": true, "skip_alignment": true, "save_unaligned": false, "save_align_intermeds": false, "rmats": true, "rmats_novel_splice_site": false, "rmats_paired_stats": false, "rmats_read_len": 100, "dexseq_exon": false, "edger_exon":false, "dexseq_dtu": false, "sashimi_plot": false, "suppa": false } Side question: at present, I have 6 samples listed in both the samplesheet and contrast files. Is it possible to 1. have a comprehensive list of files listed in the samplesheet and make the pipeline consider as input only the files specified in the contrast file, and 2. run the pipeline with multiple contrasts at once?
t
Glad to hear the extra memory made it work! @James Ashmore, @Ben Southgate, do you guys have any insights into why sorted and indexed BAMs are being sorted (again)? Does it maybe need a different sort order? Regarding sample sheet, I think (someone correct me if I'm wrong) it will run all samples through the basic per-sample steps, but the contrast sheet determines which comparisons are made, i.e. the differential splicing analysis steps. But it is easy to subset a comprehensive sample sheet with 
awk
, 
grep
, etc., to make one just for the comparisons you are interested at this point: 
awk 'NR == 1 || $5 == "group1" || $5 == "group2"' FS=, OFS=, all_samples.csv > group1-2_samples.csv
(assuming 
conditon
is column 5 and the labels you want from there are 
group1
and 
group2
). I think you can have as many contrasts as are possible from the groups you have, I have certainly run the pipeline with five at a time.
Open in Slack
PreviousNext
Powered by