https://nf-co.re logo
Join Slack
Powered by
I'm trying to run rnasplice on a set of samples th...
# rnasplice
b
I'm trying to run rnasplice on a set of samples that I've already successfully run through the rnaseq pipeline, and I'm specifying the same gtf and fasta file for both pipelines. When I run rnasplice, I get
Copy code
ERROR ~ Error executing process > 'NFCORE_RNASPLICE:RNASPLICE:DEXSEQ_DEU:DEXSEQ_ANNOTATION (genes.gtf)'

Caused by:
  Process `NFCORE_RNASPLICE:RNASPLICE:DEXSEQ_DEU:DEXSEQ_ANNOTATION (genes.gtf)` terminated with an error exit status (1)

Command executed:

  dexseq_prepare_annotation.py genes.gtf DEXSeq.gff

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_RNASPLICE:RNASPLICE:DEXSEQ_DEU:DEXSEQ_ANNOTATION":
      htseq: $(python -c "import pkg_resources; print(pkg_resources.get_distribution('htseq').version)")
  END_VERSIONS

Command exit status:
  1

Command output:
  (empty)

Command error:
  WARNING: While bind mounting '/vf:/vf': destination is already in the mount point list
  Traceback (most recent call last):
    File "/home/capaldobj/.nextflow/23.10.0/assets/nf-core/rnasplice/bin/dexseq_prepare_annotation.py", line 130, in <module>
      raise ValueError("Same name found on two chromosomes: %s, %s" % (str(l[i]), str(l[i + 1])))
  ValueError: Same name found on two chromosomes: <GenomicFeature: exonic_part 'MIR1302-11+MIR1302-10+MIR1302-2+MIR1302-9' at chr1: 30365 -> 30503 (strand '+')>, <GenomicFeature: exonic_part 'MIR1302-11+MIR1302-10+MIR1302-2+MIR1302-9' at chr15: 101960595 -> 101960457 (strand '-')>

Work dir:
  /vf/users/RUNX1/project/18. Menezes_RNAseq/work/a4/9657e159251e8bd3eed9546ce9135b

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

 -- Check '.nextflow.log' file for details
Here's my shell script
Copy code
#! /bin/bash
#SBATCH --cpus-per-task=4
#SBATCH --mem=16g
#SBATCH --time=96:00:00
#SBATCH --mail-type=BEGIN,TIME_LIMIT_90,END
#SBATCH --mail-user=capaldobj@nih.gov

module purge
module load nextflow
module load singularity
module load graphviz

nextflow run nf-core/rnasplice \
--input 'sample-input.csv' \
--contrasts 'contrasts.csv' \
-profile biowulf \
--gtf /fdb/STAR_current/UCSC/hg38/genes.gtf \
--fasta /fdb/STAR_current/UCSC/hg38/ref.fa \
--outdir 'human-samples-hg19-rna-seq-splice-results'
and my input
Copy code
sample,fastq_1,fastq_2,strandedness,condition
Ctrl1_1,/data/RUNX1/project/18. Menezes_RNAseq//Ctrl1_1_S21_L001_R1_001.fastq.gz,/data/RUNX1/project/18. Menezes_RNAseq//Ctrl1_1_S21_L001_R2_001.fastq.gz,reverse,CONTROL
Ctrl2,/data/RUNX1/project/18. Menezes_RNAseq//Ctrl2_S22_L001_R1_001.fastq.gz,/data/RUNX1/project/18. Menezes_RNAseq//Ctrl2_S22_L001_R2_001.fastq.gz,reverse,CONTROL
FDP3,/data/RUNX1/project/18. Menezes_RNAseq//FDP3_S23_L001_R1_001.fastq.gz,/data/RUNX1/project/18. Menezes_RNAseq//FDP3_S23_L001_R2_001.fastq.gz,reverse,MUTANT
FDP4,/data/RUNX1/project/18. Menezes_RNAseq//FDP4_S24_L001_R1_001.fastq.gz,/data/RUNX1/project/18. Menezes_RNAseq//FDP4_S24_L001_R2_001.fastq.gz,reverse,MUTANT
t
The error message is 
ValueError: Same name found on two chromosomes
, which to me sounds like the same gene ID was used more than once. If gene names are use as IDs, that is a problem, because they are not always unique. Can you use a different GTF (e.g. from Ensembl)? If not, if you look at the GTF you have been using, do the 
gene_id
entries have gene names? If so, you would have to find a way of disambiguating them. (If not, i.e. there are proper IDs, then I'm confused where the 
MIR1302-11+MIR1302-10+MIR1302-2+MIR1302-9
comes from ...)
b
Yeah, I was able to get the pipeline to run using GRCh37, but really need it to run on hg38. Using GRCh38, I get a different error (code 137), so I'm still trying to figure that one out. I'm surprised that the reference from UCSC has this sort of issue.
t
Exit code 
137
is usually associated with "killed due to running out of memory" (at least on our SLURM cluster). There are two ways I can think of to address that: • The pipeline can rerun failed jobs and by default increase resources with each attempt (double, triple, etc., up to the maximum that your system allows). By default, 1 rerun is permitted (i.e. a doubling of resources). You can increase that by setting 
process.maxRetries
in a config file. By default, exit code 
137
is in the "retry" range, but you can modify that as well via 
process.errorStrategy
. • You can set a specific amount of memory for a type of process with
Copy code
process {
    withName: 'SOME_PROCESS' {  // replace SOME_PROCESS with the name of the process (last part after ":")
        memeory = XX.GB  // replace XX with a number
    }
}
in a config file. Note that such a setting would override the automatic increase mentioned earlier, so retrying would not help (you could explicitly forbid it with 
maxRetries = 0
).
b
Yeah, just surprised 38 failed when 37 succeeded, I didn't think 38 would be that more intensive. I'll keep plugging away
🤷 1
🤞 1
brief update, the job dies pretty much instantly, so that's weird
Ah, so the BAM file isn't in the work directory
🤔 1
I reran the pipeline from scratch and still have the same issue. the bam file isn't showing up
t
Which process are we talking about? And what is actually in that work directory? Do you get any error messages?
b
So, the alignment completes, but the samtools sort command fails because the aligned bam file isn't being symlinked into the work directory for samtools sort.
Copy code
22      8f/045ddf       27657364        NFCORE_RNASPLICE:RNASPLICE:FASTQ_FASTQC_UMITOOLS_TRIMGALORE:FASTQC (FDP4)       CACHED  0       2024-05-30 10:29:50.576 11m     8m      110.4%  1.6 GB       6.1 GB  5.8 GB  4 MB
19      4d/91ed4b       27657347        NFCORE_RNASPLICE:RNASPLICE:FASTQ_FASTQC_UMITOOLS_TRIMGALORE:TRIMGALORE (FDP4)   CACHED  0       2024-05-30 10:29:30.584 25m 20s 21m 37s 543.2%  6 GB
7.3 GB  178.9 GB        173 GB
33      0e/c4bc0a       27674166        NFCORE_RNASPLICE:RNASPLICE:SALMON_QUANT_SALMON (FDP4)   COMPLETED       0       2024-05-30 12:30:41.902 47m 57s 45m 31s 572.5%  17.4 GB 21.1 GB 21.2 GB      45.7 MB
29      bd/8f4f7b       27668245        NFCORE_RNASPLICE:RNASPLICE:ALIGN_STAR:STAR_ALIGN_IGENOMES (FDP4)        COMPLETED       0       2024-05-30 11:52:41.349 1h 35m 57s      1h 33m 16s  1057.6%  36.1 GB 37.9 GB 82.3 GB 13.7 GB
65      3c/b0abb5       27682326        NFCORE_RNASPLICE:RNASPLICE:ALIGN_STAR:BAM_SORT_STATS_SAMTOOLS:SAMTOOLS_SORT (FDP4)      FAILED  137     2024-05-30 13:28:41.933 3m 57s  2m      -   --       -       -
66      bd/52dda7       27682338        NFCORE_RNASPLICE:RNASPLICE:SALMON_QUANT_STAR (FDP4)     COMPLETED       0       2024-05-30 13:28:49.647 5m 51s  2m 49s  494.6%  4.3 GB  5 GB    4.1 GB       45.9 MB
70      b8/46c75f       27682584        NFCORE_RNASPLICE:RNASPLICE:ALIGN_STAR:BAM_SORT_STATS_SAMTOOLS:SAMTOOLS_SORT (FDP4)      FAILED  137     2024-05-30 13:32:40.447 5m 58s  2m      -   --       -       -
that's the execution trace for the sample that isn't getting to the samtools_sort work directory
t
Interesting. What is actually in the 
b8/46c75f...
directory? And does this just occur for this particular sample, or all of them?
b
Copy code
b29f1320385227e114aa0cd2e08f86]$ ls -lhta
total 5.1G
drwxrws---. 2 capaldobj RUNX1 4.0K May 30 13:38 ..
-rw-rw----. 1 capaldobj RUNX1    1 May 30 12:47 .exitcode
drwxrws---. 2 capaldobj RUNX1 4.0K May 30 12:47 .
-rw-rw----. 1 capaldobj RUNX1   99 May 30 12:47 versions.yml
-rw-rw----. 1 capaldobj RUNX1 5.1G May 30 12:47 FDP3_sorted.bam
lrwxrwxrwx. 1 capaldobj RUNX1  102 May 30 12:47 FDP3.Aligned.out.bam -> '/vf/users/RUNX1/project/18. Menezes_RNAseq/work/de/782d2ff27a78b3b16708086413dd64/FDP3.Aligned.out.bam'
-rw-rw----. 1 capaldobj RUNX1  117 May 30 12:47 .command.log
-rw-rw----. 1 capaldobj RUNX1  299 May 30 12:47 .command.trace
-rw-rw----. 1 capaldobj RUNX1   64 May 30 12:47 .command.err
-rw-rw----. 1 capaldobj RUNX1    0 May 30 12:47 .command.out
-rw-rw----. 1 capaldobj RUNX1    0 May 30 12:43 .command.begin
-rw-rw----. 1 capaldobj RUNX1  363 May 30 12:42 .command.sh
-rw-rw----. 1 capaldobj RUNX1  13K May 30 12:42 .command.run
so that's the contents of the replicate work directory for the same process
and here's the failed sample work directory
Copy code
ls -lhta
total 15K
-rw-rw----. 1 capaldobj RUNX1    3 May 30 13:38 .exitcode
drwxrws---. 2 capaldobj RUNX1 4.0K May 30 13:38 .
-rw-rw----. 1 capaldobj RUNX1    0 May 30 13:38 .command.trace
-rw-rw----. 1 capaldobj RUNX1  219 May 30 13:38 .command.err
-rw-rw----. 1 capaldobj RUNX1    0 May 30 13:38 .command.out
-rw-rw----. 1 capaldobj RUNX1  219 May 30 13:38 .command.log
-rw-rw----. 1 capaldobj RUNX1    0 May 30 13:35 .command.begin
-rw-rw----. 1 capaldobj RUNX1  363 May 30 13:32 .command.sh
-rw-rw----. 1 capaldobj RUNX1  13K May 30 13:32 .command.run
drwxrws---. 2 capaldobj RUNX1 4.0K May 30 13:32 ..
t
Weird. Can you verify that the BAM that should be linked into this directory actually exists? I have had weird stuff happen (in other pipelines) where some jobs silently (exit code 0) failed and produced invalid (empty/truncated) files, which eventually caused errors in downstream processes.
b
Already did
Copy code
ls -lhta
total 14G
drwxrws---. 2 capaldobj RUNX1 4.0K May 30 13:28 ..
-rw-rw----. 1 capaldobj RUNX1    1 May 30 13:28 .exitcode
drwxrws---. 2 capaldobj RUNX1 4.0K May 30 13:28 .
-rw-rw----. 1 capaldobj RUNX1  114 May 30 13:28 versions.yml
-rw-rw----. 1 capaldobj RUNX1 7.3M May 30 13:28 FDP4.SJ.out.tab
-rw-rw----. 1 capaldobj RUNX1 8.8K May 30 13:28 FDP4.Log.progress.out
-rw-rw----. 1 capaldobj RUNX1  19M May 30 13:28 FDP4.Log.out
-rw-rw----. 1 capaldobj RUNX1 1.9K May 30 13:28 FDP4.Log.final.out
-rw-rw----. 1 capaldobj RUNX1 3.6G May 30 13:28 FDP4.Aligned.toTranscriptome.out.bam
-rw-rw----. 1 capaldobj RUNX1  11G May 30 13:28 FDP4.Aligned.out.bam
-rw-rw----. 1 capaldobj RUNX1  838 May 30 13:27 .command.log
-rw-rw----. 1 capaldobj RUNX1  313 May 30 13:27 .command.trace
-rw-rw----. 1 capaldobj RUNX1   87 May 30 13:27 .command.err
-rw-rw----. 1 capaldobj RUNX1  434 May 30 13:27 .command.out
-rw-rw----. 1 capaldobj RUNX1    0 May 30 11:53 .command.begin
-rw-rw----. 1 capaldobj RUNX1 1.1K May 30 11:52 .command.sh
-rw-rw----. 1 capaldobj RUNX1  13K May 30 11:52 .command.run
t
I don't know why Nextflow would not stage that one particular file ... Check that you are not out of disk space there. I'm out of ideas, sorry.
b
No worries, I just created my own symlink, we'll see if that fixes it. Still no idea why it wouldn't stage that file
well, never mind, it doesn't reuse that work directory, it creates a new one
t
Yeah, that's what I thought. I wonder if there i a way to run it manually and trick Nextflow into using the results in 
-resume
, but I'm not holding my breath.
b
I doubt it. I imagine the same thing will happen. I've been looking around for issues related to staging, but not getting much, and I've actually had this issue before with other pipelines, I wonder if this is a bigger issue than just rnasplice
t
I would assume so -- staging is fundamental to any Nextflow pipeline. If there is something in your system that messes with it, you'd see that everywhere. You could try a different Nextflow version to see if that makes a difference, but I doubt it.
7 Views
Open in Slack
PreviousNext
Powered by