https://nf-co.re logo
Join Slack
Powered by
Hi, I'm trying to run the sarek pipeline but runni...
# sarek
m
Hi, I'm trying to run the sarek pipeline but running into issues. A test run using 
-profile test,singularity
works fine, but as soon as I add our cluster scheduler as 
-profile test,singularity,sge
the pipeline fails. I noted that that the 
configFiles
option remains empty when starting the pipeline, although details in the log file suggests that my 
nextflow.config
file in my home directory appears to be found and parsed. I still don't think its working correctly though since jobs are failing when 
sge
is included in the profile option. My 
nextflow.config
file has important options that our cluster needs for jobs to run properly, which I don't see added in the resulting 
.command.run
files. I have had no such problems running several other nf-core pipelines. Can someone please help me figure out what the problem is?
m
can you share your 
nextflow.config
?
👍 1
m
Copy code
singularity.autoMounts = true
singularity.runOptions = '--no-mount tmp -B /tmp -B /ebio'
singularity.cacheDir = '/tmp/global2/mborg/.nextflow/singularity_cache'
singularity.enabled = true
profiles {
         sge {
             process.executor = 'sge'
             process.penv = 'parallel'
             process.maxForks = 16
             process.maxRetries = 3
             process.errorStrategy = 'retry'
             process.clusterOptions = { "-l h_vmem=${(task.attempt*task.memory/(0.75*task.cpus)).toString().replaceAll(/[\sB]/,'')}" }
         }
}
m
can you try:
Copy code
singularity.autoMounts = true
singularity.runOptions = '--no-mount tmp -B /tmp -B /ebio'
singularity.cacheDir = '/tmp/global2/mborg/.nextflow/singularity_cache'
singularity.enabled = true
process.executor = 'sge'
process.penv = 'parallel'
process.maxForks = 16
process.maxRetries = 3
process.errorStrategy = 'retry'
process.clusterOptions = { "-l h_vmem=${(task.attempt*task.memory/(0.75*task.cpus)).toString().replaceAll(/[\sB]/,'')}" }
for your 
nextflow.config
instead, and launch with just 
-profile test
m
the test run appears to be proceeding now... although there are failures in the initial runs, the retries appear to get processed...
Copy code
[7e/1ac2c3] process > NFCORE_SAREK:SAREK:FASTQC (test-test_L1)                                                         [100%] 4 of 4, failed: 2...
[18/9f79b3] process > NFCORE_SAREK:SAREK:FASTQ_ALIGN_BWAMEM_MEM2_DRAGMAP_SENTIEON:BWAMEM1_MEM (test-test_L2)           [100%] 2 of 2 ✔
[-        ] process > NFCORE_SAREK:SAREK:FASTQ_ALIGN_BWAMEM_MEM2_DRAGMAP_SENTIEON:BWAMEM2_MEM                          -
[-        ] process > NFCORE_SAREK:SAREK:FASTQ_ALIGN_BWAMEM_MEM2_DRAGMAP_SENTIEON:DRAGMAP_ALIGN                        -
[-        ] process > NFCORE_SAREK:SAREK:FASTQ_ALIGN_BWAMEM_MEM2_DRAGMAP_SENTIEON:SENTIEON_BWAMEM                      -
[8f/86b89d] process > NFCORE_SAREK:SAREK:BAM_MARKDUPLICATES:GATK4_MARKDUPLICATES (test)                                [100%] 2 of 2, failed: 1...
[8a/86d0d7] process > NFCORE_SAREK:SAREK:BAM_MARKDUPLICATES:CRAM_QC_MOSDEPTH_SAMTOOLS:SAMTOOLS_STATS (test)            [  0%] 0 of 1
[4a/7d85a8] process > NFCORE_SAREK:SAREK:BAM_MARKDUPLICATES:CRAM_QC_MOSDEPTH_SAMTOOLS:MOSDEPTH (test)                  [  0%] 0 of 1
[-        ] process > NFCORE_SAREK:SAREK:CRAM_TO_BAM                                                                   -
[f9/ccff3b] process > NFCORE_SAREK:SAREK:BAM_BASERECALIBRATOR:GATK4_BASERECALIBRATOR (test)                            [  0%] 0 of 1
[-        ] process > NFCORE_SAREK:SAREK:BAM_BASERECALIBRATOR:GATK4_GATHERBQSRREPORTS                                  -
[-        ] process > NFCORE_SAREK:SAREK:BAM_APPLYBQSR:GATK4_APPLYBQSR                                                 -
[-        ] process > NFCORE_SAREK:SAREK:BAM_APPLYBQSR:CRAM_MERGE_INDEX_SAMTOOLS:MERGE_CRAM                            -
[-        ] process > NFCORE_SAREK:SAREK:BAM_APPLYBQSR:CRAM_MERGE_INDEX_SAMTOOLS:INDEX_CRAM                            -
[-        ] process > NFCORE_SAREK:SAREK:CRAM_TO_BAM_RECAL                                                             -
[-        ] process > NFCORE_SAREK:SAREK:CRAM_SAMPLEQC:CRAM_QC_RECAL:SAMTOOLS_STATS                                    -
[-        ] process > NFCORE_SAREK:SAREK:CRAM_SAMPLEQC:CRAM_QC_RECAL:MOSDEPTH                                          -
[-        ] process > NFCORE_SAREK:SAREK:CRAM_SAMPLEQC:BAM_NGSCHECKMATE:BCFTOOLS_MPILEUP                               -
[-        ] process > NFCORE_SAREK:SAREK:CRAM_SAMPLEQC:BAM_NGSCHECKMATE:NGSCHECKMATE_NCM                               -
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_GERMLINE_ALL:BAM_VARIANT_CALLING_SINGLE_STRELKA:STRELK... -
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_GERMLINE_ALL:BAM_VARIANT_CALLING_SINGLE_STRELKA:MERGE_... -
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_GERMLINE_ALL:BAM_VARIANT_CALLING_SINGLE_STRELKA:MERGE_... -
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_TUMOR_ONLY_ALL:BAM_VARIANT_CALLING_SINGLE_STRELKA:STRE... -
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_TUMOR_ONLY_ALL:BAM_VARIANT_CALLING_SINGLE_STRELKA:MERG... -
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_TUMOR_ONLY_ALL:BAM_VARIANT_CALLING_SINGLE_STRELKA:MERG... -
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_SOMATIC_STRELKA:STRELK... -
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_SOMATIC_STRELKA:MERGE_... -
[-        ] process > NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_SOMATIC_STRELKA:MERGE_... -
[-        ] process > NFCORE_SAREK:SAREK:VCF_QC_BCFTOOLS_VCFTOOLS:BCFTOOLS_STATS                                       -
[-        ] process > NFCORE_SAREK:SAREK:VCF_QC_BCFTOOLS_VCFTOOLS:VCFTOOLS_TSTV_COUNT                                  -
[-        ] process > NFCORE_SAREK:SAREK:VCF_QC_BCFTOOLS_VCFTOOLS:VCFTOOLS_TSTV_QUAL                                   -
[-        ] process > NFCORE_SAREK:SAREK:VCF_QC_BCFTOOLS_VCFTOOLS:VCFTOOLS_SUMMARY                                     -
[-        ] process > NFCORE_SAREK:SAREK:MULTIQC                                                                       -
should I change my config file to what you suggested? thanks for your help btw!
m
You can
You don't need to use a profile in your own config
and you don't need the singularity profile as you're providing that within your config
m
thanks so much
2 Views
Open in Slack
PreviousNext
Powered by