Lewis Quayle
10/13/2024, 7:58 AM-profile apptainer i.e., not using the institutional config on the system in question and the pipeline ran to completion without issue (as can be seen in rnaseqv14_apptainer_profile_nextflow.log and rnaseqv14_apptainer_profile.nextflow.log attached).
The run command was as follows:
nextflow run nf-core/rnaseq \
-r 3.14.0 \
-profile apptainer \
-work-dir $WORK_DIR \
-params-file ${PARAM_DIR}/nf-params.json
I then tried with the custom config file (shu_bmrc.config) and am encountering this error:
-[nf-core/rnaseq] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_RNASEQ:RNASEQ:FASTQ_FASTQC_UMITOOLS_TRIMGALORE:TRIMGALORE (4_CTR_BC_6)'
Caused by:
Process `NFCORE_RNASEQ:RNASEQ:FASTQ_FASTQC_UMITOOLS_TRIMGALORE:TRIMGALORE (4_CTR_BC_6)` terminated for an unknown reason -- Likely it has been terminated by the external system
Command executed:
[ ! -f 4_CTR_BC_6.fastq.gz ] && ln -s 4_CTR_BC_6.fastq.gz 4_CTR_BC_6.fastq.gz
trim_galore \
--fastqc_args '-t 12' \
--cores 8 \
--gzip \
4_CTR_BC_6.fastq.gz
cat <<-END_VERSIONS > versions.yml
"NFCORE_RNASEQ:RNASEQ:FASTQ_FASTQC_UMITOOLS_TRIMGALORE:TRIMGALORE":
trimgalore: $(echo $(trim_galore --version 2>&1) | sed 's/^.*version //; s/Last.*$//')
cutadapt: $(cutadapt --version)
END_VERSIONS
Command exit status:
-
Command output:
(empty)
Work dir:
/tmp/users/lq3274/nf_work/e4/5b7c85ea6b107af6a769dbc357d604
Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
-- Check '.nextflow.log' file for details
I have attached both the nextflow log (rnaseqv14_institutional_profile_nextflow.log) and the .nextflow.log (rnaseqv14_institutional_profile.nextflow.log) for reference.
In this case the run command I used was:
nextflow run nf-core/rnaseq \
-r 3.14.0 \
-profile shu_bmrc \
--custom_config_base "<https://raw.githubusercontent.com/lquayle88/configs/shu_bmrc>" \
-work-dir $WORK_DIR \
-params-file ${PARAM_DIR}/nf-params.json
In both cases the nf-params.json was identical - also uploaded for reference.
Any ideas as to why this might be happening and what I need to do/include or remove from the config would be greatly appreciated.
Cheers,
Lewispontus
10/13/2024, 10:04 AMLewis Quayle
10/13/2024, 10:17 AMpontus
10/13/2024, 10:18 AMsacct should show them (at least at the time).Lewis Quayle
10/13/2024, 10:59 AMsacct indicates to me that they were scheduled (assuming those with 12 cores requested are the trimgalore jobs which the log seems to imply)
JobID JobName Partition Account AllocCPUS State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
8057 bash nodes 2 COMPLETED 0:0
8057.0 bash 2 COMPLETED 0:0
8058 bash nodes 2 COMPLETED 0:0
8058.0 bash 2 COMPLETED 0:0
8059 bash nodes 2 COMPLETED 0:0
8059.0 bash 2 COMPLETED 0:0
8060 nfcore_rn+ nodes 1 FAILED 1:0
8060.batch batch 1 FAILED 1:0
8061 nf-NFCORE+ nodes 1 COMPLETED 0:0
8061.batch batch 1 COMPLETED 0:0
8062 nf-NFCORE+ nodes 1 COMPLETED 0:0
8062.batch batch 1 COMPLETED 0:0
8063 nf-NFCORE+ nodes 6 COMPLETED 0:0
8063.batch batch 6 COMPLETED 0:0
8064 nf-NFCORE+ nodes 6 COMPLETED 0:0
8064.batch batch 6 COMPLETED 0:0
8065 nf-NFCORE+ nodes 6 COMPLETED 0:0
8065.batch batch 6 COMPLETED 0:0
8066 nf-NFCORE+ nodes 6 COMPLETED 0:0
8066.batch batch 6 COMPLETED 0:0
8067 nf-NFCORE+ nodes 6 COMPLETED 0:0
8067.batch batch 6 COMPLETED 0:0
8068 nf-NFCORE+ nodes 6 COMPLETED 0:0
8068.batch batch 6 COMPLETED 0:0
8069 nf-NFCORE+ nodes 6 COMPLETED 0:0
8069.batch batch 6 COMPLETED 0:0
8070 nf-NFCORE+ nodes 6 COMPLETED 0:0
8070.batch batch 6 COMPLETED 0:0
8071 nf-NFCORE+ nodes 6 COMPLETED 0:0
8071.batch batch 6 COMPLETED 0:0
8072 nf-NFCORE+ nodes 12 FAILED 127:0
8072.batch batch 12 FAILED 127:0
8073 nf-NFCORE+ nodes 12 FAILED 127:0
8073.batch batch 12 FAILED 127:0
8074 nf-NFCORE+ nodes 12 FAILED 127:0
8074.batch batch 12 FAILED 127:0
8075 nf-NFCORE+ nodes 12 FAILED 127:0
8075.batch batch 12 FAILED 127:0
8076 nf-NFCORE+ nodes 12 FAILED 127:0
8076.batch batch 12 FAILED 127:0
8077 nf-NFCORE+ nodes 12 FAILED 127:0
8077.batch batch 12 FAILED 127:0
8078 nf-NFCORE+ nodes 12 FAILED 127:0
8078.batch batch 12 FAILED 127:0
8079 nf-NFCORE+ nodes 12 FAILED 127:0
8079.batch batch 12 FAILED 127:0
8080 nf-NFCORE+ nodes 12 FAILED 127:0
8080.batch batch 12 FAILED 127:0
8081 nf-NFCORE+ nodes 2 COMPLETED 0:0
8081.batch batch 2 COMPLETED 0:0
8082 nf-NFCORE+ nodes 12 FAILED 127:0
8082.batch batch 12 FAILED 127:0
8083 nf-NFCORE+ nodes 12 FAILED 127:0
8083.batch batch 12 FAILED 127:0
8084 nfcore_rn+ nodes 1 FAILED 1:0
8084.batch batch 1 FAILED 1:0
8085 nf-NFCORE+ nodes 1 COMPLETED 0:0
8085.batch batch 1 COMPLETED 0:0
8086 nf-NFCORE+ nodes 1 COMPLETED 0:0
8086.batch batch 1 COMPLETED 0:0
8087 nf-NFCORE+ nodes 6 COMPLETED 0:0
8087.batch batch 6 COMPLETED 0:0
8088 nf-NFCORE+ nodes 6 COMPLETED 0:0
8088.batch batch 6 COMPLETED 0:0
8089 nf-NFCORE+ nodes 6 COMPLETED 0:0
8089.batch batch 6 COMPLETED 0:0
8090 nf-NFCORE+ nodes 6 COMPLETED 0:0
8090.batch batch 6 COMPLETED 0:0
8091 nf-NFCORE+ nodes 6 COMPLETED 0:0
8091.batch batch 6 COMPLETED 0:0
8092 nf-NFCORE+ nodes 6 COMPLETED 0:0
8092.batch batch 6 COMPLETED 0:0
8093 nf-NFCORE+ nodes 6 COMPLETED 0:0
8093.batch batch 6 COMPLETED 0:0
8094 nf-NFCORE+ nodes 6 COMPLETED 0:0
8094.batch batch 6 COMPLETED 0:0
8095 nf-NFCORE+ nodes 6 COMPLETED 0:0
8095.batch batch 6 COMPLETED 0:0
8096 nf-NFCORE+ nodes 12 FAILED 127:0
8096.batch batch 12 FAILED 127:0
8097 nf-NFCORE+ nodes 12 FAILED 127:0
8097.batch batch 12 FAILED 127:0
8098 nf-NFCORE+ nodes 12 FAILED 127:0
8098.batch batch 12 FAILED 127:0
8099 nf-NFCORE+ nodes 12 FAILED 127:0
8099.batch batch 12 FAILED 127:0
8100 nf-NFCORE+ nodes 12 FAILED 127:0
8100.batch batch 12 FAILED 127:0
8101 nf-NFCORE+ nodes 12 FAILED 127:0
8101.batch batch 12 FAILED 127:0
8102 nf-NFCORE+ nodes 12 FAILED 127:0
8102.batch batch 12 FAILED 127:0
8103 nf-NFCORE+ nodes 12 FAILED 127:0
8103.batch batch 12 FAILED 127:0
8104 nf-NFCORE+ nodes 12 FAILED 127:0
8104.batch batch 12 FAILED 127:0
8105 nf-NFCORE+ nodes 2 COMPLETED 0:0
8105.batch batch 2 COMPLETED 0:0
8106 nf-NFCORE+ nodes 12 FAILED 127:0
8106.batch batch 12 FAILED 127:0
8107 nf-NFCORE+ nodes 12 FAILED 127:0
8107.batch batch 12 FAILED 127:0pontus
10/13/2024, 11:12 AMLewis Quayle
10/13/2024, 11:27 AMpontus
10/13/2024, 11:52 AMscontrol show job=8078) and it will have it.Lewis Quayle
10/13/2024, 11:56 AMscontrol show job=xxxx for any of these jobs returns slurm_load_jobs error: Invalid job id specified . I could re-run to recapitulate the error if this is likely due to time elapsed since this morning. Alternatively I can look into the directory you indicated (where is this likely to be?)pontus
10/13/2024, 11:59 AM/tmp. Your cluster may be weird but normally /tmp isn't shared between nodes and can't be used when using the scheduler.Lewis Quayle
10/13/2024, 12:27 PM/tmp dir seems to work fine when using the scheduler but can only be accessed when running an interactive session e.g., to cleanup. Not sure if this is relevant to this but as I said in my initial message, using -profile apptainer worked perfectly well (with the same work path etc). Either way, I have looked inside the work directory and for the job indicated in the nextflow.log there is no .command.err if that’s what I am looking for; these do exist for jobs that were successful.pontus
10/13/2024, 12:30 PMlocal executor, which is fine in node-local file systems (local...)
The file missing totally is a strong indication it's a problem with the directory not being available (with the expected pre-loaded contents) on other nodes. Is there a filesystem you know is shared with the entire cluster?Lewis Quayle
10/13/2024, 12:32 PMLewis Quayle
10/13/2024, 12:32 PMLewis Quayle
10/13/2024, 1:57 PM