Hi All, I am trying to write an institutional con...
# configs
l
Hi All, I am trying to write an institutional config and have gotten to the point where I have pushed it to a branch of a fork of the configs repository for testing. I have decided to test it with a full RNA-Seq data set that I have successfully used previously with the same version of the RNA-Seq pipeline that I processed these data with initially (v14). I first tried with
-profile apptainer
i.e., not using the institutional config on the system in question and the pipeline ran to completion without issue (as can be seen in
rnaseqv14_apptainer_profile_nextflow.log
and
rnaseqv14_apptainer_profile.nextflow.log
attached). The run command was as follows:
Copy code
nextflow run nf-core/rnaseq \
-r 3.14.0 \
-profile apptainer \
-work-dir $WORK_DIR \
-params-file ${PARAM_DIR}/nf-params.json
I then tried with the custom config file (
shu_bmrc.config
) and am encountering this error:
Copy code
-[nf-core/rnaseq] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_RNASEQ:RNASEQ:FASTQ_FASTQC_UMITOOLS_TRIMGALORE:TRIMGALORE (4_CTR_BC_6)'

Caused by:
  Process `NFCORE_RNASEQ:RNASEQ:FASTQ_FASTQC_UMITOOLS_TRIMGALORE:TRIMGALORE (4_CTR_BC_6)` terminated for an unknown reason -- Likely it has been terminated by the external system


Command executed:

  [ ! -f  4_CTR_BC_6.fastq.gz ] && ln -s 4_CTR_BC_6.fastq.gz 4_CTR_BC_6.fastq.gz
  trim_galore \
      --fastqc_args '-t 12' \
      --cores 8 \
      --gzip \
      4_CTR_BC_6.fastq.gz
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_RNASEQ:RNASEQ:FASTQ_FASTQC_UMITOOLS_TRIMGALORE:TRIMGALORE":
      trimgalore: $(echo $(trim_galore --version 2>&1) | sed 's/^.*version //; s/Last.*$//')
      cutadapt: $(cutadapt --version)
  END_VERSIONS

Command exit status:
  -

Command output:
  (empty)

Work dir:
  /tmp/users/lq3274/nf_work/e4/5b7c85ea6b107af6a769dbc357d604

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

 -- Check '.nextflow.log' file for details
I have attached both the nextflow log (
rnaseqv14_institutional_profile_nextflow.log
) and the .nextflow.log (
rnaseqv14_institutional_profile.nextflow.log
) for reference. In this case the run command I used was:
Copy code
nextflow run nf-core/rnaseq \
-r 3.14.0 \
-profile shu_bmrc \
--custom_config_base "<https://raw.githubusercontent.com/lquayle88/configs/shu_bmrc>" \
-work-dir $WORK_DIR \
-params-file ${PARAM_DIR}/nf-params.json
In both cases the
nf-params.json
was identical - also uploaded for reference. Any ideas as to why this might be happening and what I need to do/include or remove from the config would be greatly appreciated. Cheers, Lewis
p
Assuming you use a scheduler (doing the work to see the actual profile is a bit much on mobile) this can happen sometimes, e.g. if the system is temporarily overloaded. You can check if you have jobs running still.
l
Hi Pontus, thanks for replying. Yes, I am using SLURM. I checked immediately after the failure and there were no jobs running. I was also the only person running jobs on the cluster at that time if that’s helpful.
p
Ok, next question then would be if the jobs were submitted ("accepted") properly.
sacct
should show them (at least at the time).
l
The output of
sacct
indicates to me that they were scheduled (assuming those with 12 cores requested are the trimgalore jobs which the log seems to imply)
Copy code
JobID           JobName  Partition    Account  AllocCPUS      State ExitCode 
------------ ---------- ---------- ---------- ---------- ---------- -------- 
8057               bash      nodes                     2  COMPLETED      0:0 
8057.0             bash                                2  COMPLETED      0:0 
8058               bash      nodes                     2  COMPLETED      0:0 
8058.0             bash                                2  COMPLETED      0:0 
8059               bash      nodes                     2  COMPLETED      0:0 
8059.0             bash                                2  COMPLETED      0:0 
8060         nfcore_rn+      nodes                     1     FAILED      1:0 
8060.batch        batch                                1     FAILED      1:0 
8061         nf-NFCORE+      nodes                     1  COMPLETED      0:0 
8061.batch        batch                                1  COMPLETED      0:0 
8062         nf-NFCORE+      nodes                     1  COMPLETED      0:0 
8062.batch        batch                                1  COMPLETED      0:0 
8063         nf-NFCORE+      nodes                     6  COMPLETED      0:0 
8063.batch        batch                                6  COMPLETED      0:0 
8064         nf-NFCORE+      nodes                     6  COMPLETED      0:0 
8064.batch        batch                                6  COMPLETED      0:0 
8065         nf-NFCORE+      nodes                     6  COMPLETED      0:0 
8065.batch        batch                                6  COMPLETED      0:0 
8066         nf-NFCORE+      nodes                     6  COMPLETED      0:0 
8066.batch        batch                                6  COMPLETED      0:0 
8067         nf-NFCORE+      nodes                     6  COMPLETED      0:0 
8067.batch        batch                                6  COMPLETED      0:0 
8068         nf-NFCORE+      nodes                     6  COMPLETED      0:0 
8068.batch        batch                                6  COMPLETED      0:0 
8069         nf-NFCORE+      nodes                     6  COMPLETED      0:0 
8069.batch        batch                                6  COMPLETED      0:0 
8070         nf-NFCORE+      nodes                     6  COMPLETED      0:0 
8070.batch        batch                                6  COMPLETED      0:0 
8071         nf-NFCORE+      nodes                     6  COMPLETED      0:0 
8071.batch        batch                                6  COMPLETED      0:0 
8072         nf-NFCORE+      nodes                    12     FAILED    127:0 
8072.batch        batch                               12     FAILED    127:0 
8073         nf-NFCORE+      nodes                    12     FAILED    127:0 
8073.batch        batch                               12     FAILED    127:0 
8074         nf-NFCORE+      nodes                    12     FAILED    127:0 
8074.batch        batch                               12     FAILED    127:0 
8075         nf-NFCORE+      nodes                    12     FAILED    127:0 
8075.batch        batch                               12     FAILED    127:0 
8076         nf-NFCORE+      nodes                    12     FAILED    127:0 
8076.batch        batch                               12     FAILED    127:0 
8077         nf-NFCORE+      nodes                    12     FAILED    127:0 
8077.batch        batch                               12     FAILED    127:0 
8078         nf-NFCORE+      nodes                    12     FAILED    127:0 
8078.batch        batch                               12     FAILED    127:0 
8079         nf-NFCORE+      nodes                    12     FAILED    127:0 
8079.batch        batch                               12     FAILED    127:0 
8080         nf-NFCORE+      nodes                    12     FAILED    127:0 
8080.batch        batch                               12     FAILED    127:0 
8081         nf-NFCORE+      nodes                     2  COMPLETED      0:0 
8081.batch        batch                                2  COMPLETED      0:0 
8082         nf-NFCORE+      nodes                    12     FAILED    127:0 
8082.batch        batch                               12     FAILED    127:0 
8083         nf-NFCORE+      nodes                    12     FAILED    127:0 
8083.batch        batch                               12     FAILED    127:0 
8084         nfcore_rn+      nodes                     1     FAILED      1:0 
8084.batch        batch                                1     FAILED      1:0 
8085         nf-NFCORE+      nodes                     1  COMPLETED      0:0 
8085.batch        batch                                1  COMPLETED      0:0 
8086         nf-NFCORE+      nodes                     1  COMPLETED      0:0 
8086.batch        batch                                1  COMPLETED      0:0 
8087         nf-NFCORE+      nodes                     6  COMPLETED      0:0 
8087.batch        batch                                6  COMPLETED      0:0 
8088         nf-NFCORE+      nodes                     6  COMPLETED      0:0 
8088.batch        batch                                6  COMPLETED      0:0 
8089         nf-NFCORE+      nodes                     6  COMPLETED      0:0 
8089.batch        batch                                6  COMPLETED      0:0 
8090         nf-NFCORE+      nodes                     6  COMPLETED      0:0 
8090.batch        batch                                6  COMPLETED      0:0 
8091         nf-NFCORE+      nodes                     6  COMPLETED      0:0 
8091.batch        batch                                6  COMPLETED      0:0 
8092         nf-NFCORE+      nodes                     6  COMPLETED      0:0 
8092.batch        batch                                6  COMPLETED      0:0 
8093         nf-NFCORE+      nodes                     6  COMPLETED      0:0 
8093.batch        batch                                6  COMPLETED      0:0 
8094         nf-NFCORE+      nodes                     6  COMPLETED      0:0 
8094.batch        batch                                6  COMPLETED      0:0 
8095         nf-NFCORE+      nodes                     6  COMPLETED      0:0 
8095.batch        batch                                6  COMPLETED      0:0 
8096         nf-NFCORE+      nodes                    12     FAILED    127:0 
8096.batch        batch                               12     FAILED    127:0 
8097         nf-NFCORE+      nodes                    12     FAILED    127:0 
8097.batch        batch                               12     FAILED    127:0 
8098         nf-NFCORE+      nodes                    12     FAILED    127:0 
8098.batch        batch                               12     FAILED    127:0 
8099         nf-NFCORE+      nodes                    12     FAILED    127:0 
8099.batch        batch                               12     FAILED    127:0 
8100         nf-NFCORE+      nodes                    12     FAILED    127:0 
8100.batch        batch                               12     FAILED    127:0 
8101         nf-NFCORE+      nodes                    12     FAILED    127:0 
8101.batch        batch                               12     FAILED    127:0 
8102         nf-NFCORE+      nodes                    12     FAILED    127:0 
8102.batch        batch                               12     FAILED    127:0 
8103         nf-NFCORE+      nodes                    12     FAILED    127:0 
8103.batch        batch                               12     FAILED    127:0 
8104         nf-NFCORE+      nodes                    12     FAILED    127:0 
8104.batch        batch                               12     FAILED    127:0 
8105         nf-NFCORE+      nodes                     2  COMPLETED      0:0 
8105.batch        batch                                2  COMPLETED      0:0 
8106         nf-NFCORE+      nodes                    12     FAILED    127:0 
8106.batch        batch                               12     FAILED    127:0 
8107         nf-NFCORE+      nodes                    12     FAILED    127:0 
8107.batch        batch                               12     FAILED    127:0
p
Ok, then the stderr file of those jobs would be the next step to check.
l
Apologies, how can I check these?
p
They will be in the task launch folder for nextflow but generally you can look at the job (e.g.
scontrol show job=8078
) and it will have it.
l
Running
scontrol show job=xxxx
for any of these jobs returns
slurm_load_jobs error: Invalid job id specified
. I could re-run to recapitulate the error if this is likely due to time elapsed since this morning. Alternatively I can look into the directory you indicated (where is this likely to be?)
p
They are inside the work directory. But now I see you've run under
/tmp
. Your cluster may be weird but normally
/tmp
isn't shared between nodes and can't be used when using the scheduler.
l
This cluster is a bit weird based on my past experience I agree (it’s an Alces Flight cluster) the
/tmp
dir seems to work fine when using the scheduler but can only be accessed when running an interactive session e.g., to cleanup. Not sure if this is relevant to this but as I said in my initial message, using
-profile apptainer
worked perfectly well (with the same work path etc). Either way, I have looked inside the work directory and for the job indicated in the
nextflow.log
there is no
.command.err
if that’s what I am looking for; these do exist for jobs that were successful.
p
Unless you have other configuration to tell nextflow to submit to the scheduler, it will be using the
local
executor, which is fine in node-local file systems (
local
...) The file missing totally is a strong indication it's a problem with the directory not being available (with the expected pre-loaded contents) on other nodes. Is there a filesystem you know is shared with the entire cluster?
l
Yes, I could try using the indicated shared scratch directory - I will change this in the run script and report back 👍
*indicated by Alces
Hi @pontus. I think this has solved this specific issue but I am not able to confirm that the pipeline run progresses to completion using the shared directory mentioned above as I am getting an error due to insufficient disk space (user disk allocation on this HPC is stingy to say the least). Thanks for your help on this so far - hopefully that will be the last you hear from me on this matter until I am ready to make the PR to the configs repo.