I fixed the above error, but now I’m running into seemingly unrelated errors. I apologize for the long error messages here; I want to make sure I’m reporting all the relevant information. If there is any way to make this more compact, please let me know. I run the pipeline on an HPC, and the function call in the job script is as follows:

export NXF_APPTAINER_CACHEDIR=/users/nvincen2/.apptainer

# Run the pipeline
nextflow run nf-core/ampliseq \
-profile apptainer \
-r 2.15.0 \
-params-file ampliseq-nf-params.yml \

I am running 96 samples, and I’m passing the metadata sheet as a parameter in the .yml file. I got the following error that halted the pipeline:

Plus 31 more processes waiting for tasks…
Use DADA2 taxonomy classification
ERROR ~ Invalid FASTQ format for file: /temp180/mpfrende/nvincen2/Metagenomics/02_cutadapt/demux_out/01c_rescue/mICOintF/88_S88_L001_mICOintF_R1.fastq.gz

 -- Check script '/users/nvincen2/.nextflow/assets/nf-core/ampliseq/workflows/ampliseq.nf' at line: 310 or see '.nextflow.log' file for more details

I tried to check the fastq formatting of this file, but did not find anything wrong with it. I tried excluding this file and running again, and it spit out the same error with a different file. This happened 3 times. When I excluded those files, the pipeline continued past this point without erroring. I ran fastqc on these three files and looked at the multiqc report, but everything looked totally normal. I even tried unzipping and re-zipping them. Do you have any suggestions on how to look for what’s wrong with these files? I can upload the multiqc report here if it would be useful. The pipeline continued running after those files were removed. The next point at which it stopped, it returned this error:

-[nf-core/ampliseq] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_AMPLISEQ:AMPLISEQ:QIIME2_SEQFILTERTABLE (rep-seqs.qza-filter-by-filtered-table.qza)'

Caused by:
 Process `NFCORE_AMPLISEQ:AMPLISEQ:QIIME2_SEQFILTERTABLE (rep-seqs.qza-filter-by-filtered-table.qza)` terminated with an error exit status (1)


Command executed:

 export XDG_CONFIG_HOME="./xdgconfig"
 export MPLCONFIGDIR="./mplconfigdir"
 export NUMBA_CACHE_DIR="./numbacache"

 qiime feature-table filter-seqs \
   --i-data rep-seqs.qza \
   --i-table filtered-table.qza \
   --o-filtered-data filtered-sequences.qza

 cat <<-END_VERSIONS > versions.yml
 "NFCORE_AMPLISEQ:AMPLISEQ:QIIME2_SEQFILTERTABLE":
   qiime2: $( qiime --version | sed '1!d;s/.* //' )
 END_VERSIONS

Command exit status:
 1

Command output:
 (empty)

Command error:
 QIIME is caching your current deployment for improved performance. This may take a few moments and should only happen once per deployment.
 Plugin error from feature-table:

  cannot copy tree '/tmp/2678907.1.largemem/qiime2/nvincen2/processes/41-1767738247.48@nvincen2/tmp/qiime2-provenance-12x_iw3u': not a directory
Debug info has been saved to /tmp/2678907.1.largemem/qiime2-q2cli-err-b8da8p0q.log

Work dir:
 /temp180/mpfrende/nvincen2/Metagenomics/nf-core_ampliseq/nfcore_scripts/work/8e/405830921bf9afed9ebed75851cf56

Container:
 /users/nvincen2/.apptainer/quay.io-qiime2-amplicon-2024.10.img

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

 -- Check '.nextflow.log' file for details
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: <https://nf-co.re/docs/usage/troubleshooting>

 -- Check '.nextflow.log' file for details

I thought it might be a memory issue, so I requested more cores for the job and re-submitted the same job. However, the number of cores I was requesting earlier should have been plenty - I believe it was about 600 GB. I increased it to ~800 GB. This time the error returned was:

-[nf-core/ampliseq] Pipeline completed with errors-
WARN: [nf-core/ampliseq] No reports found from process 'MULTIQC'
ERROR ~ Error executing process > 'NFCORE_AMPLISEQ:AMPLISEQ:QIIME2_EXPORT:QIIME2_EXPORT_RELASV (1)'

Caused by:
 Process `NFCORE_AMPLISEQ:AMPLISEQ:QIIME2_EXPORT:QIIME2_EXPORT_RELASV (1)` terminated with an error exit status (1)


Command executed:

 export XDG_CONFIG_HOME="./xdgconfig"
 export MPLCONFIGDIR="./mplconfigdir"
 export NUMBA_CACHE_DIR="./numbacache"

 #convert to relative abundances
 qiime feature-table relative-frequency \
   --i-table filtered-table.qza \
   --o-relative-frequency-table relative-table-ASV.qza

 #export to biom
 qiime tools export \
   --input-path relative-table-ASV.qza \
   --output-path relative-table-ASV

 #convert to tab separated text file "rel-table-ASV.tsv"
 biom convert \
   -i relative-table-ASV/feature-table.biom \
   -o rel-table-ASV.tsv --to-tsv

 cat <<-END_VERSIONS > versions.yml
 "NFCORE_AMPLISEQ:AMPLISEQ:QIIME2_EXPORT:QIIME2_EXPORT_RELASV":
   qiime2: $( qiime --version | sed '1!d;s/.* //' )
 END_VERSIONS

Command exit status:
 1

Command output:
 (empty)

Command error:
 QIIME is caching your current deployment for improved performance. This may take a few moments and should only happen once per deployment.
 Plugin error from feature-table:

  [Errno 2] No such file or directory: '/tmp/2679020.1.largemem/qiime2/nvincen2/processes/42-1767742149.28@nvincen2/34152e01-6845-4c3d-bfbc-e879f5ec014a.4157825574201958688/34152e01-684>

 Debug info has been saved to /tmp/2679020.1.largemem/qiime2-q2cli-err-33ld04xk.log

Work dir:
 /temp180/mpfrende/nvincen2/Metagenomics/nf-core_ampliseq/nfcore_scripts/work/f5/0e9ab5cf4de96323b92551800dfb42

Container:
 /users/nvincen2/.apptainer/quay.io-qiime2-amplicon-2024.10.img

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

 -- Check '.nextflow.log' file for details
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: <https://nf-co.re/docs/usage/troubleshooting>

 -- Check '.nextflow.log' file for details

Then I submitted the same job again, but this time with the -resume flag added, with the same ~800GB of space requested. The error this time was:

-[nf-core/ampliseq] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_AMPLISEQ:AMPLISEQ:QIIME2_ANCOM:ANCOMBC_FORMULA_TAX (filtered-table-Lake-4)'

Caused by:
 Process `NFCORE_AMPLISEQ:AMPLISEQ:QIIME2_ANCOM:ANCOMBC_FORMULA_TAX (filtered-table-Lake-4)` terminated with an error exit status (1)


Command executed:

 export XDG_CONFIG_HOME="./xdgconfig"
 export MPLCONFIGDIR="./mplconfigdir"
 export NUMBA_CACHE_DIR="./numbacache"

 # Sum data at the specified level
 qiime taxa collapse \
   --i-table "filtered-table.qza" \
   --i-taxonomy "taxonomy.qza" \
   --p-level 4 \
   --o-collapsed-table "lvl4-Lake.qza"

 # Extract summarised table and output a file with the number of taxa
 qiime tools export \
   --input-path "lvl4-Lake.qza" \
   --output-path exported/
 biom convert \
   -i exported/feature-table.biom \
   -o "lvl4-Lake.feature-table.tsv" \
   --to-tsv

 if [ $(grep -v '^#' -c "lvl4-Lake.feature-table.tsv") -lt 2 ]; then
   mkdir differentials
   echo 4 > differentials/"WARNING Summing your data at taxonomic level 4 produced less than two rows (taxa), ANCOMBC can't proceed -- did you specify a bad reference taxonomy?".txt
   mkdir da_barplot
   echo 4 > da_barplot/"WARNING Summing your data at taxonomic level 4 produced less than two rows (taxa), ANCOMBC can't proceed -- did you specify a bad reference taxonomy?".txt
 else
   qiime composition ancombc \
     --i-table "lvl4-Lake.qza" \
     --m-metadata-file "zoop96_metadata.tsv" \
      --p-prv-cut 0.1 --p-lib-cut 500 --p-alpha 0.05 --p-conserve \
     --p-formula 'Lake' \
     --o-differentials "lvl4-Lake.differentials.qza" \
     --verbose
qiime tools export \
     --input-path "lvl4-Lake.differentials.qza" \
     --output-path "differentials/Category-Lake-level-4"

   # Generate tabular view of ANCOM-BC output
   qiime composition tabulate \
     --i-data "lvl4-Lake.differentials.qza" \
     --o-visualization "lvl4-Lake.differentials.qzv"
   qiime tools export \
     --input-path "lvl4-Lake.differentials.qzv" \
     --output-path "differentials/Category-Lake-level-4"

   # Generate bar plot views of ANCOM-BC output
   qiime composition da-barplot \
     --i-data "lvl4-Lake.differentials.qza" \
     --p-effect-size-threshold 1.0 --p-significance-threshold 0.05 --p-label-limit 1000 \
     --o-visualization "lvl4-Lake.da_barplot.qzv"
   qiime tools export --input-path "lvl4-Lake.da_barplot.qzv" \
     --output-path "da_barplot/Category-Lake-level-4"
 fi

 cat <<-END_VERSIONS > versions.yml
 "NFCORE_AMPLISEQ:AMPLISEQ:QIIME2_ANCOM:ANCOMBC_FORMULA_TAX":
   qiime2: $( qiime --version | sed '1!d;s/.* //' )
 END_VERSIONS

Command exit status:
 1

Command output:
 (empty)

Command error:
 QIIME is caching your current deployment for improved performance. This may take a few moments and should only happen once per deployment.
 Traceback (most recent call last):
  File "/opt/conda/envs/qiime2-amplicon-2024.10/bin/qiime", line 11, in <module>
   sys.exit(qiime())
  File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
   return self.main(*args, **kwargs)
  File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 1078, in main
   rv = self.invoke(ctx)
  File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
   return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
   return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
   return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/click/core.py", line 783, in invoke
   return __callback(*args, **kwargs)
  File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/q2cli/commands.py", line 463, in __call__
   value_ = cache.process_pool.save(value)
  File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/core/cache.py", line 1532, in save
   return self.load(ref)
  File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/core/cache.py", line 1689, in load
   archiver = Archiver.load_raw(path, self.cache)
  File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/core/archive/archiver.py", line 386, in load_raw
   process_alias = cache._alias(str(archive.uuid))
  File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/core/cache.py", line 1193, in _alias
   self.process_pool._make_symlink(uuid, process_alias)
  File "/opt/conda/envs/qiime2-amplicon-2024.10/lib/python3.10/site-packages/qiime2/core/cache.py", line 1625, in _make_symlink
   os.symlink(src, dest)
 FileNotFoundError: [Errno 2] No such file or directory: '/tmp/2685223.1.largemem/qiime2/nvincen2/data/1add9ede-3c86-4c39-a993-cf22ff6b68a1' -> '/tmp/2685223.1.largemem/qiime2/nvincen2/p>

Work dir:
 /temp180/mpfrende/nvincen2/Metagenomics/nf-core_ampliseq/nfcore_scripts/work/18/9d4a898bd7c22bacfbcffaace508e5

Container:
 /users/nvincen2/.apptainer/quay.io-qiime2-amplicon-2024.10.img

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: <https://nf-co.re/docs/usage/troubleshooting>

 -- Check '.nextflow.log' file for details

Seeing as I’m not changing any of the pipeline parameters or input files, and it’s returned 3 different errors each time the pipeline has stopped, I’m inclined to believe it might be a memory issue? But the highest maxvmem reported when the job quits is only ~250 GB. Should I just try to feed it more and more memory?

> If there is any way to make this more compact, please let me know. You could make a short post in the channel itself and the details in the thread? > When I excluded those files, the pipeline continued past this point without erroring. This can be a coincident (looking at your other file errors) or something is wrong with the files. You could re-try (i.e. -resume) or using a fastq validation program on it. > I run the pipeline on an HPC, and the function call in the job script is as follows: I am not sure whether you are aware: You might not use nextflow properly. An HPC typically has a scheduler that ideally is specified to nextflow (there is plenty of documentation both on nf-core as well as for nextflow itself). Specifying a sheduler & queues & computational resources properly might resolve your problems (or not). > and it’s returned 3 different errors each time the pipeline has stopped Its always a file error, 1x

not a directory

and twice

[Errno 2] No such file or directory:

. That hints to a problem with you file system, more particularly the temporary files management. This can have a multitude of origins, including HPC scratch limits, space limits, file system hiccups, apptainer issues. Potentially specifying the sheduler for nextflow as indicated above might solve your issues. Resuming isnt fixing the issue but it might lead to a complete pipeline run if its a non-reproducible problem that occurs not too often.

Hi again, I contacted my HPC support to ask about this issue and they suggested a couple of things. Since the process seems to be hanging up on /tmp/ directories/files not existing, they believed it to be an issue with how the pipeline (or QIIME2) was interacting with the

$TMPDIR

variable, which is set to a job-specific location (e.g.,

/tmp/JOBID.queue_name

). So they suggested I change where

$TMPDIR

was pointing by adding

export TMPDIR=/temp180/mpfrende/nvincen2/tmp

Which is a directory in my personal space. However, upon running the pipeline again, I am still getting the following error:

ERROR ~ Error executing process > 'NFCORE_AMPLISEQ:AMPLISEQ:QIIME2_DIVERSITY:QIIME2_TREE (1)'
Caused by:
  Process NFCORE_AMPLISEQ:AMPLISEQ:QIIME2_DIVERSITY:QIIME2_TREE (1) terminated with an error exit status (1)
Command executed:
  export XDG_CONFIG_HOME="./xdgconfig"
  export MPLCONFIGDIR="./mplconfigdir"
  export NUMBA_CACHE_DIR="./numbacache"
  qiime alignment mafft \
      --i-sequences filtered-sequences.qza \
      --o-alignment aligned-rep-seqs.qza \
      --p-n-threads 6
  qiime alignment mask \
      --i-alignment aligned-rep-seqs.qza \
      --o-masked-alignment masked-aligned-rep-seqs.qza
  qiime phylogeny fasttree \
      --i-alignment masked-aligned-rep-seqs.qza \
      --p-n-threads 6 \
      --o-tree unrooted-tree.qza
  qiime phylogeny midpoint-root \
      --i-tree unrooted-tree.qza \
      --o-rooted-tree rooted-tree.qza
  qiime tools export \
      --input-path rooted-tree.qza  \
      --output-path phylogenetic_tree
  cp phylogenetic_tree/tree.nwk .
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_AMPLISEQ:AMPLISEQ:QIIME2_DIVERSITY:QIIME2_TREE":
      qiime2: $( qiime --version | sed '1!d;s/.* //' )
  END_VERSIONS

Command exit status:
  1
Command output:
  (empty)
Command error:
  QIIME is caching your current deployment for improved performance. This may take a few moments and should only happen once>
  Plugin error from alignment:
    Command '['mafft', '--preservecase', '--inputorder', '--thread', '6', '/tmp/37880.1.largemem/qiime2/nvincen2/data/374754>
  Debug info has been saved to /tmp/37880.1.largemem/qiime2-q2cli-err-at0nv13u.log
Work dir:
  /temp180/mpfrende/nvincen2/Metagenomics/nf-core_ampliseq/nfcore_scripts/work/8f/c1c76559e2ef99e37b9252a4bb7fe3
Container:
  /temp180/mpfrende/nvincen2/.apptainer/quay.io-qiime2-amplicon-2024.10.img
Tip: when you have fixed the problem you can continue the execution adding the option -resume to the run command line

It looks like it is still trying to access the

/tmp/

folder for this process rather than what I set

$TMPDIR

to in the submission script. However, the directory I set as

TMPDIR

does exist, and is not empty, so it seems like some parts of the process are accessing it. Trying the pipeline again with

-resume

results in the exact same error. Since all the errors are stemming from this

/tmp/

directory, do you have any insight into how to change where the pipeline is saving temporary files?

do you have any insight into how to change where the pipeline is saving temporary files?

The tool QIIME2 is likely the problem, in combination with your system. As you can see the process already sets 3 environmental variables in an attempt to control QIIME2. Specifically about QIIME2_TREE there is an issue that might be helpful to you (towards the end are the solutions): https://github.com/nf-core/ampliseq/issues/516

I also looked into specifying the scheduler.

I have only experience with slurm clusters. And this is a very general problem, so I advice you to seek help in more general channels, such as #CE6SDBX2A. Of course you can use the pipelines without using a scheduler, if thats to your liking and you system setup seems to allow it (that unusual, in my experience).