Hi everyone! Having some issues here with the following error:

ERROR ~ Error executing process > 'NFCORE_PROTEINFOLD:ALPHAFOLD2:RUN_ALPHAFOLD2 ([])'

Caused by:
  Process exceeded running time limit (8h)

Apparently this is an error related to

nextflow

, some users had reported the same, please check the following GitHub #449 and #275 (post not related to

proteinfold

). Based on one of the comments I checked the file related to this pipeline

nf-core/config.nextflow

gitpod {
        executor.name           = 'local'
        executor.cpus           = 4
        executor.memory         = 8.GB
        process {
            resourceLimits = [
                memory: 8.GB,
                cpus  : 4,
                time  : 1.h
            ]
        }
    }

This is the only part that I found that could explain something about the time limit. Just to mention, my current job last 16 hours, and got this error. Now the context of my job: I'm using

AF-multimer

, I'm using a initial testing for my sequences. Each

fasta

file contain 3 proteins, I have 5

fasta

files in total, so 15 sequences. The length of the complex is around 2500 amino acids, so I wonder if this could be related with this error? I'm asking that because I read some paper that divide the analysis in batches of 300 amino acids, so that made me think if there is any relationship between the length and time limit? You can check the complete error here:

executor >  local (2)
[40/677087] NFC…AFOLD2:RUN_ALPHAFOLD2 ([]) | 2 of 5, failed: 2
[-        ] NFC…PROCESSING:GENERATE_REPORT -
[-        ] NFC…LD:POST_PROCESSING:MULTIQC -
-[nf-core/proteinfold] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_PROTEINFOLD:ALPHAFOLD2:RUN_ALPHAFOLD2 ([])'

Caused by:
  Process exceeded running time limit (8h)


Command executed:

  if [ -f pdb_seqres/pdb_seqres.txt ]
      then sed -i "/^\w*0/d" pdb_seqres/pdb_seqres.txt
  fi
  if [ -d params/alphafold_params_* ]; then ln -r -s params/alphafold_params_*/* params/; fi
  python3 /app/alphafold/run_alphafold.py         --fasta_paths=PA-20-T_A-PB1-548-F_L-PB2-122-V_I.fasta         --model_preset=multimer --pdb_seqres_database_path=./pdb_seqres/pdb_seqres.txt --uniprot_database_path=./uniprot/uniprot.fasta          --db_preset=reduced_dbs --small_bfd_database_path=./small_bfd/bfd-first_non_consensus_sequences.fasta         --output_dir=$PWD         --data_dir=$PWD         --uniref90_database_path=./uniref90/uniref90.fasta         --mgnify_database_path=./mgnify/mgy_clusters_2022_05.fa         --template_mmcif_dir=./mmcif_files         --obsolete_pdbs_path=./obsolete_pdb/obsolete.dat         --use_gpu_relax=true --max_template_date 2020-05-14

  cp "PA-20-T_A-PB1-548-F_L-PB2-122-V_I"/ranked_0.pdb ./"[]"_alphafold2.pdb
  cd "PA-20-T_A-PB1-548-F_L-PB2-122-V_I"
  awk '{print $6"\t"$11}' ranked_0.pdb | uniq > ranked_0_plddt.tsv
  for i in 1 2 3 4
      do awk '{print $6"\t"$11}' ranked_$i.pdb | uniq | awk '{print $2}' > ranked_"$i"_plddt.tsv
  done
  paste ranked_0_plddt.tsv ranked_1_plddt.tsv ranked_2_plddt.tsv ranked_3_plddt.tsv ranked_4_plddt.tsv > plddt.tsv
  echo -e Positions"\t"rank_0"\t"rank_1"\t"rank_2"\t"rank_3"\t"rank_4 > header.tsv
  cat header.tsv plddt.tsv > ../"[]"_plddt_mqc.tsv

  extract_output.py --name [] \
      --pkls features.pkl
  cd ..

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_PROTEINFOLD:ALPHAFOLD2:RUN_ALPHAFOLD2":
      python: $(python3 --version | sed 's/Python //g')
  END_VERSIONS

Command exit status:
  -

Command output:
  (empty)

Command error:
  I0407 15:01:59.672246 140710661338944 utils.py:40] Finished Jackhmmer (uniprot.fasta) query in 81.787 seconds
  I0407 15:01:59.691127 140710661338944 pipeline_multimer.py:210] Running monomer pipeline on chain C: PB2|A/Sea_Lion/Argentina/3893-1/2023|2.3.4.4b|2023-08-11|EPI_ISL_18698755 Mutant sequence for mutation: V_I, position: [122]
  I0407 15:01:59.691341 140710661338944 jackhmmer.py:133] Launching subprocess "/usr/bin/jackhmmer -o /dev/null -A /tmp/tmplp99axdv/output.sto --noali --F1 0.0005 --F2 5e-05 --F3 5e-07 --incE 0.0001 -E 0.0001 --cpu 8 -N 1 /tmp/tmpbbm86k5x.fasta ./uniref90/uniref90.fasta"
  I0407 15:01:59.691870 140710661338944 utils.py:36] Started Jackhmmer (uniref90.fasta) query
  I0407 15:03:31.377278 140710661338944 utils.py:40] Finished Jackhmmer (uniref90.fasta) query in 91.685 seconds
  I0407 15:03:31.390923 140710661338944 jackhmmer.py:133] Launching subprocess "/usr/bin/jackhmmer -o /dev/null -A /tmp/tmpglpvw8k5/output.sto --noali --F1 0.0005 --F2 5e-05 --F3 5e-07 --incE 0.0001 -E 0.0001 --cpu 8 -N 1 /tmp/tmpbbm86k5x.fasta ./mgnify/mgy_clusters_2022_05.fa"
  I0407 15:03:31.391493 140710661338944 utils.py:36] Started Jackhmmer (mgy_clusters_2022_05.fa) query
  I0407 15:16:07.249151 140710661338944 utils.py:40] Finished Jackhmmer (mgy_clusters_2022_05.fa) query in 755.858 seconds
  I0407 15:16:07.263861 140710661338944 hmmbuild.py:121] Launching subprocess ['/usr/bin/hmmbuild', '--hand', '--amino', '/tmp/tmp0vuvsl0l/output.hmm', '/tmp/tmp0vuvsl0l/query.msa']
  I0407 15:16:07.265030 140710661338944 utils.py:36] Started hmmbuild query
  I0407 15:16:07.550858 140710661338944 hmmbuild.py:128] hmmbuild stdout:
  # hmmbuild :: profile HMM construction from multiple sequence alignments
  # HMMER 3.3 (Nov 2019); <http://hmmer.org/>
  # Copyright (C) 2019 Howard Hughes Medical Institute.
  # Freely distributed under the BSD open source license.
  # - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  # input alignment file:             /tmp/tmp0vuvsl0l/query.msa
  # output HMM file:                  /tmp/tmp0vuvsl0l/output.hmm
  # input alignment is asserted as:   protein
  # model architecture construction:  hand-specified by RF annotation
  # - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

  # idx name                  nseq  alen  mlen eff_nseq re/pos description
  #---- -------------------- ----- ----- ----- -------- ------ -----------
  1     query                   15   887   759     2.84  0.590

  # CPU time: 0.28u 0.00s 00:00:00.28 Elapsed: 00:00:00.28


  stderr:


  I0407 15:16:07.550925 140710661338944 utils.py:40] Finished hmmbuild query in 0.286 seconds
  I0407 15:16:07.551663 140710661338944 hmmsearch.py:103] Launching sub-process ['/usr/bin/hmmsearch', '--noali', '--cpu', '8', '--F1', '0.1', '--F2', '0.1', '--F3', '0.1', '--incE', '100', '-E', '100', '--domE', '100', '--incdomE', '100', '-A', '/tmp/tmpk6bm7x3r/output.sto', '/tmp/tmpk6bm7x3r/query.hmm', './pdb_seqres/pdb_seqres.txt']
  I0407 15:16:07.552118 140710661338944 utils.py:36] Started hmmsearch (pdb_seqres.txt) query
  I0407 15:16:07.577770 140710661338944 utils.py:40] Finished hmmsearch (pdb_seqres.txt) query in 0.026 seconds
  I0407 15:16:07.583930 140710661338944 jackhmmer.py:133] Launching subprocess "/usr/bin/jackhmmer -o /dev/null -A /tmp/tmpvp7teoox/output.sto --noali --F1 0.0005 --F2 5e-05 --F3 5e-07 --incE 0.0001 -E 0.0001 --cpu 8 -N 1 /tmp/tmpbbm86k5x.fasta ./small_bfd/bfd-first_non_consensus_sequences.fasta"
  I0407 15:16:07.584345 140710661338944 utils.py:36] Started Jackhmmer (bfd-first_non_consensus_sequences.fasta) query
  I0407 15:17:37.748358 140710661338944 utils.py:40] Finished Jackhmmer (bfd-first_non_consensus_sequences.fasta) query in 90.164 seconds
  I0407 15:17:37.754861 140710661338944 templates.py:940] Searching for template for: MERIKELRDLMSQSRTREILTKTTVDHMAIIKKYTSGRQEKNPALRMKWMMAMKYPITADKRIMEMIPERNEQGQTLWSKTNDAGSDRVMVSPLAVTWWNRNGPTTSTVHYPKVYKTYFEKIERLKHGTFGPVHFRNQVKIRRRVDINPGHADLSAKEAQDVIMEVVFPNEVGARILTSESQLTITKEKKEELQDCKIAPLMVAYMLERELVRKTRFLPVAGGTSSVYIEVLHLTQGTCWEQMYTPGGEVRNDDVDQSLIIAARNIVRRATVSADPLASLLEMCHSTQIGGIRMVDILRQNPTEEQAVDICKAAMGLRISSSFSFGGFTFKRTSGSSVKKEEEVLTGNLQTLKIRVHEGYEEFTMVGRRATAILRKATRRLIQLIVSGRDEQSIAEAIIVAMVFSQEDCMIKAVRGDLNFVNRANQRLNPMHQLLRHFQKDAKVLFQNWGIEPIDNVMGMIGVMPDMTPSTEMSLRGIRVSKMGVDEYSSTERVVVSIDRFLRVRDQRGNVLLSPEEVSETQGTEKLTITYSSSMMWEINGPESVLVNTYQWIIRNWETVKIQWSQDPTMLYNKMEFEPFQSLVPKAARGKYSGFVRTLFQQMRDVLGTFDTVQIVKLLPFAAAPPEQSRMQFSSLTVNVRGSGMRILVRGNSPVFNYNKATKRLTVLGKDAGALTEDPDEGTAGVESAVLRGFLILGKENKRYGPALSINELSNLAKGEKANVLIGQGDVVLVMKRKRDSSILTDSQTATKRIRMAIN
  I0407 15:17:37.756692 140710661338944 pipeline.py:234] Uniref90 MSA size: 15 sequences.
  I0407 15:17:37.756767 140710661338944 pipeline.py:235] BFD MSA size: 3 sequences.
  I0407 15:17:37.756811 140710661338944 pipeline.py:236] MGnify MSA size: 2 sequences.
  I0407 15:17:37.756847 140710661338944 pipeline.py:237] Final (deduplicated) MSA size: 17 sequences.
  I0407 15:17:37.756883 140710661338944 pipeline.py:239] Total number of templates (NB: this can include bad templates and is later filtered to top 4): 1.
  I0407 15:17:37.757044 140710661338944 jackhmmer.py:133] Launching subprocess "/usr/bin/jackhmmer -o /dev/null -A /tmp/tmp6c_txo_t/output.sto --noali --F1 0.0005 --F2 5e-05 --F3 5e-07 --incE 0.0001 -E 0.0001 --cpu 8 -N 1 /tmp/tmpbbm86k5x.fasta ./uniprot/uniprot.fasta"
  I0407 15:17:37.757589 140710661338944 utils.py:36] Started Jackhmmer (uniprot.fasta) query
  I0407 15:18:59.588793 140710661338944 utils.py:40] Finished Jackhmmer (uniprot.fasta) query in 81.831 seconds
  I0407 15:18:59.748902 140710661338944 run_alphafold.py:276] Running model model_1_multimer_v3_pred_0 on PA-20-T_A-PB1-548-F_L-PB2-122-V_I
  I0407 15:18:59.749322 140710661338944 model.py:165] Running predict with shape(feat) = {'aatype': (2232,), 'residue_index': (2232,), 'seq_length': (), 'msa': (512, 2232), 'num_alignments': (), 'template_aatype': (4, 2232), 'template_all_atom_mask': (4, 2232, 37), 'template_all_atom_positions': (4, 2232, 37, 3), 'asym_id': (2232,), 'sym_id': (2232,), 'entity_id': (2232,), 'deletion_matrix': (512, 2232), 'deletion_mean': (2232,), 'all_atom_mask': (2232, 37), 'all_atom_positions': (2232, 37, 3), 'assembly_num_chains': (), 'entity_mask': (2232,), 'num_templates': (), 'cluster_bias_mask': (512,), 'bert_mask': (512, 2232), 'seq_mask': (2232,), 'msa_mask': (512, 2232)}
...

I know that the error says:

time limit (8h)

and the configuration file says

time_limit: 1h

but so far is the only explanation that I can find, no make sense to change the time for more if I do not understand the reason of why the limit is exceeded. I have in total 1200 sequence that I need to predict, so I need to understand this before go bigger. Any comment and help are greatly appreciate! Looking forward for your comment! Thank you everyone

Hi, are you using the

-profile podman

option in your command, I’d say that the resource time limit is not being applied and the process is being killed when the process reach the time limit of a module labeled as

process_medium

that is set here and labeled here

I'm using

-profile singularity

. The complete command is the following:

nextflow run nf-core/proteinfold -r dev -resume --input '/users/mmora30/mmora30/workspace/alphafold/alphafold2/alphafold-2.3.1/run_01_h5n1/input/testing_files/samplesheet.csv' --outdir '/users/mmora30/mmora30/workspace/alphafold/alphafold2/alphafold-2.3.1/run_01_h5n1/output/testing_files_out' --mode alphafold2 --alphafold2_db ${path_to_alphafold_db} --full_dbs false --alphafold2_model_preset multimer --use_gpu true -profile singularity

So, I do not understand why is this. What do you recommend?