Hi team I am trying to run phyloplace workflow Can someone p nf-core #phyloplace

Hi team, I am trying to run phyloplace workflow. C...

Jash Gopani

02/23/2024, 3:34 PM

Hi team, I am trying to run phyloplace workflow. Can someone please help me with this error. I am not able to understand what is going wrong. The command that I executed:

Copy code

nextflow run ./phyloplace/main.nf -profile local_trial --queryseqfile "./seqtab_nochim_refs.fasta" --refseqfile "./aligned_seqs.
filtered.fas"  --refphylogeny "tree.nwk" --outdir "./local_trial" --id "trial_phylogeny" --model "GTR+G" --alignmethod "mafft"

The error:

Copy code

executor >  local (3)
[-        ] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:HMMER_HMMBUILD                -
[-        ] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:HMMER_UNALIGNREF              -
[-        ] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:HMMER_HMMALIGNREF             -
[-        ] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:HMMER_HMMALIGNQUERY           -
[-        ] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:HMMER_MASKREF                 -
[-        ] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:HMMER_MASKQUERY               -
[-        ] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:HMMER_AFAFORMATREF            -
[-        ] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:HMMER_AFAFORMATQUERY          -
[67/7191e9] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:MAFFT (trial_phylogeny)       [100%] 1 of 1 ✔
[09/bffc69] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:EPANG_SPLIT (trial_phylogeny) [100%] 2 of 2, failed: 2, retries: 1 ✘
[-        ] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:EPANG_PLACE                   -
[-        ] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:GAPPA_GRAFT                   -
[-        ] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:GAPPA_ASSIGN                  -
[-        ] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:GAPPA_HEATTREE                -
[-        ] process > NFCORE_PHYLOPLACE:PHYLOPLACE:CUSTOM_DUMPSOFTWAREVERSIONS                            [  0%] 0 of 1
[-        ] process > NFCORE_PHYLOPLACE:PHYLOPLACE:MULTIQC                                                -

[91/13ae32] NOTE: Process `NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:EPANG_SPLIT (trial_phylogeny)` terminated with an error exit status (134) -- Execution is retried (1)
ERROR ~ Error executing process > 'NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:EPANG_SPLIT (trial_phylogeny)'

Caused by:
  Process `NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:EPANG_SPLIT (trial_phylogeny)` terminated with an error exit status (134)

Command executed:

  epa-ng \
       \
      --split aligned_seqs.filtered.fas trial_phylogeny.fas
  
  gzip -c query.fasta > trial_phylogeny.query.fasta.gz; rm query.fasta
  gzip -c reference.fasta > trial_phylogeny.reference.fasta.gz; rm reference.fasta
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:EPANG_SPLIT":
      epang: $(echo $(epa-ng --version 2>&1) | sed 's/^EPA-ng v//')
  END_VERSIONS

Command exit status:
  134

Command output:
  INFO Splitting files based on reference: aligned_seqs.filtered.fas

Command error:
  INFO Splitting files based on reference: aligned_seqs.filtered.fas
  terminate called after throwing an instance of 'std::out_of_range'
    what():  vector::_M_range_check: __n (which is 0) >= this->size() (which is 0)
  .command.sh: line 4: 639272 Aborted                 (core dumped) epa-ng --split aligned_seqs.filtered.fas trial_phylogeny.fas

Work dir:
  /home/jbgopani/Projects/phyloplace/work/09/bffc69e5cc9f01988ae4f04e6bcdf5

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details

Daniel Lundin

02/23/2024, 4:59 PM

Difficult to say. The failing step takes the alignment and splits it into two, one with the reference sequences, one with the queries. Could you post the three input files so I can try it out? Either here or by email to matricaria.suaveolens@gmail.com.

Daniel Lundin

02/23/2024, 5:00 PM

Another option would be to try the default hmmer strategy (i.e.

--alignmethod hmmer

) which is not dependent on this step.

Jash Gopani

02/24/2024, 11:36 PM

Hi @Daniel Lundin, I tried with hmmer also, but that option was also not working. I've emailed you the input files, let me know once you try out. Thanks a lot for the help 🙌

Daniel Lundin

02/26/2024, 7:12 AM

Query sequence missing in the email, see my reply.

Daniel Lundin

02/26/2024, 6:42 PM

You have

in the reference alignment sequences. Replacing them with

should make the pipeline work. (It's running in hmmer mode on my laptop and got past the place where it faulted before.)

Daniel Lundin

02/26/2024, 6:43 PM

I have no idea what marker you're using but if it's protein coding I'd be suspicious about all the single nucleotide (frameshifting) gaps. Overall, it looks a bit odd. But again, I have no idea what the marker is.

Daniel Lundin

02/26/2024, 9:08 PM

I can confirm that removing the `?`s solves your issue. Another thing I thought of: You didn't specify a taxonomy, i.e. a tab separated field with sequence name (1st column) and full taxonomy (

-separated, 2nd column). If you do, you'll get a tsv file with taxonomy assigned to each of your input sequences.

Jash Gopani

02/28/2024, 7:25 PM

Great...I'll try replacing the

and let you know. Thanks a lot 🙌

Jash Gopani

02/29/2024, 3:42 AM

Hi Daniel, when you tried running this workflow, did it successfully complete on your machine?

Jash Gopani

02/29/2024, 3:43 AM

As you mentioned, by replacing the "?" I was able to move ahead of the step where I was getting an error but I am getting a different error now

Jash Gopani

02/29/2024, 3:45 AM

Copy code

[nf-core/phyloplace] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:GAPPA_HEATTREE (trial_phylogeny)'

Caused by:
  Process `NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:GAPPA_HEATTREE (trial_phylogeny)` terminated with an error exit status (106)

Command executed:

  gappa \
      examine \
      heat-tree \
      --threads 2 \
      --file-prefix trial_phylogeny.heattree. \
      --jplace-path trial_phylogeny.epa_result.jplace.gz \
      --write-nexus-tree --write-phyloxml-tree --write-svg-tree \
      --log-file trial_phylogeny.heattree.log
  
executor >  local (11)
[66/971563] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:HMMER_HMMBUILD (trial_phylogeny)       [100%] 1 of 1 ✔
[f0/dd6a53] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:HMMER_UNALIGNREF (trial_phylogeny)     [100%] 1 of 1 ✔
[79/9ab428] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:HMMER_HMMALIGNREF (trial_phylogeny)    [100%] 1 of 1 ✔
[34/8b9cda] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:HMMER_HMMALIGNQUERY (trial_phylogeny)  [100%] 1 of 1 ✔
[5e/6b17bf] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:HMMER_MASKREF (trial_phylogeny)        [100%] 1 of 1 ✔
[34/49dda5] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:HMMER_MASKQUERY (trial_phylogeny)      [100%] 1 of 1 ✔
[71/4bf4c3] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:HMMER_AFAFORMATREF (trial_phylogeny)   [100%] 1 of 1 ✔
[5a/ae3774] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:HMMER_AFAFORMATQUERY (trial_phylogeny) [100%] 1 of 1 ✔
[-        ] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:MAFFT                                  -
[-        ] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:EPANG_SPLIT                            -
[40/e10715] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:EPANG_PLACE (trial_phylogeny)          [100%] 1 of 1 ✔
[b9/068b20] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:GAPPA_GRAFT (trial_phylogeny)          [100%] 1 of 1, failed: 1 ✘
[-        ] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:GAPPA_ASSIGN                           -
[42/7486b9] process > NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:GAPPA_HEATTREE (trial_phylogeny)       [100%] 1 of 1, failed: 1 ✘
[-        ] process > NFCORE_PHYLOPLACE:PHYLOPLACE:CUSTOM_DUMPSOFTWAREVERSIONS                                     -
[-        ] process > NFCORE_PHYLOPLACE:PHYLOPLACE:MULTIQC                                                         -
INFO Selected: Output dir: ./
INFO Selected: Query file: trial_phylogeny.query.hmmalign.masked.afa.gz
INFO Selected: Tree file: tree_v2.nwk
INFO Selected: Reference MSA: trial_phylogeny.ref.hmmalign.masked.afa.gz
INFO Selected: Automatic switching of use of per rate scalers
INFO Selected: Preserving the root of the input tree
INFO Selected: Specified model: GTR+G
INFO Selected: Using threads: 8
INFO     ______ ____   ___           _   __ ______
        / ____// __ \ /   |         / | / // ____/
       / __/  / /_/ // /| | ______ /  |/ // / __  
      / /___ / ____// ___ |/_____// /|  // /_/ /  
     /_____//_/    /_/  |_|      /_/ |_/ \____/ (v0.3.8)
WARN The reference MSA and tree have differing number of taxa! 2522 vs. 2507
INFO Using model parameters:
INFO    Rate heterogeneity: GAMMA (4 cats, mean),  alpha: 1 (ML),  weights&rates: (0.25,0.136954) (0.25,0.476752) (0.25,1) (0.25,2.38629) 
        Base frequencies (ML): 0.25 0.25 0.25 0.25 
        Substitution rates (ML): 0.5 0.5 0.5 0.5 0.5 1
INFO Output file: ./epa_result.jplace
INFO 5000 Sequences done!
INFO 10000 Sequences done!
INFO 14361 Sequences done!
INFO Time spent placing: 1033s
INFO Elapsed Time: 1041s

Execution cancelled -- Finishing pending tasks before exit
-[nf-core/phyloplace] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:GAPPA_HEATTREE (trial_phylogeny)'

Caused by:
  Process `NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:GAPPA_HEATTREE (trial_phylogeny)` terminated with an error exit status (106)

Command executed:

  gappa \
      examine \
      heat-tree \
      --threads 2 \
      --file-prefix trial_phylogeny.heattree. \
      --jplace-path trial_phylogeny.epa_result.jplace.gz \
      --write-nexus-tree --write-phyloxml-tree --write-svg-tree \
      --log-file trial_phylogeny.heattree.log
  
  grep '^ *At' trial_phylogeny.heattree.log > trial_phylogeny.heattree.colours.txt
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_PHYLOPLACE:PHYLOPLACE:FASTA_NEWICK_EPANG_GAPPA:GAPPA_HEATTREE":
      gappa: $(echo $(gappa --version 2>&1 | sed 's/v//' ))
  END_VERSIONS

Command exit status:
  106

Command output:
  (empty)

Command error:
  A subcommand is required
  Run with --help for more information.

Daniel Lundin

02/29/2024, 6:23 AM

Strange. It finished fine for me. The error message ("A subcommand is required") seems contradictory to the command actually run (

gappa examine heat-tree ...

) that does include a subcommand. What command line did you run? Did you use mafft? (BTW, you get a warning from EPA-NG ("WARN The reference MSA and tree have differing number of taxa! 2522 vs. 2507"). Although it finished anyway, you should make sure the reference alignment is the alignment used to estimate the tree, which might not be the case since you get this warning. I'm writing "might" because certain phylogeny programs remove duplicate sequences and add them back, and something like might have happened. In any case, make sure the reference alignment and tree contain the same taxa.)

Daniel Lundin

02/29/2024, 6:41 AM

I just ran with mafft and it also finished fine. I'm attaching my parameter file (for hmmer; I overrode that with

--alignmethod mafft

when I ran with mafft).

params.yml

Jash Gopani

02/29/2024, 4:58 PM

I ran the same command with the corrected file names

Copy code

nextflow run ./phyloplace/main.nf -profile local_trial --queryseqfile "./seqtab_nochim_refs.fasta" --refseqfile "./aligned_seqs.filtered.fas"  --refphylogeny "tree_v2.nwk" --outdir "./local_trial" --id "trial_phylogeny" --model "GTR+G" --alignmethod "mafft"

tried with and without mafft but both times the workflow failed. I have made changes to the cpu and memory requirement in the config file, could that cause any issues? I don't think it should affect but since you've mentioned max_memory in the params.yml I thought I'd confirm once

Jash Gopani

02/29/2024, 4:59 PM

Also, very basic doubt, how are you passing this params.yml file into the command line? It seems a convenient option

Daniel Lundin

02/29/2024, 5:03 PM

No, memory and cpu settings should not cause this problem. By and large, however, the pipeline is not very resource hungry. I ran your data on my laptop (quite modern, with 32 GiB RAM, allocating max. 20 to the pipeline) and that went fine. I hence don't think you need to fiddle so much with cpu and memory settings besides maybe setting

--max_memory

to make sure you're below physical RAM.

Daniel Lundin

02/29/2024, 5:03 PM

You pass parameter files to Nextflow with

-params-file

. My command was hence:

nextflow run nf-core/phyloplace -r 1.0.0 -profile docker -params-file params.yml -resume

👍 1

Daniel Lundin

02/29/2024, 5:08 PM

The only thing I can see that differs for your setting and mine is the

-profile local_trial

, is that something you define in a config file? (I can't see how that could cause the actual error you see, but might be worth turning all stones over.) Generally, to apply settings to a particular project you'd just create a

nextflow.config

with settings, but without creating a profile. You are using some container engine, or?

Jash Gopani

02/29/2024, 5:43 PM

Yes, that profile I've defined in the config file.

Copy code

report {
    file = "report.html"
    enabled = true
    overwrite = true
}

timeline {
    file = "timeline.html"
    enabled = true
    overwrite = true
}

process {
    echo = true
}

params {
    limit = 5
}

profiles{
    server31 {
        params.limit = -1
        params.resultFileName = 'results.txt'
        params.storeDir = "/server31_work/nhmmer/"
        process.echo = false
        resume = true

    }

    server31_trial {
        params.limit = 5
        params.resultFileName = 'results_trial.txt'
        params.storeDir = "/server31_work/nhmmer/"
        resume = true

    }

    local_trial {
        params.limit = 5
        params.resultFileName = 'results_local.txt'
        params.storeDir = "./local_trial/"
    }

}

I use this config file for all my workflows

Jash Gopani

02/29/2024, 5:44 PM

I am not using any container also, just running workflows directly

Jash Gopani

02/29/2024, 5:45 PM

Some of the params in this config file are not relevant to this phyloplace workflow, they were used in few workflows I created.

Daniel Lundin

02/29/2024, 5:54 PM

I see. Not using a container engine makes it a bit more difficult to debug your error (and your runs less reproducible) since I can't be certain you're using the same software as I do. Would it be possible for you to run a test with Docker or Singularity, preferably just using the

docker

inbuilt nf-core profile? In theory, Conda should also work but might be trickier.

Jash Gopani

02/29/2024, 6:09 PM

Sure, I'll try that and let you know.

Jash Gopani

03/02/2024, 1:32 AM

Hi Daniel, I haven't run workflows in docker container before, just want to ensure that I am doing things correctly 1. I created a docker image with the required softwares in it (java8, python, conda, hmmer, mafft, gappa ) and activated the conda environment 2. I have not installed nextflow in it 3. I have also put all my params in the params.yml file For running the workflow I just need to run the following command right:

Copy code

nextflow ./phyloplace/main.nf -with-docker nextflow_container -params-file params.yml

Jash Gopani

03/02/2024, 1:35 AM

This is my docker file

Dockerfile

Jash Gopani

03/02/2024, 2:16 AM

Good news! My workflow ran in this container!

Jash Gopani

03/02/2024, 2:16 AM

Thanks a lot for your help 🙌 salute

Daniel Lundin

03/04/2024, 9:33 AM

Right. You actually don't need to write your own container. Each step in the pipeline is defined with a Docker container, so all you need to do is use

-profile docker

and it will pull the correct containers.

Daniel Lundin

03/04/2024, 9:35 AM

Moreover, you don't need to clone or in any other way download the pipeline repository; Nextflow does that for you. Your Nextflow command should hence be:

nextflow run nf-core/phyloplace -r 1.0.0 -profile docker -params-file params.yml

. (The

-r 1.0.0

isn't necessary but good practice.)

Daniel Lundin

03/04/2024, 9:36 AM

It's easier than you think!

Jash Gopani

03/07/2024, 12:58 AM

Ohh, great to know

Open in Slack

Previous Next