Workflow execution completed unsuccessfully ```Error executi nf-core #methylong

Workflow execution completed unsuccessfully ```Er...

Jin Khoo

04/28/2025, 11:06 AM

Workflow execution completed unsuccessfully

Copy code

Error executing process > 'NFCORE_METHYLONG:METHYLONG:ONT:ONT_TRIM_REPAIR:PORECHOP_PORECHOP (HG002)'

Caused by:
  Essential container in task exited


Command executed:

  porechop \
      -i HG002_ont.fastq.gz \
      -t 3 \
      --discard_middle --extra_end_trim 10 \
      -o HG002.fastq.gz \
      > HG002.log
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_METHYLONG:METHYLONG:ONT:ONT_TRIM_REPAIR:PORECHOP_PORECHOP":
      porechop: $( porechop --version )
  END_VERSIONS

Command exit status:
  -

Command output:
  (empty)

Command error:
  10:07PM INF instance metadata ami-id=ami-082efb76d7dd52982 instance-id=i-001da517311a473fb instance-type=r6id.32xlarge
  Fusion Info:
      max_open_files: 1048576
      fusion_version: 2.4.11-8ead802
      clone_namespace: false
      kernel_version: 6.1
      disk_cache_size: 7076Gb

Work dir:
  <s3://nf-core-awsmegatests/work/methylong/work-f48b21e6a41757689b3951e23c28d70e0b274004/40/48a5519838ee5fb37d95d4148163df>

Container:
  <http://wave.seqera.io/wt/0f1b6e7ec6c7/library/porechop_pigz:d1655e5b5bad786c|wave.seqera.io/wt/0f1b6e7ec6c7/library/porechop_pigz:d1655e5b5bad786c>

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

hello all, do u know what was the reason for this error? This was from the full test running on aws.

Phil Ewels

04/28/2025, 12:26 PM

I imagine that the spot instance that the task was running on was reclaimed by AWS. This has been happening a lot lately. @Rike Hanssen has been trying to set up Fusion Snapshots to avoid failures, but perhaps it's gone wrong here?

Phil Ewels

04/28/2025, 9:24 PM

Yup, looks like Fusion tried to make snapshots but wasn't able to do it in time before AWS terminated the instance.

Phil Ewels

04/28/2025, 9:24 PM

It seems porechop is particularly vulnerable to this for some reason

Phil Ewels

04/28/2025, 9:24 PM

Only thing we can really do is run On-Demand, rather than using spot instances. It costs more, but means that stuff won't be terminated by AWS the whole time

Phil Ewels

04/28/2025, 9:25 PM

Please ask for help with this on #CTY8L26C8

Jin Khoo

04/29/2025, 5:58 AM

thanks Phil, will do so

Rike Hanssen

04/29/2025, 6:42 AM

We need to change the config and setup a proper on-demand queue for this, which someone from the <!subteam^S016E36N7TQ> needs to do. I am out for training for today, if somebody has time to do this

Jin Khoo

05/02/2025, 8:20 AM

i can also just disable running the porechop workflow part if thats easier

Jin Khoo

05/02/2025, 1:32 PM

Hi <!subteam^S016E36N7TQ>, could someone help in setting up a on-demand queue for this workflow? Ive been trying to run this aws full test couple of times, each time it failed but with slightly different errors, not sure if they are all related to this issue @Phil Ewels mentioned. the recent error I received was

Copy code

Host EC2 (instance i-0e20d5262cb594250) terminated.

from this run . thanks alot in advance

Rike Hanssen

05/02/2025, 3:53 PM

Alright I created new CEs. The spot one should retry better. I also added an on demand one. Submitting it now using both queues. As a side note, something seems to be off with the parameter validation

🙌 1

Rike Hanssen

05/02/2025, 3:59 PM

The outdir points by default to a local folder. I think we have set this empty on purpose so that people don't accidentally loose their results when running on clouds.

party this 1

Rike Hanssen

05/02/2025, 4:09 PM

https://cloud.seqera.io/orgs/nf-core/workspaces/AWSmegatests/watch/43iMnRwzPDt1oP Run with snapshots and on-demand fall back

Jin Khoo

05/05/2025, 8:35 AM

this one failed again, I think is with the same error:

Copy code

Terminating due to java.lang.OutOfMemoryError: Java heap space
  Fusion Info:
      clone_namespace: false
      kernel_version: 6.1
      disk_cache_size: 7076Gb
      max_open_files: 1048576
      fusion_version: 2.4.11-8ead802

Rike Hanssen

05/05/2025, 8:43 AM

I think that is a memory error, not because the instance got reclaimed. although it already uses 48GB

Jin Khoo

05/05/2025, 9:03 AM

hmm ok I think the requested memory wasnt recognised, according to the log here:

Copy code

Command error:
  4:41PM INF instance metadata ami-id=ami-076452d1c290afa09 instance-id=i-0f11d332d1eb46b15 instance-type=r6id.32xlarge
  Value "4096.0" invalid for option memory (number expected)

which is a little strange, because in the previous run it was also with the same requested resources and it worked

Jin Khoo

05/05/2025, 9:12 AM

ah figured out the issue, while updating the nf-core template I somehow downgraded the module doge

Jin Khoo

05/05/2025, 9:29 AM

fixed the issue, now it should work fine again

Open in Slack

Previous Next