https://nf-co.re logo
Join Slack
Powered by
I saw a couple of months ago the AWS full size tes...
# aws-megatests
s
I saw a couple of months ago the AWS full size test needed to be triggered manually, is this still the case?
e
I am having a look..is this the full_test samplesheet you want to run? fastq_eval.subject_a.wgts.tndna_trna.minimal.csv?
s
Hi @Evangelos Karatzas, thanks so much for taking a look. I’d like to have the profile ‘test_full’ run, which uses the ‘full_size.hcc1395.wgts.tndna_trna.fastq.csv’ samplesheet
This is the profile definition here: conf/test_full.config
And I think this is the profile that is used in the awsfulltest.yml test GH Action workflow
e
I don't have the power to re-run it but these are my findings for this run: https://cloud.seqera.io/orgs/nf-core/workspaces/AWSmegatests/watch/YxaGU7ZwWm6PI/v2/logs Changed Revision to 2.3.0 (was dev) Changed work directory to s3://nf-core-awsmegatests input to: https://raw.githubusercontent.com/nf-core/test-datasets/oncoanalyser/samplesheet/full_size.hcc1395.wgts.tndna_trna.fastq.csv (not sure if this the samplesheet you want) output to s3://nf-core-awsmegatests/oncoanalyser/results-8f8ffa24ffde97d1a16215292289d2f2c1ed9fda (was s3://nf-core-awsmegatests/oncoanalyser/results-dev, and the only one that didnt match a current release on the site was s3://nf-core-awsmegatests/oncoanalyser/results-8f8ffa24ffde97d1a16215292289d2f2c1ed9fda) fastp_umi_length was complaining so turned that to 0 (not sure which one should be the default) -> Edited for correct full test samplesheet
🙏 1
@Rike Hanssen if you could try re-running with these changes maybe?
s
This is super helpful, do you think it is possible to get access to the Seqera Platform nf-core organisation?
That way I could take a look at the logs myself, but totally understandable if access is controlled and that isn’t possible
e
You can be promoted for a while to debug your own stuff I guess xD
😃 1
s
I wouldn’t say no to log access even if it’s temporary, but again very understandable if that might not be feasible
I’m on Australian time and will call it a day in a moment, tagging @Rayan Hassaïne who might be able to field some oncoanalyser questions
It seems like the cause probably involves incomplete / invalid schema definitions or handling that is only apparent when run on Seqera Platform. Keen to get this fixed, so thanks everyone for the help
🙌 1
r
@Stephen Watts yes you can. All pipeline maintainers should have at least launch permissions
added you with your slack email
🙏🏻 1
🙏 1
s
Thank you, @Rike Hanssen!
The cause just looks to be an OOM error on the pipeline EC2 instance that occurred during staging foreign files from a HTTP remote to AWS S3:
Copy code
Dec-08 11:31:25.750 [Actor Thread 110] ERROR nextflow.processor.TaskProcessor - Execution aborted due to an unexpected error
java.lang.OutOfMemoryError: Java heap space
	at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:71)
	at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:391)
	at nextflow.cloud.aws.nio.S3OutputStream.allocate(S3OutputStream.java:301)
	at nextflow.cloud.aws.nio.S3OutputStream.write(S3OutputStream.java:259)
	at java.base/java.io.OutputStream.write(OutputStream.java:167)
	at java.base/java.io.InputStream.transferTo(InputStream.java:797)
	at java.base/java.nio.file.Files.copy(Files.java:3164)
	at nextflow.file.CopyMoveHelper.copyFile(CopyMoveHelper.java:95)
	at nextflow.file.CopyMoveHelper.copyToForeignTarget(CopyMoveHelper.java:178)
	at nextflow.file.FileHelper.copyPath(FileHelper.groovy:1009)
	at nextflow.file.FilePorter$FileTransfer.stageForeignFile0(FilePorter.groovy:373)
	at nextflow.file.FilePorter$FileTransfer.stageForeignFile(FilePorter.groovy:328)
	at nextflow.file.FilePorter$FileTransfer.run(FilePorter.groovy:300)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.runWith(Thread.java:1596)
	at java.base/java.lang.Thread.run(Thread.java:1583)
The error occurs after staging of reference data files and during retrieval of 24 separate 10 GB FASTQ files, which I’d guess is more than usual and exceeds resources on the default pipeline EC2 instance
If the cause is OOM during foreign file staging then the pipeline instance would need to be given more memory to resolve, which I think is possible at least from the UI
I’ll hold off from trying to do anything other than looking through outputs / logs. Can anyone let me know whether adjusting pipeline instance resources is possible?
e
Ouch.. isn't there a smaller test dataset than can test the full pipeline? XD
s
Ha, yes that is possible, though I was aiming to have typical full sized data used in the ‘test_full’ profile to help with benchmarking / user expectations and provide an official nf-core location for these results (see here for an example of user benchmark and temporary location)
Ideally we would be able to run ‘test_full’ as it is with a pipeline instance that has sufficient memory to complete foreign file staging but I can find some compromises re the actual test data if that isn’t possible
r
that's why we normally store them on our s3 bucket. Can we move the data over?
looks like part of it is really similar to the sarek full size test data. Can you reuse that one?
r
Hi Rike! I've done quite a bit of work related to use the Magic Cache for the ci/cd on the repo - could that be of any help ?
r
don't think so. Why can't we move the data to the s3 bucket/ re-use existing data?
r
do you mean staging our own resource/reference files on s3 buckets, I don't see why not ? Just not sure how to do it
r
You can request it in #C09H6NYHR9T
s
I agree with Rike, the Magic Cache is specific to RunsOn and doesn’t have the persistence we would need - it’s usefulness is in caching data for a short period of time in the RunsOn S3 infra rather than the GH Action cache
I have been able to manually run the full size test on AWS Batch without any errors when the pipeline EC2 instance has 16 GB memory. For comparison, the pipeline EC2 instance used in the nf-core Seqera Platform workspace here has only 4 GB memory according to the NF logs
I see the error as being more directly caused by instance configuration rather than storage location e.g. if a user had this error I would suggest to increase instance memory so the built-in Nextflow file staging feature could function as expected
I do understand your point about having test data on S3 so the pipeline EC2 instance can be kept small, and copying this full sized test data from Cloudflare R2 into the standard AWS S3 location for nf-core pipelines makes sense for consistency too. I’ll put in a request, thank you for the suggestion!
I’ve also now had a look at the Sarek test data and it unsurprisingly includes HCC1395 cell line data too! Though they differing in sequence experiment types , targeted (WES / WXS) v. WGS. The other important difference is that the HCC1395 tumor / normal WGS data is paired with tumour WTS
r
sounds good and make sense that you use a different subset of that cell line data
😃 1
s
I’ll switch to the migrated data in the next release, and for the current 2.3.0 release I’m still aiming to have outputs from the full sized test appear in the pipeline’s ‘Results’ page on the nf-core website
I think the easiest approach for this would be to manually launch the full test in the nf-ore Seqera Platform workspace and adjust the pipeline instance memory from the default 4 GB to 16 GB. Is that possible?
Otherwise I can imagine it is probably possible to manually upload results, which I can provide. Also open to alternatives to get 2.3.0 outputs to appear in the Results page!
r
yep we can bump the head job resources for this release. did the run work for the PR from dev -> master?
s
Hello again, apologies for the slow response - we’re wrapping up here for the year
we can bump the head job resources for this release. did the run work for the PR from dev -> master?
Thanks so much, this would be amazing! And yes, I separately completed a successful run of the ‘test_full’ profile, which is the same used in the AWS full sized test via Seqera Platform that we’ve been discussing
The corresponding job submission (that eventually failed due to insufficient memory on the pipeline node) from the GH Action run from the ‘Run AWS full tests’ workflow is here
Weirdly, it returned a fail status for the submission, not sure what is happening there but I believe it is unrelated to the outcome of the actual pipeline execution
All other tests (both run by GH Actions and manually) for the dev → master PR also succeeded
And no worries if this is left until the new year, happy holidays!
r
yes the failed submission is unrelated to the actual test
s
This dropped off the todo list but coming back to it now - @Rike Hanssen would you be able to kick this off for us when you have time?
👀 1
r
Submitted: https://cloud.seqera.io/orgs/nf-core/workspaces/AWSmegatests/watch/3nbaOAAEDJ9J2v/v2/tasks You should have sufficient platform permissions to also run this yourself in case the action is broken again. Let me know if I need to bump anything
Done
🙏 1
s
Thank you! I’m seeing some very unusual errors in one of the bwamem2 alignment jobs that caused the run to fall over:
Copy code
12:22PM INF [seqera-fusion] setting up AWS Batch spot instance monitoring provider=aws-batch
12:22PM INF [seqera-fusion] instance metadata ami-id=ami-068145e3a19edb6fd instance-id=i-0e1fc173f52f69e75 instance-type=c6id.24xlarge
12:22PM INF [seqera-fusion] starting incremental snapshot timer interval=300000
FUSION_GPU_USED=false
[2026-02-16 12:22:19.838] [ERROR] Error (helpers.c:1011): dump #6: size mismatch for pages-7.img (expected 21234061312, got 0)
[2026-02-16 12:22:19.839] [ERROR] Error (helpers.c:1024): dump #6: 1 of 28 files failed verification
[2026-02-16 12:22:20.081] [ERROR] Error (helpers.c:1011): dump #8: size mismatch for pages-7.img (expected 21129981952, got 0)
[2026-02-16 12:22:20.087] [ERROR] Error (helpers.c:1024): dump #8: 1 of 224 files failed verification
[2026-02-16 12:22:20.087] [ERROR] Error (helpers.c:1062): chain #0: 2 dumps failed verification
Fusion Info:
    fusion_version: 2.4.20-ce52500
    clone_namespace: false
    kernel_version: 6.1
    disk_cache_size: 5306Gb
    max_open_files: 1048576
I suspect it might be an intermittent error and so I’ll kick off another run tomorrow unless you have any other suggestions!
r
I already did
these are errors from spot reclamations. Fusion snapshotting is then trying to rescue the run and move it to a different machine in an attempt to not immediatly fail, which didn't help in this instance
s
Amazing, thank you! Is there anything further we need to do for the results to appear in the nf-co.re website here?
Following this up with a question / request - the results are now available in the nf-co.re results section for oncoanalyser but there are a few files in the root directory that shouldn’t be there like .command files and intermediary BAM files. It looks like files from a couple of work directories have copied into the root directory
Is it possible to manually delete these files? Here is a full list of files on the 2.3.0 results pages that shouldn’t be in the output directory for a typical analysis. Thank for the help so far!
oncoanalyser-2.3.0--nf-core-results--unexpected-files.txt
Is there anyone that might be able to take a look at this for us? Any help would be greatly appreciated!
p
Please put this into #C09H6NYHR9T as it has to be a core team member
🙏 1
m
All files were removed
r
@Maxime Garcia which files are you referring to ?
m
r
Great, thanks for that! And the fact that they're still on the website is due to the website updating not being instantaneous ?
s
Thank you @Maxime Garcia!
I was going to wait until tomorrow to ask that question 🙂 I would guess you are right, Rayan!
m
yeah, website takes a bit of time to update
👍 1
Open in Slack
PreviousNext
Powered by