https://nf-co.re logo
Join Slack
Powered by
A new request to upload a dataset to nf-core AWS S...
# request-core
t
A new request to upload a dataset to nf-core AWS S3 bucket has been made! • Requested by: @Luis Heinzlmeier • Pipeline name: hadge • Why is this needed?: A real-life dataset for 
test_full
. • Size of the dataset: 60 GB • URL: . • Related files: Core team will add this dataset to the nf-core bucket. Please react to the message with ✅ when this is done. [This process is not automated]
l
Where should I upload the dataset?
j
In the past you would supply a link of someform that someone can download
👍 1
And then the on call team member (this week it is @Sateesh_Peri if he is around) would download to their location and upload it.
But I vaguely remember @Phil Ewels maybe saying we could do it another way? But I don't remember exactly
s
I can help. plz provide link to download
🙏 1
l
I've submitted a large data transfer request to my organization, but based on my past experience, it might take a while. If you have any other suggestions for where I can upload something larger than 50 GB, let me know.
p
Where is the data now? If it's somewhere we one of the core team can download it directly already then no need for you to send us a copy
l
It’s on our institution’s cluster with very strict permission rules, so I either upload it to an external drive from there or wait for the IT department.
👍 1
👍🏻 1
I would really appreciate it if you could help me set up 
test_full
for hadge using this data.
j
Would be @Phil Ewels this week 🙂
👍🏻 1
👍 1
p
working on it now..
❤️ 1
Uploading now:
Copy code
<s3://nf-core-awsmegatests/hadge/input_data/>
🥳 1
..done! ✅
gratitude thank you 1
l
Let me know if there's anything I need to do.
p
Just check that they're there and look as you expect, then wire them up into your pipeline 🙂
They're publicly visible, it's just the uploading that needs special access.
l
All right! I'll give it a try.
I added a samplesheet for test_full and updated `input` in `test_full.config`. How can I now trigger AWS tests?
j
If you go to the actions tab of your repo, go to the relevant test, and i nthe top right of that page you should see a 'Run manual' drop down button or something like that
gratitude thank you 1
l
j
No no that's OK
That triggers a Seqera Platform API call 😉
One sec
It did start!
But failed 😆
image.png
But let me give you access
👍 1
What is the email you use for for Seqera Platform?
(if you use it)
Or your handle?
Ah your TUM one
?
👍 1
l
Yes, I can 🙂
But failed 😆
So the problem is an Access Denied error!? 😅
j
Yes...
I'm not sure here...
Riiiiiiikkeeeee (@Rike Hanssen) platform help pleaaasseeee
😂 1
r
watcha need?
j
Why did Luis' run fail with s3 403
r
are you accessing an s3 bucket that is not defined in the CE @Luis Heinzlmeier?
From our infra you can only access things that are in s3://ngi-igenomes, s3://nf-core-awsmegatests , and s3://annotation-cache
l
are you accessing an s3 bucket that is not defined in the CE @Luis Heinzlmeier?
That might be true. This is my first time trying this.
This is all I did so far:
I added a samplesheet for test_full and updated `input` in `test_full.config`. How can I now trigger AWS tests?
r
which branch are you on?
l
👍 1
r
l
test_full.csv
Wait it is missing
r
I don't see it
l
Should be there now!
r
ok resuming with that, and then will check your code
gratitude thank you 1
quick check that the CE is healthy: https://cloud.seqera.io/orgs/nf-core/workspaces/AWSmegatests/watch/1toNW5TLvsno3x/logs then I'll let claude checkour code
ah I know what it is: https://github.com/nf-core/hadge/blob/067f0a18b0bc6422c7665ac6b1e83b36a3bfb5de/nextflow.config#L427 are you on the latest template? This line should be removed and added only for testing: https://github.com/nf-core/tools/commit/40cca1f0da270d34076f7d56dc0de6155b92c491 was release with template: 3.5.2 3.5.0
presumably you are on the right template version though 🤨 https://github.com/nf-core/hadge/blob/dev/.nf-core.yml maybe something went wrong when resolving conflicts
yep the nextflow config was not touched here: https://github.com/nf-core/hadge/pull/103/changes @Nico Trummer confused dog
l
Thank you! I removed 
outdir
from all 
.config
files. Let's wait for @Nico Trummer 🙂
n
Hey, sorry I did not have time to look into this, as I had a submission deadline. Seems I missed that, but you should not have waited for me to fix this, it's quite a small thing. But I will do it now
Done via this commit
l
Thank you, @Nico Trummer! I am currently running the AWS tests. It's looking good so far. https://cloud.seqera.io/orgs/nf-core/workspaces/AWSmegatests/watch/5yjMnxhsg1GbUK/tasks
It failed at 
VIREO
. How can I debug this now? 🤔
n
It gives a very clear error: 
Essential container in task exited - OutOfMemoryError: Container killed due to memory usage
So most likely you just have to adjust the resource requirements
gratitude thank you 1
Details here
l
I see, there are the logs! 😄 Thank you. Try to adjust it...
n
If you look at the failed tasks, you can see that there are three attempts. The first attempt uses 36GB of memory, the second 64, and the third one 64 again. The usual formula for resource scaling is 'initial amount' * 'attempt', so it would scale to 72 and 108, but it seems to be limited by global resource limits, not process-specific ones. So basically this is bottlenecking you
Here, it is easy to adjust the limits, but some users might not be able to do the same. So it might be worth checking the implementation if there is any potential to reduce memory consumption
l
This was the final run on our Cluster at Helmholtz. I'm just wondering why it didn't crash if 
VIREO
took 357.3 GB and my memory limit was 64 GB. 😅
execution_timeline_2026-03-16_22-26-45_test_full.html
n
I think it depends on which containerization and executor you used. While slurm, AWS batch and docker allow setting strict limits, there might be other setups where the limits are not really enforced - I think when using the local executor with conda this might be the case
gratitude thank you 1
l
Here, it is easy to adjust the limits, but some users might not be able to do the same. So it might be worth checking the implementation if there is any potential to reduce memory consumption
In their documentation, they state that p can lead to massive memory consumption.
Copy code
-p NPROC, --nproc=NPROC
                        Number of subprocesses for computing - this sacrifices
                        memory for speedups [default: 1]
We ended up using 18 CPUs due to the 3 retries...
I've limited Vireo's CPUs for now and am running it again.
n
I think this thread is not the right place for discussing too pipeline-specific topics, as all the core team members will be pinged on every message - let's continue this elsewhere
👍 1
❤️ 1
Hey, Luis and I encountered an error we are not familiar with: https://cloud.seqera.io/orgs/nf-core/workspaces/AWSmegatests/watch/3Ylc9xIEIbflS3/logs
Copy code
2:38AM ERR [seqera-fusion] creating temporal file error="open /tmp/fusion_cache_1424582382/-1585F82EE821D5914147834749: no such file or directory" goid=21349 key=/s3/nf-core-awsmegatests/work/hadge/work-cb343aa4a66bb6327b379ce46cf152745773952e/c6/58368ed209150fbec95a1c9b402710/test1/souporcell_minimap_tmp_12.sam.B4000000.chk
I am unsure if this is an issue/hickup with fusion or a problem in the pipeline - I cannot identify a clear issue in the pipeline, and I don't want to spend too much time if I am not sure it is even a pipeline problem. Any insights from the experts? @Rike Hanssen @James Fellows Yates
j
I'm not a platform/fusion expert sorry 🙈 Is this after a retry?
n
No, it's the initial attempt
p
Looks like it's to do with fusion snapshots, I guess
Trying to save you from spot reclamation
n
Ah I just realized that it's actually a timeout error, somehow I missed that before🤦‍♂️
👍🏻 1
2 Views
Open in Slack
PreviousNext
Powered by