https://nf-co.re logo
Join Slack
Powered by
Regarding this: <https://nfcore.slack.com/archives...
# team-maintainers
e
Regarding this: https://nfcore.slack.com/archives/C043UU89KKQ/p1764239828991889?thread_ts=1761835217.787399&amp;cid=C043UU89KKQ Which would be the proposed course of action? 1. skip AWS results for v2.1.0 2. remove test_full strict memory assignment and patch release 2.1.1 3. any other proposed way?
m
can you rerun the aws run but with the settings overwritten?
e
how can I do that?
If I could comment all the mem settings in the test_full.config via the platform, that could do the trick..but can I?
m
when you relaunch the failed run (three dots on the top right), you should be able to set them in advanced settings
r
you can provide a config section here:
e
let me try
r
I am wondering in general though if it makes sense to be as strict as you are in the test_full profile as people may run this on a variety of infestructures to test out the pipeline. not saying the default would be best, but maybe some sort of in between solution here for a smoother user experience? @Florian Wuennemann I know you are very passionate about this.
e
everything is grayed out for me at the advanced settings tab, I guess need other permissions
r
hm yes, will temporary bump you so you can fix the resource requests
e
Does output dir need to be something specific?:
m
<s3://nf-core-awsmegatests/proteinfamilies/results-f398731e665c8f36034af0fb32d88187880aef0e>
e
Alright, launching!
🤞🏻 1
m
no luck
e
Copy code
Caused by:
  Can't stage file <https://github.com/nf-core/test-datasets/raw/proteinfamilies/test_data/mgnifams_input_small.faa> -- reason: Failed to complete Amazon S3 multipart upload
no idea what this is
r
shouldn't this be a raw link? like so: 
<https://raw.githubusercontent.com/nf-core/test-datasets/proteinfamilies/test_data/mgnifams_input_small.faa>
m
true
e
My input samplesheet on the platform is 
<https://raw.githubusercontent.com/nf-core/test-datasets/proteinfamilies/samplesheets/samplesheet_full.csv>
The samplesheet contents have always been the same and were working in previous releases
m
what's your 
params.pipelines_testdata_base_path
e
In the main nextflow.config
m
but it resolve to the same raw file, so shouldn't matter
e
Should I just retry once more like this? Or is it some other AWS setting maybe?
r
I tried with the raw link but same error. it failed twiecc but worked on the third try. I don't know why: https://cloud.seqera.io/orgs/nf-core/workspaces/AWSmegatests/watch/3xsSBJZBIYvHfD/v2/logs seqera AI is suggesting some intermittent network issues. How big is that faa file? Maybe we should just move it to s3 where all of our proper megatests are
m
mgnifams_input_small.faa
is 10MB, so shouldn't be a problem
r
then idk, maybe raise it on the nextflow slack
m
something is a bit weird in this log, why does it retry after staging?
Copy code
WARN: Unable to stage foreign file: <https://github.com/nf-core/test-datasets/raw/proteinfamilies/test_data/mgnifams_input_small.faa> (try 1 of 3) -- Cause: Failed to complete Amazon S3 multipart upload
WARN: Unable to stage foreign file: <https://github.com/nf-core/test-datasets/raw/proteinfamilies/test_data/mgnifams_extra.faa> (try 1 of 3) -- Cause: Failed to complete Amazon S3 multipart upload
Staging foreign file: <https://github.com/nf-core/test-datasets/raw/proteinfamilies/test_data/mgnifams_extra.faa>
Staging foreign file: <https://github.com/nf-core/test-datasets/raw/proteinfamilies/test_data/mgnifams_input_small.faa>
WARN: Unable to stage foreign file: <https://github.com/nf-core/test-datasets/raw/proteinfamilies/test_data/mgnifams_input_small.faa> (try 2 of 3) -- Cause: Failed to complete Amazon S3 multipart upload
WARN: Unable to stage foreign file: <https://github.com/nf-core/test-datasets/raw/proteinfamilies/test_data/mgnifams_extra.faa> (try 2 of 3) -- Cause: Failed to complete Amazon S3 multipart upload
WARN: Unable to stage foreign file: <https://github.com/nf-core/test-datasets/raw/proteinfamilies/test_data/mgnifams_extra.faa> (try 3 of 3) -- Cause: Failed to complete Amazon S3 multipart upload
WARN: Unable to stage foreign file: <https://github.com/nf-core/test-datasets/raw/proteinfamilies/test_data/mgnifams_input_small.faa> (try 3 of 3) -- Cause: Failed to complete Amazon S3 multipart upload
🤷 1
e
Thanks @Rike Hanssen and @mashehu, it finally succeeded. I guess the results will show on the website tomorrow, right?
r
so I think that because you are using the latest nextflow version you are testing out a bunch of new things. It looks like there is new `*`aws.client.minimumPartSize``* that defaults to 8MB, so your upload ends up in that range. For why it retries, I found this: https://github.com/nextflow-io/nextflow/blob/bcf81d907d354958748ad052efca6f3fa0f3e[…]odules/nextflow/src/main/groovy/nextflow/file/FilePorter.groovy but I am not so familiar with the nextflow code base to give you a good answer
m
e
The most important folder for which this release was made is not there :'D The 
proteinfold
folder with the downstream samplesheet, made with the new workflows output syntax
m
is that a top level folder?
e
it should be alongside the rest of the folders, yes. again not sure if the output block in AWS behaves differently and by default saves results somewhere else. This was my block in main.nf:
Copy code
output {
    proteinfold_samplesheet {
        path { sample -> "proteinfold/${sample.id}/" }
        mode params.publish_dir_mode
        enabled !params.skip_proteinfold_samplesheet
        index {
            path 'proteinfold/samplesheet.csv'
            header true
            sep ','
        }
    }
}
should be something like this, and I just figured out that the update_families folder is also missing, which doesn't even use the workflow output syntax
m
don't see it on the aws s3 bucket either...
e
I think I might know what's going on
The params from my test_full.config didn't register, and it skipped stuff
Maybe I need to re-define them in the advanced settings config?
(or alter them in the -run params page-
r
hm that is strange. ideally they should be picked up from the test_full config.
e
I re-ran (twice) and it finally finished again
r
which param is not picked up?
e
none of them
I inserted them manually
r
hm then we need to figure it out.
e
previous releases worked fine again. not sure if we changed the config in advanced settings this time
r
it works for me on a fresh run
e
Hmm can you do my a favor and re-run it
r
yep one sec need to get back to my laptop
e
Sorry second, meesage was clipped
reurn it but make sure it uses samplesheet_full.csv as input (I used samplesheet.csv by mistake on last try)
setting NXF_VER and rerunning
is it even possible to change 
aws.client.minimumPartSize
 manually form the launchpad? :')
4th time I am relaunching...
r
yep in the config
e
I am trying with aws.client.minimumPartSize='30 MB' in the config..
fail..
I'll stop spamming it for now, until we tackle this one, and will revisit
r
where is the run where you bumped this?
e
Should be 
cheeky_bernard
:
thankyou 1
r
Fix was released in 25.10.1 earlier today party dead . I am testing it here and so far not a single issue with staging: https://cloud.seqera.io/orgs/nf-core/workspaces/AWSmegatests/watch/2TNNBeJAYiJGfe/v2/logs
😍 1
e
Awesome thanks Rike, succeeded! :D
How do we now change the 2.1.0 AWS website results to the new ones?
m
a rebuild should fix it one sec
e
Let me know if you do that, I still see the old results. Probably need to search at a new link for the updated ones? How does it work?
Almost all nice and well, the results have been updated: https://nf-co.re/proteinfamilies/2.1.0/results/proteinfamilies/results-f398731e665c8f36034af0fb32d88187880aef0e/ But! Still the 
proteinfold
output folder that is supposed to be generated when 
skip_proteinfold_samplesheet = false
(which is in the latest AWS run), with the new workflow output syntax, isn't there 😞
It's fine for this release, but let's just keep in mind, in case this is a bug and pipelines change to the new output syntax and then resutls dont show
I am curious though, have you seen any other pipeline so far that uses the new output syntax and the files appear on the AWS results on the website? Is it a "me" thing?
r
I think you might be the first one that uses the new output syntax and has a release out
e
Returning to this...during some training I was carrying out, I found out that despite the --outdir that I provide the pipeline, the output syntax stuff files are getting published in a folder named 
results
instead (and I can't find out why)..can we somehow see if this is the case with the nf-core/proteinfamilies AWS full_test as well?
Any chance that 
results
as an output directory is somewhere hardcoded someplace that the new output syntax uses?
r
e
It's missing the 
proteinfold
output directory. Can you see if it is in a folder named 
resutls
next to the publish dir?
Which is the only thing created with the new output syntax
r
are you not able to navugate in the data explorer?
m
Wasn't there a chance with the outdid params, @Maxime Garcia?
e
So here: https://cloud.seqera.io/orgs/nf-core/workspaces/AWSmegatests/data-explorer/v1-cloud-[…]e?credentialsId=yMaeZgeycjtgq6F36qyoA&amp;path=proteinfamilies%2F there is no 
results
folder. It is named 
results-f398731e665c8f36034af0fb32d88187880aef0e/
and is properly copying/publishing files according to the old way of defining publishDirs per module. But I have a hunch that the new output syntax always publishes things in a folder named exactly 
results
, which is not the same. Does this make sense? :P
r
yes we are naming it like this on purpose when running pipelines for release tests so that each results folder is matched to a release. The suffix matches the git sha of the release
no I am confused 😄 I checked the bucket and didn't find a 
results
folder
e
exactly, because the new output syntax is probably publishing it on a different folder next to the outdir folder.
m
let me check
I added that in #C073ZDLNC92 to make sure to map params.outdir back to the outdir for workflow output: https://github.com/nf-core/references/blob/dev/nextflow.config#L54-L56
⭐ 1
I think one might be able to add the outputDir via a custom config
r
I still don't get it 😄 but if you and maxime understand each other I'll see myself out
😂 1
m
@Evangelos Karatzas did you catch my idea?
e
Yep, will try with the next release. Can you also add that to the template maybe?
It works locally 😉
m
Not there yet with workflow output in the template I'm afraid
e
No worries! Just keep in mind for the future. Thanks 😄
m
Yeah I tried mentioning that to everyone starting to play with workflow output, but I probably missed you
Open in Slack
PreviousNext
Powered by