https://nf-co.re logo
Join Slack
Powered by
We’re missing some output files from the full size...
# aws-megatests
s
We’re missing some output files from the full sized oncoanalyser 2.3.0 test in the ‘Results’ tab. If we were to re-trigger the full size test, and assuming it proceeded successfully with all output files generated as expected, would the new outputs replace the existing ones on the ‘Results’ tab?
this 1
p
Yes it should do, the output path is based on the git commit which should be the same. And Nextflow overwrites / writes to the same path.
Can you elaborate on which files are missing? @Rike Hanssen has been busily bashing bugs recently where megatest files have been going missing. Might be helpful.
🙌 1
Note that if the run completed successfully, you should be able to resume and use 100% cached tasks. Then Nextflow will just re-publish the files. This is assuming that the previous run was recent and the work dirs haven't been cleaned up automatically yet.
s
Thanks Phil, and that’s great to hear!
Rayan first noticed that some of the Neo outputs were missing, and I’ve just flicked through and found that many other files are absent:
Copy code
HCC1395/alignments/dna/<several files>
HCC1395/alignments/rna/HCC1395.tumor_rna.md.bam.bai
HCC1395/alignments/rna/HCC1395.tumor_rna.md.bam.metrics
HCC1395/amber/<several files>
HCC1395/bamtools/<all files>
HCC1395/chord/<all files>
HCC1395/cider/<all files>
HCC1395/cobalt/<several files>
HCC1395/esvee/<several files>
HCC1395/isofox/<several files>
HCC1395/lilac/<all files>
HCC1395/linx/{germline_annotations,somatic_annotations,somatic_plots}/<all files>
HCC1395/neo/<all files>
HCC1395/pave/<several files>
HCC1395/peach/<all files>
HCC1395/purple/<several files>
HCC1395/sage/germline/<all files>
HCC1395/sage/somatic/<several files>
HCC1395/sage_append/germline/<all files>
HCC1395/sage_append/somatic/<several files>
HCC1395/sigs/<all files>
HCC1395/teal/<several files>
HCC1395/virsubreakend/<all files>
HCC1395/virusinterpreter/<all files>
The last successful run was about a month or so ago, it would be super convenient if we were able resume and get these results sorted out easily
What do you think is the best way for us to kick off a resume run?
p
Hmm, a month is probably too long. I suspect that they're gone (and it's likely the same bug that @Rike Hanssen has been fighting if its from them - some overly eager intermediate file cleaning that was also cleaning published files).
🙏 1
You can try resuming but probably easier to just kick it off fresh again, you can do that from within github
just make sure that you target the correct tag / commit
s
Got it, and thanks for the advice! I’ll kick that off our tomorrow morning
👍🏻 1
I tried launching via the GH Action but it failed: nf-core AWS full size tests #157
I wasn’t able to see any new oncoanalyser Run in the nf-core AWSmegatests Seqera Platform Workspace either
Any advice / recommendations welcome :)
p
@Rike Hanssen @Edmund @Maxime Garcia did I see some messages from you the other day about API tokens? Any ideas? Looks like the launch failed here
m
I'll have a look
ok, so I'm not sure what happened there to be honest
s
Thank you!!
The run unfortunately failed due to connection issues with staging in data from Cloudflare R2, I tried to resume but was met with this error:
@Maxime Garcia would you be able to trying resume if you think that is the right approach? Otherwise I can relaunch via the GH Action with the 
master
gitref
m
resume would be the right approach indeed
Let me do that
s
The resume looks to have failed due to this change in configuration giving this error in the bwa-mem2 process:
Copy code
Missing AWS Batch job queue -- provide it by using the process `queue` directive
The different configuration as far as I can tell is the following: _*Initial run (cheesy_baggage)*_
Copy code
queue = 'TowerForge-29o9qiv0F5P6DSWkbWgj7w-work'
_*Resume run (cheesy_baggage_2)*_
Copy code
queue = { task.attempt > 1 ? 'TowerForge-6nSba1hDuNFP08JP7SjigJ' : null }
I’m guessing all processes at the start of a run (even when resumed) are considered to have been attempted zero times, so in our second run the queue would be set to 
null
for each processes, which explains the error seen
@Maxime Garcia if you have some time this week, would you be able to take a look at this for us?
m
I'll try to see what I can
🙏 1
r
Disabled more clean up rules 🤞
Hey @Stephen Watts! Experimenting if there is a better solution for the on-demand fallback. In the meantime you can set the actual queue name in the config. Do you have permissions to edit when submitting the pipeline?
Copy code
queue = { task.attempt > 1 ? 'TowerForge-6nSba1hDuNFP08JP7SjigJ' : 'TowerForge-29o9qiv0F5P6DSWkbWgj7w-work' }
s
Hello, thank you for the update! I’m unfortunately unable to edit the config:
r
Can you send the run link then I'll submit quickly
🙏 1
s
The run is over here!
r
s
The updated config applied! The run hit a timeout / interruption on a Cloudflare R2 connection for staging data, so I’ve resumed the run here: https://cloud.seqera.io/orgs/nf-core/workspaces/AWSmegatests/watch/tzLAh5MQn0cvo
Looks to be working 😃
2 Views
Open in Slack
PreviousNext
Powered by