https://nf-co.re logo
Join Slack
Powered by
A new request to upload a dataset to nf-core AWS S...
# request-core
t
A new request to upload a dataset to nf-core AWS S3 bucket has been made! • Requested by: @Stephen Watts • Pipeline name: nf-core/oncoanalyser • Why is this needed?: Full size test for the 
wgts
mode from FASTQ: - normal WGS (40x) - tumor WGS (80x) - tumor WTS (4x) (migration of data was requested in previous release review) • Size of the dataset: ~260 GB • URL: https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__normal_wgs__WGS_IL_N_1__SRR7890859__subsampled.1.fastq.gz https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__normal_wgs__WGS_IL_N_1__SRR7890859__subsampled.2.fastq.gz https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__normal_wgs__WGS_IL_N_1__SRR7945633__subsampled.1.fastq.gz https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__normal_wgs__WGS_IL_N_1__SRR7945633__subsampled.2.fastq.gz https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__normal_wgs__WGS_IL_N_1__SRR7945634__subsampled.1.fastq.gz https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__normal_wgs__WGS_IL_N_1__SRR7945634__subsampled.2.fastq.gz https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__normal_wgs__WGS_IL_N_1__SRR7945663__subsampled.1.fastq.gz https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__normal_wgs__WGS_IL_N_1__SRR7945663__subsampled.2.fastq.gz https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__tumour_wgs__WGS_IL_T_1__SRR7890856__subsampled__split_1.1.fastq.gz https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__tumour_wgs__WGS_IL_T_1__SRR7890856__subsampled__split_1.2.fastq.gz https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__tumour_wgs__WGS_IL_T_1__SRR7890856__subsampled__split_2.1.fastq.gz https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__tumour_wgs__WGS_IL_T_1__SRR7890856__subsampled__split_2.2.fastq.gz https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__tumour_wgs__WGS_IL_T_1__SRR7890856__subsampled__split_3.1.fastq.gz https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__tumour_wgs__WGS_IL_T_1__SRR7890856__subsampled__split_3.2.fastq.gz https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__tumour_wgs__WGS_IL_T_1__SRR7890856__subsampled__split_4.1.fastq.gz https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__tumour_wgs__WGS_IL_T_1__SRR7890856__subsampled__split_4.2.fastq.gz https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__tumour_wgs__WGS_IL_T_1__SRR7890856__subsampled__split_5.1.fastq.gz https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__tumour_wgs__WGS_IL_T_1__SRR7890856__subsampled__split_5.2.fastq.gz https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__tumour_wgs__WGS_IL_T_1__SRR7890856__subsampled__split_6.1.fastq.gz https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__tumour_wgs__WGS_IL_T_1__SRR7890856__subsampled__split_6.2.fastq.gz https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__tumour_wgs__WGS_IL_T_1__SRR7890856__subsampled__split_7.1.fastq.gz https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__tumour_wgs__WGS_IL_T_1__SRR7890856__subsampled__split_7.2.fastq.gz https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__tumour_wgs__WGS_IL_T_1__SRR7890856__subsampled__split_8.1.fastq.gz https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__tumour_wgs__WGS_IL_T_1__SRR7890856__subsampled__split_8.2.fastq.gz https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__tumour_wts__2891075264__SRR892423__subsampled.1.fastq.gz https://pub-349bcb8decb44bf7acbddf90b270a061.r2.dev/HCC1395-SRA/25.0/data/wgts/fastq/HCC1395__tumour_wts__2891075264__SRR892423__subsampled.2.fastq.gz • Related files: Core team will add this dataset to the nf-core bucket. Please react to the message with ✅ when this is done. [This process is not automated]
j
@Anabella Trigila don't forget (in case you've not seen it yet)
a
I don't have access to the bucket yet, but perhaps now @Ken Brewer can take over.
s
Let me know if I can do anything else to help this along!
j
<!subteam^S016E36N7TQ> please can someone with s3 Access please help Stephen? It's been more than Week now. (Sorry @Stephen Watts - most of the core team are on some form of leave - myself included 😬)
❤️ 1
m
I'll try to have a look later today
j
Reminder @Maxime Garcia
👀 2
p
@Marcel Ribeiro-Dantas as you're now on call, please could you take a look at this?
m
Yes!
🙏 1
s
Thanks all! Did this go through on Friday or is it planned for this week?
I'll update the oncoanalyser full-sized test to use the migrated data paths, so please let us know the S3 URIs when they're available (:
m
I believe it’ll be worked on this week, Stephen. I was unable to do so, but we have someone else in charge this week 😄
❤️ 1
m
I'm on it, I should be able to do things before looking at anything else
❤️ 5
thankyou 1
Data is now in: s3://nf-core-awsmegatests/oncoanalyser/input/HCC1395-SRA/ @Stephen Watts I hope the path make sense
🙌 2
🙏 1
you should be able to list with : aws s3 ls --no-sign-request --human-readable s3://nf-core-awsmegatests/oncoanalyser/input/HCC1395-SRA/ and I think I got all 26 files
p
Is this for full-size tests?
How much data is this?
It looks like quite a lot. Note that the nf-core AWS account (credits) pay for all data egress every time this data is pulled. Please don't make full-size tests utterly enormous 😬
j
The second question's answer is already in the OP (260gb)
p
If they are massive and we do definitely want them, we should discuss moving them somewhere different
r
would downsampling to 10X or so still result in useful outputs?
party this 1
p
260GB means it costs ~$25 per 
-profile test_full
in just data egress. Probably fine, until people want to run benchmarking to see if different platforms / execution tweaks makes the runs faster. Then it's easy to rack up 50 runs and suddenly it's cost $1,250 in egress.
This isn't extreme and we do have credits to cover, but it's worth a quick sanity check that it's needed.
And whether we should sync to the igenomes bucket as we have with some other pipelines, where we don't pay for egress 👀
party this 2
o
It's the reason we had them in Cloudflare -- to avoid the egress costs.
igenomes would make sense, though; those are also standard cell lines that might be useful to others.
s
Happy for this to be an nf-core team call
The full-sized data on my side was aiming to give users a feel for how an 80x tumor / 40x normal run behaves in oncoanalyser, what real outputs look like along with runtimes and resource usage via the nf-core pipeline 'Results' page
I had subsampled to those depths deliberately, so they can be certainly be adjusted if needed. Only thing I'd flag is that changing them moves us away from the original aim of showing what a typical tumor / normal run looks like at full size
Along that line of thinking, there are also minified readsets that run under the test profile, these give mostly useful outputs with a much smaller footprint in both size and compute, which cover the 'fast and light' test case
To be upfront I'd rather keep this data on Cloudflare R2, mainly so nf-core doesn't need to pick up the egress costs and so we can readily update or expand the dataset when needed
On the Cloudflare R2 front, I'll take the opportuity to draw attention to this Nextflow PR (#7494), which attempts to address one of the difficulties we’ve had with interrupted R2 connections: • Implement resumable staging of foreign files from HTTP sources - #7494
👀 1
We had been holding the oncoanalyser 3.0.0 release for the data migration, but this might need further discussion to reach a consensus, so if no objections we'll descope a potential switch of full-sized test data to 3.1.0
p
Yeah generally we advise people to point 
test_full
config profiles to the ENA or similar as egress is free then. But those connections can be wobbly, which can be irritating. Hence we have moved a few pipelines over to s3 over the years.
Ok so our options are: 1. Go ahead with aws-megatests and monitor bandwidth, react if needed 2. Stick with r2 and deal with flaky downloads if they happen 3. Move to aws-igenomes bucket
Let's do (3). It's fast and means we can move ahead now without any concerns.
this 2
m
Can't we use the same test data that we already have for sarek, we do have a somatic test
s
The full-sized test dataset includes matched WTS data :)
p
I'm running the transfer to the aws-igenomes bucket now
@Stephen Watts could you please write a quick README.md to upload alongside the data explaining the provenance, any processing and where it's used? 🙏🏻
ok, it's up:
Copy code
<s3://ngi-igenomes/test-data/oncoanalyser/>
I'll wipe the copy on aws-megatests now to avoid any accidental use
Done
o
Thank you!
👍🏻 1
s
@Phil Ewels thanks! Here is a README.md plus SHA256 hashes for each of the files in a compressed tarball:
README.mdseqcii-hcc1395-sha256-hashes.tar.gz
And is there a preference to use the S3 or HTTP URIs for the iGenomes bucket?
m
s3 for the samplesheet I'd say, that's what we usually do
🙏 1
s
Got it, I'll update the samplesheet
p
Nice one, thanks!
Uploaded ✅
2 Views
Open in Slack
PreviousNext
Powered by