https://nf-co.re logo
Join Slack
Powered by
I've prepared 272GB (uncompressed) of FASTA files ...
# createtaxdb
j
I've prepared 272GB (uncompressed) of FASTA files for database building for the full test 😅
🤯 1
😮 1
m
272 GB ?
j
Yipu
I wonder if that's too big 😅
m
Do you cover ALL species?
j
Oh god no
Just 63947 genomes from bacteria, virus, and archaea with genomes considered to be 'complete'
I can maybe condense it a bit further (one genome per species, for example ) if we think it's too big
In retrospect maybe I should do that 😅
âž• 1
m
1 per species that's a lot, no?
j
It'll be much less than the 63 thousand
m
true
j
I think I took 'realistic' too seriously possibly 😆
Yeaaaaah I could take it down to about 16k 😅
m
16K vs 272GB
j
No no no
16k genomes vs 63k genomes
m
oh I see
j
So maybe about 100Gb
I could compress them as well to amke it smaller, but it means loads of decompression jobs on teh AWS node
m
yeah, no need for that
Nice
j
Or one job but which requires a large node I presume
m
I'll call you when I need MASSIVE data for stress tests
😆 1
j
Actually what Would be better for the megatest
Upload 100GB to bucket uncompressed data, skip decompression on compute node
OR
Upload much smaller compressed data to bucket, perform decompression on compute node
m
If I'm not mistaken compute is more expensive than storage
At this level I mean
j
OK cool, so I go with the former (once I've reduced it a bit)
m
also, I assume you'd want to use this data many time
j
Well it's for the AWS full test
Not that I imagine many releases
m
you never know
2 Views
Open in Slack
PreviousNext
Powered by