https://nf-co.re logo
Join Slack
Powered by
Hi everyone, Quick question regarding some public ...
# microscopy
f
Hi everyone, Quick question regarding some public microscopy image datasets. Are there any larger datasets (10GB-1TB) available on AWS S3 buckets with public read access, such as the 1000 Genomes for genomics (https://registry.opendata.aws/1000-genomes/)? I saw that the exemplar datasets are on S3, but don't have public read access on the bucket and only on https, right?
👀 1
a
I don’t think I have any from HTAN that have open bucket access. That said the MCMICRO EMIT datasets (cc @Jeremy Muhlich) have been relatively popular on Synapse and may be good candidates for an internal program where we are able to make some data also available through AWS open data sets. I’ll open a ticket at Sage to open that question (but likely a solution for a few months time) rather than immediate
What about with IDR?
f
What is IDR? 😳
a
Or maybe here. But probably too small and likely in OME-ZARR. https://www.ebi.ac.uk/bioimage-archive/
f
Thanks Adam, those are great resources! I will check them out and report if I find any S3 buckets that are readable by public access!
j
We have free data egress on AWS so it would not be a problem to host some raw data. The S3 storage costs are minimal.
🎉 1
f
So do you think it would be possible to put exemplar-001 and exemplar-002 on S3 buckets publicly facing @Jeremy Muhlich? Or did I misunderstand your suggestion?
j
yes
We have the stitched images there already but not primary raw tiles
Were you looking for raw tiles or stitched images?
Oh wait, we do have the raw images. They have been converted from the original RareCyte .rcpnl format to .ome.tif (but a multi-image tiff with the raw tiles)
Copy code
<s3://mcmicro/exemplars/001/exemplar-001/raw/>
<s3://mcmicro/exemplars/002/exemplar-002/raw/>
headbangingparrot 2
What led you to believe that bucket did not have public read access? If there is any outdated/incorrect documentation out there I'd like to fix it.
The exemplars are still not 10GB+ though. exemplar-002 is 3 GB
We have the entire TMA it was cropped from but I forget if it's in S3. I will check.
Oh right, so it's the full TMA that is only available online in pre-stitched form, and even that is only on Synapse.
f
Ahh seems they are available to read publicly. I think I reverse engineered the S3 links wrong from the https links yesterday... I am still new to S3 so sometimes make dumb mistakes 😉
This works: 
aws s3 ls <s3://mcmicro/exemplars/001/exemplar-001/raw/>
👍 2
k
Not sure this is what you’re looking for, but we have several petabytes of microscopy data in public AWS Open Data buckets: https://registry.opendata.aws/janelia-flylight/ https://registry.opendata.aws/janelia-mouselight/ https://registry.opendata.aws/janelia-cosem/ These are currently mostly stored in various homegrown formats, but there are open source tools that work with all these formats (e.g. Fiji supports H5J and N5). We’re also in the process of publishing a new 50 TB data set that will be in OME-Zarr format. The Allen institutes also have a lot of imagery in public buckets, e.g.: https://registry.opendata.aws/allen-cell-imaging-collections/
👍 2
🤩 1
f
Thanks @Konrad Rokicki! Those are amazing resources to have!
k
No problem! Let me know if you have any questions about any of the formats.
gratitude thank you 1
Open in Slack
PreviousNext
Powered by