Hey spatialvi team, I tried running yesterday with

test_full

on AWS and then realized the samplesheet for that test doesn't exist. What is the current full test that people run on the pipeline?

Hi! We haven't added a full test dataset yet, as we're still just developing and adding the final features to the pipeline. Tests are currently on subset real data, coming from 10x, and the full test data would likely run on the same, just not subset. We're not quite sure how to do this though, as the full data is 15 GB (first dataset for Spaceranger v1) and 85 GB (for Spaceranger 2). This is not even mentioning the work on adding support for Visium HD that we're currently working on, but that data is probably 10 times bigger than that, at least. We're not quite sure what the limits on full test data sizes are, nor where to upload them, but we have sort of waited a bit with that issue 😅 Help would be appreciated!

I believe full test-datasets for pipeline live on s3 and we can help you get it there. That being said, there is certainly still a need to reduce a full sized dataset to a reasonable file size if otherwise it would be TB. Pinging @Edmund since he can help you get your full-sized dataset for tests set up I believe 😊

Yes,

test

is the default that checks all steps (including Spaceranger v2), while

test_spaceranger_v1

and

test_downstream

are other test profiles you can also use for either SP v1 or just the downstream analyses (i.e. no SR)

<!subteam^S044DU0NG68> We are dicsussing full sized test data and where to put it for spatialvi. What is "reasonably" sized? For me it sounds like the actual full sized data for this type of analysis is still too large (~85GB for VISIUM, ~TB for VISIUM HD)...

I think that's a question for <!subteam^S016E36N7TQ> and the S3 bucket

Thanks @James Fellows Yates ! Wasn't sure if that's maintainer or core 😊

Does 10x have public test data hosted somewhere themselves?

Yes, we are currently using their own test data (subset, that is). I haven't seen any minimal data hosted by them, though

But this is

test_full

so that's fine, right?

Yeah. Our current thinking was to use the same test data for the full profile as for the subset ones for consistency, but we haven't finished getting to the HD data yet (development is still ongoing for getting that feature into the pipeline)

ok, but samsampled is easy to self host, so if we can pull directly for the heavy full size test data then that'd be ideal

@Phil Ewels 80GB full test data VISIUM seems fine, but what if the VISIUM_HD data is ~ 1TB raw?! Still want to include that as a full test? 😅

You mean for data hosting? (We could pull direct from there?) or in terms of compute costs of running it?

Yeah I meant more for run costs of testing this massive data 😄