Is this new? Seems like a native AWS replacement f...
# random
g
Is this new? Seems like a native AWS replacement for fusion. https://aws.amazon.com/s3/features/files/
đź‘€ 1
a
My initial thoughts: It’s not quite the same thing, although it is somewhat overlapping. Fusion uses the on-instance storage (preferably super-fast NVMe) and is per-instance. This S3 Files is similarly caching the data “locally” and exposing a POSIX interface to S3 data but using EFS as the underlying “local” storage mechanism. This is actually a shared (within your parameters) NFS filesystem which is going to be much slower than the on-instance storage (EBS or NVMe) and not have per-instance isolation AFAICT. I think something like archil is more similar to Fusion, although they strongly want to differentiate themselves from being just an S3 Fuse wrapper and highlight that they are actually managing hardware (SSD) themselves. I am also deeply sceptical of a YC backed company which has plastered “AI” and “agents” all over its marketing (was not really present when I first came across them). There is the little-documented xpack-amzn but that requires a license too and I’m unclear what the support is like. Nextflow itself makes working with some of these other options difficult also because you have to specify the actual data path for anything other than
fusion
which is special-cased to it’s own mountpoint when enabled, which requires
wave
. The free usage limits on
fusion
are also quite low and there doesn’t appear to be an easy or transparent way to access
fusion
as a service right now, although these docs suggest that it may be possible to arrange stand-alone usage. So far personally the friction to start using any of these over simply paying for giant EBS volumes has not been worth it for me.
p
The numbers they present suggest this is built on Lustre rather than just NFS (although I suspect they also use NFS interfaces as a fallback to get around dealing with problematic kernels). And I'm quite certain they can get numbers beating EBS backed fusion (and quite probably NVMe as well). I suspect there's still a lot being missed out on that are gains with local filesystems (rather than storage, e.g. knowing that noone else is missing with stuff allows for more aggressive caching than makes sense with non-token-network file systems). I'm often bad at guessing what people will do, but to me this seems like AWS hearing that some customer wanted to get file access to S3 and wanted CTOs to feel they get something better when they pay a lot to use S3 files instead of a FUSE mount 🙂
a
Hmm I got the EFS/NFS thing from that page:
S3 Files is built using Amazon EFS, which intelligently loads your active working set onto high-performance storage
But maybe there is more to it than that
p
Unless they've changed things, EFS can be either NFS or Lustre.
(looking now they seem to have migrated to the term "FSx for Lustre", but the numbers they state seem much larger than what is listed for EFS.)
It is backed by EFS, but with some neat optimisations such as
When you first access a directory, S3 Files imports metadata from S3 and populates a synchronized view. For files under 128 KB it also pulls the data itself. For larger files only metadata comes over and the data is fetched from S3 when you actually read it. This lazy hydration is important because it means that you can mount a bucket with millions of objects in it and just start working immediately.
Or
One of the improvements that I think is really cool is what we are calling “read bypass.” For high-throughput sequential reads, read bypass automatically reroutes the read data path to not use traditional NFS access, and instead to perform parallel GET requests directly to S3 itself, this approach achieves 3 GB/s per client (with further room to improve) and scales to terabits per second across multiple clients.