Hi everyone, I’ve been thinking about best practices for designing Nextflow workflows that operate on very large datasets (multi-terabyte scale, e.g., ~8 TB) stored on AWS. When data resides in S3, what are the recommended approaches for structuring pipelines to minimize I/O overhead and unnecessary data movement while keeping workflows scalable? For example, are there preferred patterns around partitioning inputs, staging data to local scratch space, or executor choices such as AWS Batch? I’d be interested in hearing how others architect workflows at this scale.

The nextflow model to offer portability also means it's not possible to steer this in pipeline code. Trying to optimise too much on this is probably not helpful (and there are competing constraints), but partitioning is probably helpful if possible (both to avoid moving around data that isn't required at that point as well as to improve possibility to process in parallel).

Thanks a lot for this, really appreciate it. Focusing on partitioning makes a lot of sense, both for reducing data movement and improving parallelization. Thanks again for taking the time to share this.

Seems I forgot/decided not to mention, but specifically for AWS, it's possible fusion is an improvement. Or not. There can also be other s3 fuse drivers that could be helpful - or using EFS. There are typically many different ways of addressing issues with a specific implemented pipeline, but if you're still designing things here, partitioning seems a good start that should help generally.