Flotilla 🐟 ⚔️ 🐟 is here! Scaling multimodal pipelines is fundamentally different from scaling tabular analytics: 🖼️ Images + video inflate memory 100x ⚡ GPUs + CPUs need coordinated scheduling 📦 Object stores become bottlenecks Most distributed engines weren’t built with multimodal workloads in mind. Introducing Flotilla, Daft’s new distributed execution engine. Key design changes: • Micro-batch + streaming execution • Arrow Flight shuffle (measured up to 10× faster than Ray’s object store) • Better observability for debugging pipelines The goal: make multimodal pipelines reliable and scalable without endless tuning. https://www.daft.ai/blog/introducing-flotilla-simplifying-multimodal-data-processing-at-scale