Good question. Thanks Coury also for your answer! Here's a few more reasons:
• Daft provides a declarative API to do data processing, which means you can write code to describe the outcome you want, and Daft will determine how to execute it as efficiently as possible to get to that outcome. Think SQL vs writing for loops over a database
• In terms of efficient execution:
◦ Daft vectorizes and pipelines compute to get the most out of your compute resources
◦ Daft has a powerful query optimizer that
▪︎ pushes down filters to reduce the amount of data that actually needs to be read
▪︎ reorders joins to make your query as fast as possible
• Daft provides a unified API for
accessing data sources and
functions for working with them, which makes reading, transforming, and writing data a lot easier
• Daft also has built-in fault tolerance which means if a machine goes down in your cluster, the query will still be able to continue