Community for the Daft project and all things distributed data

Distributed Data Community

daft x hf social.png

It's now easier than ever to use Hugging Face :hugging_face: Datasets at scale! Introducing Daft's native support for reading from and writing to Hugging Face.

:arrow_down: *Reading a HF dataset*
With `daft.read_huggingface`, simply pass in the Hugging Face repo name and Daft will handle the rest, no matter the file types or the size of the data.
```df = daft.read_huggingface("EssentialAI/essential-web-v1.0")```
:arrow_up: *Writing to your HF dataset*
Uploading data from Daft to Hugging Face is as easy as calling `DataFrame.write_huggingface` !
```df.write_huggingface("my-org/my-repo")```
Daft automatically takes advantage of &lt;https://huggingface.co/blog/parquet-cdc
|Parquet Content-Defined-Chunking&gt; and &lt;https://huggingface.co/blog/xet-on-the-hub
|Xet&gt; for ultimate write performance.

:question: *What's Next*
• Reading private datasets
• Native support for Hugging Face image, audio, and video data types
• Specifying the dataset split or revision to read from
See the docs here for more info!
<https://docs.daft.ai/en/stable/connectors/huggingface/>