Community for the Daft project and all things distributed data

Distributed Data Community

<@U063GHWEXSM> Made this channel to talk about writing a blog together from this post

<https://dist-data.slack.com/archives/C041NA2RBFD/p1714339992634259>

:rocket: :rocket: :rocket: :rocket: :rocket: 

<@U063GHWEXSM> any thoughts on making a quick LinkedIn/Twitter post of your benchmarks? I’d hate to let you work go to waste… You should just tag us like you just did :stuck_out_tongue: it’s snarky and I love it

&gt; <@U042126MG49>_chia @sammysidhu What did you do to Daft between these versions?! :clap: Or I did something wrong? :grin:
&gt; 
&gt; From time to time I play with my own benchmarks with different versions of some DataFrame libraries (Daft, Polars, DuckDB, DataFusion...).
&gt; Few days back I noticed this drastic improvement in Daft, around 55% in one of the queries on sorting parquet file by a column and writing to disk:
&gt; 
&gt; Here is a plot of the timings. *Daft is actually the fastest engine out of all those benchmarked*. 
&gt; 
&gt; (Link to <https://gist.github.com/|github gist>)


You can find Sammy and I here on LinkedIn:
1. <https://www.linkedin.com/in/chiajay/>
2. <https://www.linkedin.com/in/sammy-sidhu/>


<@U042126MG49> I was away a bit, so let me get back to this and try to figure out how to do this...:thinking_face:

Anyone to suggest other common queries besides the one I did here that typical everyday DE/DS use? I am thinking of adding 2 or 3 other queries/ops to the benchmark should be enough and also to keep it short and concise :thinking_face:

You could try:
1. Simple groupby + aggregation (e.g. `col("x").mean()`)
2. Some data processing (e.g. extracting data from a JSON column using `.json.query("….")`, or truncating strings etc)