What is the easiest way to find a bottleneck in a distributed query pipeline?
With pretty colors and stats to back it up.
Daft v0.7.8 adds a live heatmap coloring to the query dashboard — slow operators light up red, completed nodes turn green, and arrows trace data flow through the pipeline. You see the hot path the moment a query starts running.
What's new:
\- Dashboard heatmaps — real-time operator coloring by execution time
\- Vectorized grouped aggregation — dense group-id arrays replace per-group index lists, delivering up to 4.4x faster groupbys at high cardinality. ClickBench total dropped 10% below the v0.7.4 baseline
\- Native image\_hash() — 8 perceptual hashing algorithms in Rust, bit-exact with Python imagehash, 5-25x faster than a UDF. For large-scale image deduplication
\- df.skip\_existing() — checkpoint-based row deduplication for idempotent ETL and RAG pipelines
\- DataFrame.skew() and .var() — statistical aggregation convenience methods
\- Subscriber API improvements — nicer public APIs, aimed at OpenLineage support
9 community contributors shipped image hashing, checkpoint operators, statistical aggregations, and more.
blog: https://www.daft.ai/blog/daft-v0.7.8