😏 Chat Completions have arrived! As of daft release 0.6.13, the

prompt

function supports sending requests via either the

v1/chat/completions/

or

v1/responses

endpoints to any OpenAI API compatible Server. Since it's introduction, the OpenAI API has been adopted by a majority industry as the standard means of chatting with Language Models. This includes use-cases like structured outputs where users can specify Pydantic Models to coerce model outputs to a specific schema. We’ve added compatibility with the completions endpoint so you can still leverage prompts structured outputs even if your provider doesn’t yet support responses. Additionally, as you can see below, prompt now supports

daft.DataType.Images

and

daft.Files

. In fact, the actual OpenAI Responses API only works with PDFs, but with

prompt

, @Colin Ho added the ability to inject any

text/*

mime type into the prompt context. This means that any markdown, html, or csv file you reference in a prompt is automatically added to your prompt input wrapped in xml tags!

# /// script
# description = "Prompt Gemini 2.5 Flash Lite on OpenRouter with Images and Files"
# requires-python = ">=3.10, <3.13"
# dependencies = ["daft>=0.6.13", "openai", "numpy", "pillow"]
# ///
import os
import daft
from daft.functions import prompt, download, decode_image, file

daft.set_provider(
    "openai",
    base_url="<https://openrouter.ai/api/v1>",
    api_key=os.environ.get("OPENROUTER_API_KEY")
)

df = daft.from_pydict({
    "prompt": ["Whats in this image and file?"],
    "my_image": ["/Users/MyName/Desktop/SomeImage.png"],
    "my_file": ["/Users/MyName/Downloads/SomePdf.pdf"],
})

# Decode the image and file paths
df = df.with_column(
    "my_image",
    decode_image(download(daft.col("my_image")))
)

df = df.with_column(
    "my_file",
    file(daft.col("my_file"))
)

# Prompt Usage for Chat Completions with Images and Files
df = df.with_column(
    "result",
    prompt(
        messages=[daft.col("prompt"), daft.col("my_image"), daft.col("my_file")],
        system_message="You are a helpful assistant.", 
        model="google/gemini-2.5-flash-lite",
        provider="openai",
        use_chat_completions=True,
    )
)

print(df.select("result").to_pydict())

There has been a tremendous amount of development on the prompt api recently to get to this point. We can't wait to see what you build with

prompt

!

Hello all! I know its been a bit quiet over here. We've been waiting to hear back from PyPi on raising our limits for pushing new releases. In the meantime, we just launched a podcast, Zero Shot Espresso, where we explore the real challenges teams face with data and AI. In our first episode, Jay sits down with data legend for a deep and surprisingly funny dive into the future of data infrastructure. Josh traces his journey from the early days of on-prem warehouses and Hadoop hype to the rise of multimodal data that is reshaping how modern AI systems work. https://open.spotify.com/episode/6Rgxc7chwPmBgFXWTsRred

Daft 0.7.0 has been released! This was a massive update for Daft due to a confluence of conditions beyond our control, so while I'll take a moment to highlight a few core features and breaking changes, please see the full change log with the link above! ✨*Major Features* • New dynamic batching algorithm for UDFs • New google provider for

prompt

• New daft.AudioFile subtype to daft.File !!! Additionally 0.7.0 comes with a few important removals • removal of daft spark connect 💥 BREAKING CHANGE • removal of the legacy ray runner We've been waiting to hear back from PyPi to increase our quota for storage, but responses have been slow. For this reason we have needed to delete an older release with an existing duplicate under the old

getdaft

package. If you are using daft==0.4.7, please switch to getdaft==0.4.7 as daft release 0.4.7 no longer exists. 🎄 As always lets celebrate all of the hard work from our amazing contributors that have made this project possible!

Daft has just reached 5000 stars on Github!

🎄As many of you know the daft team will be going on holiday from December 20th through January 4th. For more details on how this affects the project, please see the pinned an issue made by @Cory Grinstead on Github: https://github.com/Eventual-Inc/Daft/issues/5861

Daft 0.7.1 was released on December 18th! Here is what changed: ✨ Features feat: Support pattern filtering for SHOW TABLES @aaron-ang (#5423) feat(docs): add copy page as markdown button @ykdojo (#5828) feat: support overwrite files by native IO @stayrascal (#5728) feat: Add support for Series[start:end] @everySympathy (#5815) feat: Implement retry-after mechanism for model apis and udfs @colin-ho (#5769) 🐛 Bug Fixes fix: using estimate memory bytes at first for display scan task source @stayrascal (#5845) fix: Remove pop_all assertion @colin-ho (#5850) fix: Check if deletion vector propagation is supported in deltalake @cckellogg (#5829) fix: Set default ImageMode in decode_image to RGB @colin-ho (#5827) fix: handle FileNotFoundError in read_huggingface fallback @ykdojo (#5831) fix: Add overflow protection to memory estimation @yudduy (#5417) ♻️ Refactor refactor(arrow2): get field and dtype roundtrips working @universalmind303 (#5849) 📖 Documentation docs: improve docstrings of IO read methods for remote URLs @aaron-ang (#5841) docs: fix broken links causing CI failure @ykdojo (#5832) 👷 CI ci: enable Windows Rust tests on PRs @ykdojo (#5823) ci: skip quickstart notebook in notebook-checker workflow @ykdojo (#5804) 🔧 Maintenance chore: remove overwrite_files & write_empty_tabular method @stayrascal (#5838) chore: Fix a minor ambiguity in the README docs @plotor (#5830)

📣 NEW BLOG for daft.File: AI pipelines don't just run on Parquet files anymore. Audio, video, PDFs, source code — they're all central to modern workflows, but they're large, remote, and expensive to load eagerly. daft.File brings lazy, distributed file handling to all of them: • One interface. • Local or remote. • Open only when needed. In today's blog I walk through real examples: resampling audio, extracting video keyframes, parsing PDFs, and building code intelligence pipelines with AST — all inside Daft DataFrames!

🚢 Daft v0.7.3 is out! https://www.daft.ai/blog/daft-v073-otel-for-flotilla-nightly-builds-and-lance-nn-search 🪝 @Srinivas Lade shipped distributed observability in Daft v0.7.3. df.metrics now returns per-node execution stats as a RecordBatch from Flotilla and Ray runs, same API as local, just from a cluster. Added standard OTEL environment variable support so metrics route to your existing collector without code changes. This is milestone 2 of 4 on the observability roadmap. ⛴️ What else shipped: - Nightly builds at nightly.daft.ai. - Lance namespace read/write with native nearest-neighbor vector search - Zero-copy arrays from Rust vectors and smarter UDF serialization uv add daft==0.7.3

🏛️ Important Governance Updates for Contributors As the Daft community has grown, from a handful of contributors to a community of developers, users, and organizations, we've reached a point where our informal processes no longer match the scale of what we're building together. As a reminder, as of PR 6117, the project is adopting a tiered governance structure with three distinct roles: • Contributor: Anyone who has made a contribution to the Daft project. This includes code, documentation, bug reports, community support, and more. • Maintainer: A recognized contributor who has demonstrated sustained, meaningful contributions over time. Maintainers are nominated by existing Maintainers or PMC members and approved by a majority PMC vote. ◦ Maintainer/Reviewer: Recognized contributors with review and approval permissions, but without merge access. These are trusted voices in code review who help shape the quality of the project. ◦ Maintainer/Merge: Maintainers with merge access to the repository. They carry the additional responsibility of ensuring that what lands in main meets the project's standards. • PMC (Project Management Committee): The highest governance tier, responsible for the overall direction of the project. The PMC has full authority over project decisions, including the nomination and approval of new Maintainers, release processes, and strategic direction. We've also improved codeownership to make it easier for contributors to push changes into the main branch. With this change most of the daft codebase is now reviewable by community maintainers. Finally, in order to ensure a uniform issue triage process, all non-trivial PRs are required to be linked to an issue that has been approved by a maintainer. This also means, non-trivial PRs that do not meet the requirements may be closed without review. Trivial PRs (docs, typos, small test fixes, clearly scoped bugfixes) may bypass the issue requirement, but must be labeled no-issue-needed by a maintainer. See the Contributing Code for what counts as non-trivial. Full Details can be found the Contributing Guide

Daft 0.7.4 has been released! https://www.daft.ai/blog/daft-v074-arrow-rs-opendal-flight-shuffle-and-better-metrics Here are the major features that shipped: • The Great Arrow Migration has completed (only IO modules remain) HUGE shoutout to @Cory Grinstead for pushing this through the finish line. • Metrics got a glow up with consolidated naming, new docs, split duration support, and dashboard cli improvements. • Apache OpenDAL is now Supported for unified data access across dozens of storage services. • Flight Shuffle on Flotilla for more performant zero-copy data movement between nodes. Upgrade with:

uv add "daft=>0.7.4"

New from the blog: ***How We're Making Observability Better in Daft*** @Srinivas Lade wrote up the observability framework we're building — covering local debugging, cluster monitoring, and post-incident analysis. The post breaks down four categories (internal monitoring, DataFrame APIs, dashboard, and OpenTelemetry) and what's coming next (EXPLAIN ANALYZE-style query analysis, plan DAG visualizations, real progress bars). Worth a read if you've ever wondered what your Daft pipeline is actually doing: https://www.daft.ai/blog/how-were-making-observability-better-in-daft Feedback welcome in https://github.com/Eventual-Inc/Daft/discussions/5155

📣 Daft 0.7.5 has been released! Read more about it in our blog: https://www.daft.ai/blog/daft-v075-a-plugin-system-5x-faster-parquet-and-a-real-time-query-debugger Lets give a big 🙏 to all of our amazing contributors!

It's week 4 of our UDF series, and I realized we haven't been circulating them here on the community slack 🙀 ! Don't worry here are the links: • Wk 1: Intro to daft UDFs https://www.daft.ai/blog/daft-udfs-what-is-a-udf-and-why-do-you-need-one- • Wk 2: Stateless daft.func https://www.daft.ai/blog/stateless-udfs-with-daftfunc • Wk 3: Statefull daft.cls https://www.daft.ai/blog/stateful-udfs-with-daftcls • Wk 4: GPU Inference UDFs https://www.daft.ai/blog/gpu-inference-with-daftcls

added an integration to this channel: Ordinal

Work with any file, anywhere! For all of you building Claude Code personal operating systems in markdown, daft.File is a lazily evaluated file reference with subtypes designed for unstructured data at scale (think Markdown, Audio, Video, Code, PDFs) Just run

daft.from_files(uri)

and daft will glob your skills into a dataframe. Have Claude extract structured insights with daft.functions.prompt (use structured outputs)! Heres what makes daft.File different: 👉 Lazy by default — point at a million S3 files, nothing downloads until you need it 👉 Subtypes — VideoFile, AudioFile carry domain-specific operations (seek, keyframes, resample) 👉 Open and seek — distributed concurrent reads across byte ranges, like WebDatasets but native to the type system 👉 Extensible — any Python library works via .open() and .to_tempfile() in UDFs PDFs, video, audio, code — all processed as first-class data with distributed execution and cloud-native I/O. Daft is the data engine for AI. daft.File is the type that makes it possible. Blog: https://www.daft.ai/blog/daftfile-work-with-any-file-anywhere Docs: https://docs.daft.ai/en/stable/modalities/files/ Example: https://github.com/Eventual-Inc/daft-examples/blob/main/pipelines/context\_engineering/claude\_insights.py

Still on daft.udf? (Legacy User Defined Functions) You're going to want to get that migrated. Just point Claude at our migration guide! For stateless functions -> daft.func For stateful classes -> daft.cls The swap: > daft.udf(return\_dtype=...) becomes daft.func.batch(return_dtype=...) > Class UDFs become daft.cls + daft.method > Init args move to the decorator or init The new API is more fully featured: > async UDFs > generators > Pydantic types > stateful classes with GPU allocation. Seriously though, check out the migration guide on the @daftengine docs: https://docs.daft.ai/en/stable/custom-code/migration/

“Daft is a distributed query engine that’s going to be replacing Spark.” I remember this talk like it was yesterday 👇 Two years ago @ SF Systems Meetup, Colin Ho and Desmond Cheong walked up and floored the crowd - debuting v1 of Swordfish, our local execution engine. Even at the time, Daft demonstrated dramatically lower memory than Spark on TPCH. 17 months later we've delivered multiple iterations of Swordfish and Flotilla with compounding adoption across top labs and startups. Daft's execution model contrasts directly with Spark: Spark: \- Pull based, Partition Driven \- JVM runtime \- Bulk Synchronous Daft: \- Push-based, morsel-driven streaming. \- Async Rust runtime. \- Dynamic parallelism. Even at the time, the original benchmarks showed dramatic improvements on both memory and speed. It was a seminal moment for the team and the open source community, not just for it's technical depth but it's coherence of vision. Our latest benchmarks: https://docs.daft.ai/en/stable/benchmarks/ Watch the full talk on youtube:

What is the easiest way to find a bottleneck in a distributed query pipeline? With pretty colors and stats to back it up. Daft v0.7.8 adds a live heatmap coloring to the query dashboard — slow operators light up red, completed nodes turn green, and arrows trace data flow through the pipeline. You see the hot path the moment a query starts running. What's new: \- Dashboard heatmaps — real-time operator coloring by execution time \- Vectorized grouped aggregation — dense group-id arrays replace per-group index lists, delivering up to 4.4x faster groupbys at high cardinality. ClickBench total dropped 10% below the v0.7.4 baseline \- Native image\_hash() — 8 perceptual hashing algorithms in Rust, bit-exact with Python imagehash, 5-25x faster than a UDF. For large-scale image deduplication \- df.skip\_existing() — checkpoint-based row deduplication for idempotent ETL and RAG pipelines \- DataFrame.skew() and .var() — statistical aggregation convenience methods \- Subscriber API improvements — nicer public APIs, aimed at OpenLineage support 9 community contributors shipped image hashing, checkpoint operators, statistical aggregations, and more. blog: https://www.daft.ai/blog/daft-v0.7.8

$1.10 to process 20 years of SEC filings into clean, tokenized plaintext. Datamule, Teraflop AI, and Eventual just open-sourced the SEC-EDGAR dataset on Hugging Face: - 590 GB of raw and parsed data - 8 million individual filings (10-K, 10-Q, 8-K, and more) - 43 billion tokens — ready for LLM training and retrieval pipelines The bulk data was collected via datamule's API, then parsed and extracted using selectolax, doc2dict, and secsgml. Daft's daft.cls stateful UDFs handled the batch document processing — loading parsers once and reusing them across millions of filings. The Daft team also shipped a fix for >2GB UTF-8 string arrays that surfaced during processing. When your framework breaks on real data, fix the framework. 12 cores. Under 24 hours. $1.10 total compute. Blog: https://www.daft.ai/blog/sec-edgar-case-study

🚢 Daft v0.7.9 has shipped This release ships 8 new temporal functions with Spark-compatible naming. Together with the batch 1 functions from v0.7.7, Daft now covers the core date arithmetic needed for most ETL migrations. What else shipped: - Native UUID type — first-class UUID across the entire type system. - video_frames() — column-level video frame decoding from File columns with full metadata (frame index, timestamps, keyframe flags, image data). Filter by time range, resize on decode - file_path() — extract paths from File columns as strings - Byte-level dashboard observability — per-operator bytes_in/bytes_out, data volume arrows, expansion/reduction tags, and version metadata on queries - ASOF joins — initial join_asof() implementation for point-in-time lookups with entity grouping blog: https://www.daft.ai/blog/daft-v0.7.9

Srinivas Lade and Colin Ho will be presenting at TokioConf 2026 in PDX on April 21. If you're in the PNW, join our two core engine maintainers for technical discussion on Daft's internals!

Ok yeah, embeddings and RAG is so 2024, but did you know embeddings can work across modalities? Here's a quick lesson 👇 Multimodal embeddings project different data types into a shared vector space so a text query can find the right image, and a chest X-ray can match a doctor's note, without modality-specific pipelines. The challenge is that most frameworks treat this as a multi-tool problem. One system for preprocessing, another for embedding generation, another for retrieval. Every boundary is a serialization cost and an ops burden. Our new tutorial walks through the full pipeline with CLIP and Daft: 👉 CLIP contrastive learning — dual encoders trained to align image-text pairs in a shared space 👉 daft.cls for stateful inference — load the model once, batch 16 images at a time, run on GPU if available 👉 cosine\_distance for retrieval — cross-join query embeddings against the image index, sort, return top-K 👉 One script, one framework — no handoff between preprocessing and ML stages The full example is ~60 lines. Runnable in a Colab notebook. Read the blog: https://www.daft.ai/blog/multimodal-embeddings

daft.File is a lazily evaluated file reference. When you call daft.from\_files(), nothing downloads — you get lightweight references that defer all I/O until a UDF explicitly opens them. 👇 The pattern: filter by metadata first (path, size, MIME type), then open only the files that survive your predicates. Cheap operations narrow the set. Expensive operations run on the survivors. Same .open() and .to_tempfile() interface works for PDFs, Python source, audio, and video. This is Week 2 of the daft.File series. Read the blog: https://daft.ai/blog/daft-file-lazy-metadata-filtering

Wait... I forgot to post it here but this blew up! --- $1.10 to process 20 years of SEC filings into clean, tokenized plaintext. 12 cores. Under 24 hours. 👉 Datamule, Teraflop AI, and Eventual just open-sourced the SEC-EDGAR dataset on Hugging Face: - 590 GB of raw and parsed data - 8 million individual filings (10-K, 10-Q, 8-K, and more) - 43 billion tokens — ready for LLM training and retrieval pipelines The bulk data was collected via datamule's API, then parsed and extracted using selectolax, doc2dict, and secsgml. Daft's daft.cls stateful UDFs handled the batch document processing — loading parsers once and reusing them across millions of filings. The Daft team also shipped a fix for >2GB UTF-8 string arrays that surfaced during processing. When your framework breaks on real data, fix the framework. Read about it in the blog: https://www.daft.ai/blog/sec-edgar-case-study 🙌 Enrico at TeraflopAI always does such amazing work.

daft.VideoFile is perfect for Physical AI. Open X-Embodiment aggregates over a million episodes. DROID alone runs 350+ hours of multi-camera 60fps footage. That's hundreds of millions of frames across a single dataset, and most action-model training doesn't need them all. daft.VideoFile decodes only the slice you describe. •

read_video_frames

— filter on keyframes; supports S3, GCS, & YouTube URLs. •

video_metadata

— resolution, fps, duration, frame count from file headers. •

video_frames(start_time, end_time)

— decode a 10-second window from a 90-minute file. Frames land as Image columns in the same DataFrame. Feed them to a vision model, compute embeddings, and write to Iceberg. Check out the blog https://www.daft.ai/blog/daft-videofile-seek-lazily-get-frames

When my fleet of flapjack flipping humanoids finishes uploading it's data to CAIOS, I need a data engine that can help me align all of my robotics data. Think, 1–10 TB of video, 5–500 GB of sensor telemetry, and lots of flapjacks. My video and sensor data come in separate payloads, so the two streams don't always align. My robot's eyeballs record video at 30 fps while super-duper-flapjack-flipping motor actuators run at 100 Hz. How do I query my video and sensor data so my video frames align against my flapjack actuator signals in order? 🦾 🥞 You guessed it 👉 an ASOF join — matching each frame to the most recent sensor reading at or before that timestamp. Every major DataFrame library that supports ASOF joins is single-node only. Spark technically has

merge_asof

but rewrites it to a correlated subquery under the hood. If you're running TB-scale pipelines, none of these work. Daft now has native distributed ASOF joins. Here's what it took to get there: • V1 (sort + two pointers): 133s on 100M rows — 3.75x slower than pandas; string comparisons dominated every sort comparison • V2 (hash grouping): 36s — hash by entity key first, then sort only the integer timestamp column within each group • V3 (streaming probe): 21s, 4 GB peak — right table streams in parallel batches; parallelism is now batch count, not entity cardinality • V4 (range partitioned + distributed): 283s → 31s scaling from 2 to 8 nodes, skew-resistant via carryovers Daft is the first distributed DataFrame library with a native ASOF join built-in. Link in the comments.

Teams running Daft in production used to craft observability dashboards by hand. The Daft Dashboard now ships with the standard install and makes Operators, Tasks, Rows, and Memory first-class. The dashboard ships with the install with a query plan tree, tasks view for distributed execution, failure surfacing per operator, per-operator memory attribution. Plus OTel endpoints for your existing collector and DAFT\_TRACE for console debugging. IMHO the team really knocked it out of the park with this one. Daft's internals aren't for the faint of heart and I think the new dashboard makes it a lot easier to grok the right mental model of how the engine actually works. https://www.daft.ai/blog/first-class-observability-in-daft

Type conversion errors halt data pipelines when source data doesn't match schema expectations. Daft v0.7.15 introduces try\_cast() for safe type conversion, Flight shuffle LZ4 compression for faster distributed shuffle, and UUIDv7 timestamp extraction for time-based partitioning. 55+ improvements from 21 contributors across type safety, performance optimization, and data source expansion. https://daft.ai/blog/daft-v0-7-15

Man! It's been a while! But we're back at it again: One corrupt Parquet file in a 10 TB data lake can abort an overnight batch job that was 6 hours into processing. Daft v0.7.16 ships resilient file reads, robotics dataset support, and native PyTorch integration. 44 changes from 22 contributors. • daft.datasets.droid — DataFrame API over 76k robot manipulation demonstrations from the DROID dataset. • to_torch_dataloader — connect Daft directly to PyTorch training loops without writing a custom Dataset wrapper! • ignore_corrupt_files — skip bad files instead of aborting the job. Every skipped file surfaces in df.skipped_corrupt_files as (path, reason, partial) tuples. Works on Parquet, CSV, Iceberg • daft.concat() — concatenate multiple DataFrames in a single call • Spark-style timezone conversions — from_utc_timestamp, to_utc_timestamp, convert_timezone with identical PySpark semantics • String distance functions — levenshtein, jaro, jaro-winkler, damerau-levenshtein for fuzzy matching and dedup https://daft.ai/blog/daft-v0-7-16

LeRobot, Droid, Hdf5File, Transformers Support for daft.functions.prompt Daft v0.7.17 ships the second robotics dataset integration, first-class HDF5 support, and local LLM inference. 18 changes from 10 contributors. Highlights: • daft.datasets.lerobot — read LeRobot v3 datasets (episodes, frames, camera video) as one lazy DataFrame, with joins and video-frame alignment handled internally • daft.Hdf5File — typed hdf5_keys, hdf5_metadata, and hdf5_attrs expressions for inspecting HDF5 files without hand-rolling h5py code • prompt(provider="transformers") run open-weight instruction-tuned models locally via HuggingFace pipeline(), no API key required • Hugging Face buckets for Common Crawl a credential-free alternative to the AWS S3 mirror for anyone outside us-east-1 • any\_value() now has its own inline accumulator for numeric types, so mixing it with sum, min, or max no longer forces the slow aggregation path https://www.daft.ai/blog/daft-v0-7-17