Does the read splitting mentioned in this blog post work in the native runner? I'm running into resource exhaustion when reading a single 2.5GB parquet file and attempting to write it back out repartitioned across 30 buckets using iceberg writer. This works for smaller files (tested with 200MB). I've set the daft execution config to the same values as in the post. The node has 32GB RAM which seems like enough for the task, but it is actually quickly exhausted (within 2 minutes), same for swap. Any other params I could tune to make this work? Would a local Ray cluster handle this task within the same resources?

Is Anyone else experiencing strange scrolling behavior on the Docs? Seems to have started post update. If you scroll down, it will scroll to the top automatically

Hey folks, wanna get some inputs from the experts to see if this pattern is something Daft could support 🧵

Hello! the recent 0.4.11 daft release included a <=16 pyarrow contraint, see https://github.com/Eventual-Inc/Daft/pull/4225 , but the latest pyiceberg requires >=17, see https://github.com/apache/iceberg-python/blob/pyiceberg-0.9.0/pyproject.toml#L64 . Anyone know what the newly added max bound on daft is about? Or more directly if there are any issues to watch that are blocking the use of the later pyarrow releases? I've been using pyiceberg 0.9.0 with daft 0.4.10 but now can't upgrade daft without downgrading pyiceberg.

hey folks, does daft have something like https://fastht.ml/docs/#getting-help-from-ai , https://fastht.ml/docs/llms-ctx.txt which can be fed to llms might make using daft more productive with cursor etc.

Hey guys, I'd like to do something like this in

daft

without needing to specify each column (since I can have anywhere from 1 to 5). Is there another syntax that would work? This is not working as is:

@daft.udf(return_dtype=daft.DataType.python())
def mean_ensemble(*depth_value_series: daft.Series) -> List[Dict[str, np.ndarray]]:
    """Apply mean ensemble to depth maps."""
    depth_value_lists = [series.to_pylist() for series in depth_value_series]
    reduced_depth_maps: List[Dict[str, np.ndarray]] = []
    for depth_value_list in depth_value_lists:
        # Calculate mean and standard deviation across all depth maps in the list
        stacked_depths = np.stack(depth_value_list, axis=0)
        mean_depth = np.mean(stacked_depths, axis=0)
        std_depth = np.std(stacked_depths, axis=0)
        reduced_depth_maps.append(
            {
                "mean": mean_depth,
                "std": std_depth,
            }
        )

    return reduced_depth_maps

👋 any timeline for when window functions will be available for ray runner?

hey folks, wanna have your inputs on whats the most efficient way of resizing partitions

I am trying to explode on a column that is a list of structs (it's a fairly nested schema) and encountering this error:

Attempting to downcast Map { key: Utf8, value: List(Utf8) } to \"daft_core::array::list_array::ListArray\"

Wondering if someone has seen this before?

hey folks im getting a weird Type error when reading from an iceberg table:

TypeError: pyarrow.lib.large_list() takes no keyword arguments

the code:

table = catalog.load_table(table)
    return df.read_iceberg(table)

has anyone experienced this before?

Im querying an iceberg table from a jyupter notebook (backed by 12Gb ram and 4 cpu)

daft.read_table("platform.messages").filter("event_time > TIMESTAMP '2025-05-24T00:00:00Z'").limit(5).show()

running this makes the process crash. looks like memory goes thru the roof. Not sure if its trying to read the whole table into memory. pre-materialization, i can get the schema just fine.

https://github.com/JanKaul/iceberg-rust 👀

Is there an equivalent to pyspark's

.dropDuplicates(subset: Optional[List[str]] = None)

where you can specify which columns to consider?

Hi there - I’m trying out Daft after meeting @ChanChan Mao and @Sammy Sidhu at Data Council a few weeks ago. I got all the queries from my recent Iceberg and Backblaze B2 blog post working - see https://gist.github.com/metadaddy/ec9e645fa0929321b626d8be6e11162e Performance in general is not great, but one query in particular is extremely slow:

# How many records are in the current Drive Stats dataset?
    count, elapsed_time = time_collect(drivestats.count())
    print(f'Total record count: {count.to_pydict()['count'][0]} ({elapsed_time:.2f} seconds)')

With the other systems I tested in my blog post, the equivalent query takes between a fraction of a second and 15 seconds. That Daft call to

drivestats.count()

takes 80 seconds. I’m guessing it’s doing way more work than it needs to - reading the record counts from each of the 365 Parquet files rather than simply reading

total-records

from the most recent metadata file. Since

SELECT COUNT(*)

is such a common operation, I think it’s worth short-circuiting the current behavior.

Question on the Daft Native runtime 🧵

Hey daft squad, If I'm using a MemoryCatalog to track lancedb tables, am I restricted to only using dataframes at the moment?

Where does the work take place when I use Daft with Ray? For example, consider the following minimal code:

import daft
import ray

ray.init("<ray://head_node_host:10001>", runtime_env={"pip": ["daft"]})

daft.context.set_runner_ray("<ray://head_node_host:10001>")

catalog = load_catalog(
    'iceberg',
    **{
        'uri': 'sqlite:///:memory:',
        # configuration to access Backblaze B2's S3-compatible API such as
        # s3.endpoint, s3.region, etc
    }
}

catalog.create_namespace('default', { 'location': f'<s3://my-bucket/'}>)
table = catalog.register_table('default.drivestats', metadata_location)

drivestats = daft.read_iceberg(table)

result = drivestats.count().collect()
print(f'Total record count: {result.to_pydict()['count'][0]}')

Presumably, the code to read Parquet files from Backblaze B2 via the AWS SDK executes on the Ray cluster, so I have to either install the necessary packages there ahead of time or specify them, and environment variables, in

runtime_env

? For example:

ray.init("ray://<head_node_host>:10001", runtime_env={
    "pip": ["daft==0.5.2", "boto3==1.34.162", "botocore==1.34.162", ...etc...],
    "env_vars": {
        "AWS_ENDPOINT_URL": os.environ["AWS_ENDPOINT_URL"],
        "AWS_ACCESS_KEY_ID": os.environ["AWS_ACCESS_KEY_ID"],
        "AWS_SECRET_ACCESS_KEY": os.environ["AWS_SECRET_ACCESS_KEY"],
        ...etc...
    }
})

Hi, how does one insert a row to a df with daft? All examples I see load an existing dataset and add columns. Thanks!

We use daft on Ray with Glue and Iceberg. We recently changed the table config from CoW to MoR to improve write perf on our Spark jobs. Unfortunately, Daft can't read the tables, let alone any operations like joins, without running out of memory. Tried up to 200GB worker and a 20GB table just loading in daft and converting to a ray dataset.

Hi, what are the recommended ways to debug UDFs? Is there a way to print or log in the console when using ray?

Hi Data Engineering Friends, I'm looking for someone to help us finalize our Daft / Ray setup. Is there anyone in the group who has solid experience with Daft and willing to do some consulting work? We're currently using Spark and I've ported our code over to Daft/Ray to give it a try. We're really love the tooling and API compared to Spark so far. I was hoping to see performance improvements, but we're hitting a few snags out of the gate. Iceberg tables writing much slower than Spark and some OOMs we weren't seeing before. I'm optimistic that these issues are solvable, but I could use the expertise of someone who's been around the block with Daft. Thank you!

Hi, do you have recommendations for performing semantic search (with the use of embeddings) on daft? I haven’t found much so far. I’m mainly interested in the distributed case.

👋 qq, I am seeing mixed support with respect to identifiers for Iceberg tables. specifically, for some methods I can provide

catalog.namepace.table

(e.g.

write_table

), but others break and seem to expect

namespace.table

(e.g.

create_table_if_not_exists

which calls pyiceberg under the hood that had breaking change that no longer allows including catalog). Any advice on how I should be providing identifiers?

👋 I am running into weird issue where native runner is "hanging" when trying to read an Iceberg table, but switching to Ray runner (local cluster) works fine (plan in 🧵). Maybe I am hitting an edge case as this is single row test table in staging environment for testing Nessie integration.

If daft.pyspark were complete, then assuming it covers the operations one needs, would there still be an advantage to using Daft's own API instead?

LFG DAFT SQUAD!

Today we're thrilled to announce that Eventual has raised $30M in funding to power the future of multimodal AI infrastructure! Jay Chia and I started this journey 3 years ago, frustrated by the same wall every AI team hits: processing images, video, and documents at scale with tools built for entirely different use cases. What began as pure frustration in my basement has become the data engine trusted by Amazon, CloudKitchens, Essential AI, Together AI and other Fortune 25 companies. The numbers speak for themselves: Daft improved Amazon's most critical data job efficiency by 24%, saving them 40,000 years of compute time annually. Together AI replaced their custom pipelines with simple Daft queries for 100TB+ datasets while achieving 10x performance gains. But this is just the beginning. AI applications are now generating massive amounts of multimodal data at machine speed, and we're building the engine to power it all. Thank you to our incredible community and supporters who made this possible - from our earliest believers at Y Combinator and Array Ventures to our lead investors CRV and Felicis, plus M12, Microsoft's Venture Fund, Citi, and everyone who's been part of this journey. What's next? We're launching early access to Eventual Cloud - the first production platform built from scratch for multimodal AI workloads. We're also hiring across Engineering, DevRel, Design and Product Marketing. - Check out our open roles here. Please help share the love on LinkedIn & Twitter! https://daft-amplify.lovable.app/ (PS, check out the video in the posts, it's pretty cool)

Hello community, I saw https://daft.ai/ being shown on this repo: https://github.com/Eventual-Inc/Daft, but the address doesn’t open. Is this a known issue?

Another huge milestone achieved this week - Daft has surpassed 3000 stars! Thank you to our growing community for continuing to support us and sharing the love of Daft. And thank you to everyone who is using Daft and believing in the future that we're building. https://github.com/Eventual-Inc/Daft

fyi, I am hitting this error https://github.com/apache/arrow/issues/21526 with

pyarrow

when using Ray data directly and also seeing errors trying to read the same data with

daft

.

pyspark

works fine. I assume daft might be impacted by the same issue (too large row groups)?