I've found that daft now moving to a new state-ful UDF implement: daft.cls. For developers who is using daft.udf already. Does the community suggest we switch to daft.cls now? (production related workloads) And what are the current functionality daft.cls does not support? Also what's the cost of migrating from daft.udf to daft.cls? IMO user only need to change the decorator name PLUS the function call argument. Is there any other thing I need to concern? Thanks!

There are a few things the newer daft.cls does not yet fully support. With the old daft.udf you could specify additional resource requirements like memory limits, but this was just a best guess effort. We want to fully revisit the design of additional resource requirements before exposing it to the new daft.cls.

Follow up: Also, for daft.cls. Does it support batch processing?

Yes this is supported by using the

@daft.method.batch

decorator on your function. It's a bit buried in the docs, but it can be found here

Yeah, totally agree to not exposing the memory / concurrency / batch_size directly for user to fine-tune it. it is better for the daft Flotilla Runner do the resource estimation and generate a plan that has the best resouce usage But I think some kind of workload might require tuning these args e.g batch_size for some specific model predict scenario. So i think it might be better to keep an option to override it.

We do expose

batch_size

, and

concurrency

on the new daft.cls but for the other ones such as memory limits, Like you said, ideally the engine should just handle it automatically. FWIW, the daft.cls is relatively new, so we're totally open to feedback on what's needed to best support various workloads.

There are other Key Usage differences with daft.cls. 1.

daft.cls()

UDFs require instantiation like normal python classes do. Previous

daft.udf

was instantiated with defaults automatically or

__init__

overrides could be passed with the

MyUDF(*call_args).with_init_args(**init_args)

2. You pass

return_dtype

in

daft.method()

OR the return type hint is automatically converted to a Daft DataType from a Typed Dict, Python Type Hint, or Pydantic BaseModel. You can just run this script with

uv run this_script.py

# /// script
# description = "Example Usage for new daft.cls() and daft.method() decorators"
# requires-python = ">=3.10, <3.13"
# dependencies = ["daft", "pydantic"]
# ///

import daft
import typing
import pydantic


class DoSomethingResultPydantic(pydantic.BaseModel):
    x3: int
    y3: int
    some_arg: str


class DoSomethingResultTypedDict(typing.TypedDict):
    x3: int
    y3: int
    some_arg: str


DoSomethingDaftDataType = daft.DataType.struct(
    {
        "x3": daft.DataType.int64(),
        "y3": daft.DataType.int64(),
        "some_arg": daft.DataType.string(),
    }
)


@daft.cls()
class NewUDFUsageExample:
    def __init__(self, x1: int, y1: int):
        self.x1 = x1
        self.y1 = y1

    # With Python Type Hint Resolution
    @daft.method()
    def do_something_typeddict(
        self, x2: int, y2: int, some_arg: str
    ) -> DoSomethingResultTypedDict:

        x3 = self.x1 + x2
        y3 = self.y1 - y2
        return {"x3": x3, "y3": y3, "some_arg": some_arg}

    @daft.method()
    def do_something_pydantic(
        self, x2: int, y2: int, some_arg: str
    ) -> DoSomethingResultPydantic:

        x3 = self.x1 + x2
        y3 = self.y1 - y2
        return DoSomethingResultPydantic(x3=x3, y3=y3, some_arg=some_arg)

    # With Return Dtype and no lint errors
    @daft.method(return_dtype=DoSomethingDaftDataType)
    def do_something_daft(self, x2: int, y2: int, some_arg: str):

        x3 = self.x1 * x2
        y3 = self.y1 // y2
        return {"x3": x3, "y3": y3, "some_arg": some_arg}


if __name__ == "__main__":
    # Instantiate the UDF
    my_udf = NewUDFUsageExample(x1=1, y1=2)

    # Create a dataframe
    df = daft.from_pydict({"x1": [1, 2, 3], "y1": [4, 5, 6]})

    # Use the UDF
    df = df.with_column(
        "something_typeddict",
        my_udf.do_something_typeddict(
            daft.col("x1"), daft.col("y1"), daft.lit("some_arg")
        ),
    )
    df = df.with_column(
        "something_pydantic",
        my_udf.do_something_pydantic(
            daft.col("x1"), daft.col("y1"), daft.lit("some_arg")
        ),
    )
    df = df.with_column(
        "something_daft",
        my_udf.do_something_daft(daft.col("x1"), daft.col("y1"), daft.lit("some_arg")),
    )
    df.show()

3. All daft.methods() and daft.func() work for sync and async with no need to specify. 4. If you are already using

__call__

,

@daft.method()

is not required.

Thanks for the information!