I am using

explode

with average result being 1 row --> 12 rows and max 1 row --> 366 rows (~5m rows --> ~66m rows). Seeing decently high memory usage during the explode even with a repartition prior to explode. Is the only remedy more partitions and/or reduced number cpus to reduce parallelism?

glad to hear this. Shoutout to @Kevin Wang for adding this feature!

awesome

one related question, I am finding that the legacy

udf

runs faster in some cases (with

use_process=True

). as an example, the following is taking a complex proto blob and doing some work on it to flatten into a more usable format. running in a single k8s pod with the cpu=12, memory=32gb

@daft.func(return_dtype=PROMOTION_STRUCT)  # type: ignore
def parse_promo_proto_to_features_udf2(
    proto_str: str
) -> dict[str, Any] | None:
    ....

@daft.udf(return_dtype=PROMOTION_STRUCT, use_process=True, concurrency=12)
def parse_promo_proto_to_features_udf(
    promo_protos_str: daft.Series
) -> list[dict[str, Any] | None]:
    ...

So this is a known issue and one of our top priorities at the moment. We're actively working on improving the performance of

daft.func

and getting it as fast or faster than the existing

daft.udf

.

nice, good to know, will mix and match for now based on performance