Community for the Daft project and all things distributed data

Distributed Data Community

:wave: Hi everyone! I am trying to figure out if it's possible to run ML inference pipeline on dataframe batches using all columns, eg read data -> run inference (vectorized predict(row) ) -> store predictions. I was hoping to use python batch udfs but those seem to work only with df series. Is there another way that I can use?

I figured out a solution by concatenating all columns into a new list series as an input column and calling a batch udf on top of it. Let me know if there is a better way

```from daft.functions import to_list

df = df.select(to_list(*[df[col_name] for col_name in df.column_names]).alias("input"))
prediction = df.select(ml_inference(df["input"]))```

you can definitely pass multiple columns into a batch udf!

something like this

```@daft.func.batch(return_dtype=...)
def ml_inference(col1, col2):
  ...

df = df.select(ml_inference(df["col1"], df["col2"])```