For llm_generate is it possible to run a local huggingface model? Perhaps by directly putting the local model repo path in the params instead of an open huggingface repo name?

We currently do not have this ability. How are you using the model outside of Daft?

Currently simply via transformers or on vllm, but would like to test this out for batch inferencing for local small models (such as for evals or synthesis)

We have this capability for our new embed api but we need to extend it to generate! cc: @Desmond Cheong @Robert Howell

@Kyle I’d also recommend LMStudio if you are on Mac. There’s usually MLX variants for most models and LMStudio spins up an OpenAI server which you can use with llm_generate.

Yep, if you're using LM Studio you can use a local model like so

import daft
from daft import col
from daft.functions import llm_generate, format
df = daft.from_pydict({"city": ["Paris", "Tokyo", "New York"]})
df = df.with_column(
    "description",
    llm_generate(
        format(
            "Describe the main attractions and unique features of this city: {}.",
            col("city"),
        ),
        model="openai/gpt-oss-20b",
        base_url="<http://127.0.0.1:1234/v1>",
        api_key="this-is-not-needed",
        provider="openai",
    ),
)
df.collect()

Cool thanks! And does this also work on Ray?

that becomes a little harder 😅 we're in the process of beefing up/creating new apis for embed/extract/prompt in the next few weeks, which would then let you run open models in distributed mode

@Kyle Are you dead set on offline serving?

@Everett Kleven I'm thinking of running things on finetuned models via their various checkpoints as I go along so I was thinking offline serving may be easier, but just checking out the feasibility as I plan things out!

@Kyle mind creating issue and tagging us? It would be best to track debugging there.

A bit hesitant because I think it might just be my own environment flaking on me 🫠

Totally understandable. Its still a pretty rough time getting a new model up and running for the first time.