@Kevin Wang https://github.com/Eventual-Inc/Daft/pull/4410 might help our poor github runners

small PR to fix a regression in str.substr https://github.com/Eventual-Inc/Daft/pull/4415

@Cory Grinstead little PR to add a prelude for the required ScalarUDF imports. https://github.com/Eventual-Inc/Daft/pull/4416

Cutting release today, please list any blockers in the 🧵

Very excited for this, can't wait to see the final move to logical plans so we can start refactoring the optimizer rules

PR to enable scalar function lowering e.g. overloads which let's us do type-checking then lowering in the planner rather than during evaluation hence 'dynamic' naming in the PR. https://github.com/Eventual-Inc/Daft/pull/4431

PR to create a Daft value from a JSON string! • reviewer: @Cory Grinstead • customer request: @NikkTheGreek via https://github.com/Eventual-Inc/Daft/issues/3994 • PR: https://github.com/Eventual-Inc/Daft/pull/4438 Example

import daft

df = daft.from_pydict({

    "person": [
        '{"name": "Alice", "age": 30}',
        '{"name": "Bob", "age": 25}',
        '{"name": "Charlie", "age": 35}',
    ]
})

# STRUCT<name: STRING, age: BIGINT>
person_type = dt.struct(
    {
        "name": dt.string(),
        "age": dt.int64(),
    }
)

df.select(df["person"].from_json(person_type)).show()

╭────────────────────────────────╮
│ person                         │
│ ---                            │
│ Struct[name: Utf8, age: Int64] │
╞════════════════════════════════╡
│ {name: Alice,                  │
│ age: 30,                       │
│ }                              │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ {name: Bob,                    │
│ age: 25,                       │
│ }                              │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ {name: Charlie,                │
│ age: 35,                       │
│ }                              │
╰────────────────────────────────╯

(Showing first 3 of 3 rows)

@Robert Howell PR to fully implement MemoryTable and MemoryCatalog: https://github.com/Eventual-Inc/Daft/pull/4445

@Kevin Wang back at you! Here's a deprecated API cleanup, and I've added a CTE map to daft.sql which better fits the SQL model and allows us to deprecate then remove SQLCatalog. https://github.com/Eventual-Inc/Daft/pull/4460

👋 I am querying trino via daft and running into some unexpected behavior. I have one query where I left join a table and one of the columns ends up having all null values. when written to parquet, this column ends up with a type of null, but it should have type string. this is unexpected behavior, right?

@Kevin Wang @Cory Grinstead PR for a flattened

.jq

method. https://github.com/Eventual-Inc/Daft/pull/4470

small PR to get daft-dashboard working without errors again. https://github.com/Eventual-Inc/Daft/pull/4475 Note that the visualization is completely disabled for now until we can come up with a better visualization than the ugly and broken mermaid viz.

@jay PR ready for UC volumes support!! I'll get a demo notebook for you soon as well https://github.com/Eventual-Inc/Daft/pull/4476

Expr refactor is officially done after this PR is merged! https://github.com/Eventual-Inc/Daft/pull/4480

list type column iteration/processing 🧵

isinstance() checks on data frame column values. 🧵

probably missing this in the docs, how can I get the datatype of the elements in a list type?, nvm

.dtype

FYI, I have a few PR's I'd like to get reviewed. https://github.com/Eventual-Inc/Daft/pulls?q=is%3Aopen+is%3Apr+author%3Auniversalmind303+-is%3Adraft

PR to do a little bit of performance cleanup on the expressions (ScalarFunction). https://github.com/Eventual-Inc/Daft/pull/4489 cc @Robert Howell I think this moves us a bit closer to your intended usage of the

ScalarFunctionFactory

as the

ScalarFunction

is no longer directly storing the

ScalarUDF

, but instead resolving it during planning.

@Sammy Sidhu PR to update our AWS crates. Will work on removing our dependency/vendoring of openssl as a next step https://github.com/Eventual-Inc/Daft/pull/4508

Could I get a quick review on this PR? It's to help debug an issue a daft spark user is facing https://github.com/Eventual-Inc/Daft/pull/4521

could I also get a review on this one here https://github.com/Eventual-Inc/Daft/pull/4524 related to the same issue as 4521. cc @Kevin Wang

Quick heads up: Major Service outage on GCP -> No claude.

any plans to add a Dataframe.write_json() function to compliment the write_csv() and write_parquet() etc?

Hi, may I check do we have any github workflow bot which accept rerun command to rerun the failed CI? example from hudi as bellow picture shows, or how can I rerun/re-trigger the failed ci without re-submit a new change?

Hi, has anybody met a problem such as the following when debugging with IntelliJ? Seems like the io module shadows the built-in io.

Working on an issue focused on arrow schema mismatches . Is there any known work related to this besides: https://github.com/Eventual-Inc/Daft/issues/3605 https://github.com/Eventual-Inc/Daft/issues/1958

@Desmond Cheong @Srinivas Lade here's a PR that enables us to pushdown filters into LanceDB. https://github.com/Eventual-Inc/Daft/pull/4616

Does Daft support geospatial types now? Any posts/info where I can learn more?

hello, i’m a new developer. It seems daft used both pyproject.toml and requirements.txt to manage the dependencies. what’s the relation between the two? why we don’t only use pyproject.toml and uv to manage the dependencies?