Is `parquet inflation factor` still respected in flotilla I Distributed Data Community #general

Is `parquet_inflation_factor` still respected in f...

Garrett Weaver

10/24/2025, 8:17 PM

parquet_inflation_factor

still respected in flotilla? I have a parquet file that is ~2GB, but expands to ~16GB in memory. When trying to run a select with window function, it is trying to load all data into memory and struggling (OOMing). What is the best course of action here?

✅ 1

Garrett Weaver

10/24/2025, 8:20 PM

my only thought is to rewrite into smaller/more parquet files via config.

into_partitions

and

repartition

have similar issues

Copy code

daft.context.set_execution_config(
    parquet_inflation_factor=10,
    scan_tasks_min_size_bytes=int(1 * 1024 * 1024),
    parquet_target_filesize=int(64 * 1024 * 1024),
    parquet_target_row_group_size=int(16 * 1024 * 1024),
)

Garrett Weaver

10/24/2025, 8:31 PM

Copy code

testing = (
    daft.read_parquet("~/Desktop/some_data")
    .select(
        "date",
        "id",
        row_number()
        .over(
            Window()
            .partition_by(
                daft.col("date"),
                daft.col("id"),
            )
            .order_by(daft.col("some_col"), desc=[True])
        )
        .alias("rn"),
    )
    .where(daft.col("rn") == daft.lit(1))
    .exclude("rn")
)

Malcolm Greaves

10/25/2025, 10:43 PM

cc @Colin Ho or @Srinivas Lade to investigate (please 🙏)

Colin Ho

10/27/2025, 5:56 AM

Hmmmm unfortunately right now the window functions fully materialize all the data per partition. On flotilla, we should be able to alleviate this by specifying a repartition before the window. But I'll need to add a fix in for that real quick

🙏 1

💡 1

Colin Ho

10/27/2025, 5:56 AM

Also to answer your other question • Is