I'm seeing lower than expected cpu utilization 30-...
# generalb
Ben Cornelis
10/28/2025, 3:40 PMI'm seeing lower than expected cpu utilization 30-40% (I originally posted 5-20% but misread it) on my k8s pods running a query - any tips on debugging or optimizing this?
Ben Cornelis
10/28/2025, 3:42 PMIt's the same query referenced here, although now with a manual repartitioning:
Copy code
df = daft.read_parquet(
"<abfs://my_container@my_service_account.dfs.core.windows.net/in_table_path/>"
)
df = df.select(col("group_col"), col("other_col"))
.repartition(1200, col("group_col"))
.groupby("group_col")
.agg_list(col("other_col"))
df.write_parquet("<abfs://my_container@my_service_account.dfs.core.windows.net/out_table_path/>")j
jay
10/28/2025, 6:38 PMDo you have any metrics around network bandwidth? Curious to hear if maybe we're maxed out there
b
Ben Cornelis
10/28/2025, 7:54 PMit's running on instances with 16000 Mb/s max network bandwidth documented - my understanding is we've measured it to achieve similar bandwidth in practice
Ben Cornelis
10/29/2025, 2:50 PMIt looks like the max bandwidth being using during the download phase at least is ~ 3200Mb/s
Ben Cornelis
10/31/2025, 6:24 PMSo that is lower than expected. Could that be the reason? Are there any other possibilities to look into? And what should I be expecting in terms of CPU utilization?
j
jay
11/04/2025, 4:18 AMIt does tend to depend on a bunch of factors -- I'm guessing you're actually reading a ton of smaller files and pooling them into 1200 partitions? That does tend to be a very slow exchange operation.
@Colin Ho might be able to comment more on the performance of the shuffle, and maybe we can try also a different shuffle algorithm
b
Ben Cornelis
11/04/2025, 6:42 PMThe original input set has 50k files. But I've also tried pre-processing to combine them down to ~500 files, and it doesn't really make a difference. It's also low utilization past the shuffle phase