👋 I am seeing the following after upgrading to

0.6.14

, looks like the

concat

at micropartition level not usage of

concat

by me right (I am not using it fwiw)? Will share query plan

File "/tmp/ray/session_2026-01-08_20-44-16_741313_1/runtime_resources/pip/4858717c93c532512d70d85304fc18c2c24c6c55/virtualenv/lib/python3.12/site-packages/daft/recordbatch/micropartition.py", line 135, in concat
 return MicroPartition._from_pymicropartition(_PyMicroPartition.concat(micropartitions))
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
daft.exceptions.DaftCoreException: DaftError::SchemaMismatch MicroPartition concat requires all schemas to match, ╭──────────────────────┬────────╮
│ column_name ┆ type │
╞══════════════════════╪════════╡
│ __store_cluster_id_l ┆ String │
╰──────────────────────┴────────╯
 vs ╭──────────────────────┬────────╮
│ column_name ┆ type │
╞══════════════════════╪════════╡
│ __store_cluster_id_r ┆ String │
╰──────────────────────┴────────╯

Are you using the old Ray runner or the new one? The old Ray runner was removed from

0.7.x

onwards

I am using the new ray runner, which is default in 0.6.14, right?

Yes, as long as

use_legacy_ray_runner

is not set to true

got it, then yes I am using the new ray runner

do you mind sharing the output of

df.explain(show_all=True)

? knowing what operations were performed would help narrow down where the

MicroPartition.concat

is used

plan.txt

actually, do you mind generating the explain with the ray runner enabled? it will create a slightly different plan because repartitions are removed from the native runner (since they are no-ops)

plan_ray.txt

Ah ok, I believe the problem is coming from our implementation of sort-merge join

daft party that is an easy change on my side, thanks! actually wanted hash join, but had a default somewhere to sort-merge when I was testing something

Great, we'll look into the problem with sort-merge join. i believe the bug occurs when the left and right table column names are different, so a temporary fix would be to rename to the same name

took a stab at a fix, I am still rusty in rust, but I think this captures the gist of what you mention above https://github.com/Eventual-Inc/Daft/pull/6243