2025-10-24 14:33:40,552 - ERROR - <http://lib.rs:313|lib.rs:313> - Error when running pipeline node Parquet Write
2025-10-24 14:33:40,553 - ERROR - <http://mod.rs:184|mod.rs:184> - RuntimeStatsManager finished with active nodes {8, 5, 9, 11, 2, 6, 3, 4, 10, 7}
<flow DailyPromoETLFlow step start> failed:
    Internal error
Traceback (most recent call last):
  File "/home/ray/metaflow/metaflow/cli.py", line 636, in main
    start(auto_envvar_prefix="METAFLOW", obj=state)
  File "/home/ray/metaflow/metaflow/_vendor/click/core.py", line 829, in __call__
    return self.main(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/metaflow/metaflow/_vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/ray/metaflow/metaflow/cli_components/utils.py", line 69, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/metaflow/metaflow/_vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/metaflow/metaflow/_vendor/click/core.py", line 610, in invoke
    return callback(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/metaflow/metaflow/_vendor/click/decorators.py", line 21, in new_func
    return f(get_current_context(), args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/metaflow/metaflow/cli_components/step_cmd.py", line 165, in step
    task.run_step(
  File "/home/ray/metaflow/metaflow/task.py", line 653, in run_step
    self._exec_step_function(step_func)
  File "/home/ray/metaflow/metaflow/task.py", line 62, in _exec_step_function
    step_function()
  File "/home/ray/metaflow/daily_promo_etl/flow.py", line 58, in start
    ).write_parquet(
      ^^^^^^^^^^^^^^
  File "/home/ray/metaflow/linux-64/d6b9289e73bbc10/lib/python3.12/site-packages/daft/dataframe/dataframe.py", line 822, in write_parquet
    write_df.collect()
  File "/home/ray/metaflow/linux-64/d6b9289e73bbc10/lib/python3.12/site-packages/daft/dataframe/dataframe.py", line 3988, in collect
    self._materialize_results()
  File "/home/ray/metaflow/linux-64/d6b9289e73bbc10/lib/python3.12/site-packages/daft/dataframe/dataframe.py", line 3951, in _materialize_results
    self._result_cache = get_or_create_runner().run(self._builder)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/metaflow/linux-64/d6b9289e73bbc10/lib/python3.12/site-packages/daft/runners/native_runner.py", line 74, in run
    results = list(self.run_iter(builder))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ray/metaflow/linux-64/d6b9289e73bbc10/lib/python3.12/site-packages/daft/runners/native_runner.py", line 115, in run_iter
    for result in results_gen:
  File "/home/ray/metaflow/linux-64/d6b9289e73bbc10/lib/python3.12/site-packages/daft/execution/native_executor.py", line 44, in <genexpr>
    return (
           ^
daft.exceptions.DaftCoreException: DaftError::ParquetError Arrow: Incompatible type.

Struct

metadata: {"PARQUET:field_id": "1"}