Running into a couple things that look like Planner flaws. Just sanity checking my analysis.

tEnv.createTemporaryView(“contractswithevents”, “select contract_id, isin, leg1.events as leg1events, leg2.events as leg2events from contracts”));
Table leg1Events = tEnv.sqlQuery(“select contract_id, isin, ‘leg1’ as leg, e.event_id, e.quantity from contractswithevents CROSS JOIN UNNEST(leg1events) as e(event_id, quantity)”);
Table leg2Events = …
Table events = leg1Events.unionAll(leg2Events);
TablePipeline pipeline = events.groupBy($(“contract_id”), $(“quantity”).sum()).insertInto(“results”);

The optimized execution plan has 2 `Exchange(distribution=[hash[contract_id]])`s, one after the scan, and one between the union and the aggregate. What gives? Is the planner getting baffled by the unnest?

I'm not sure if this is the Flink planner, or actually Calcite doing this 🙂

If it would be of any value to you, I’d be happy to do a deeper dive to assess if you want to point me in the right direction.

I'm guessing @Sergey Nuyanzin probably can help you better here

not ready to answer why the plan looks like that for this case. The way how i probably start debugging it:

org.apache.calcite.plan.RuleEventLogger#ruleProductionSucceeded

is called after each successfully applied rule (Calcite's or Flink's rule). Moreover it is possible to see this rule and the plan after this rule is applied. At least with Calcite 1.29.0 -1.30.0 the logging was significantly improved there.