https://flink.apache.org/ logo
Join Slack
Powered by
Testing out 1.18.0 and need some pointers on how t...
# troubleshooting
o
Testing out 1.18.0 and need some pointers on how to debug this nullpointer exception. Program explodes on 
OurStatementSets.execute()
.Code works fine in 1.17.1. Using flink sql, elasticsearch sinks, kafka sources. Seems to be an issue with the optimizer in the tableplanner, but it's near impossible to step into these functions. Stack trace in thread.
Copy code
Exception in thread "main" java.lang.NullPointerException
	at java.base/java.util.Objects.requireNonNull(Objects.java:233)
	at org.apache.calcite.sql2rel.RelDecorrelator.getNewForOldInputRef(RelDecorrelator.java:1435)
	at org.apache.calcite.sql2rel.RelDecorrelator.access$400(RelDecorrelator.java:136)
	at org.apache.calcite.sql2rel.RelDecorrelator$DecorrelateRexShuttle.visitInputRef(RelDecorrelator.java:1713)
	at org.apache.calcite.sql2rel.RelDecorrelator$DecorrelateRexShuttle.visitInputRef(RelDecorrelator.java:1670)
	at org.apache.calcite.rex.RexInputRef.accept(RexInputRef.java:113)
	at org.apache.calcite.rex.RexShuttle.visitList(RexShuttle.java:163)
	at org.apache.calcite.rex.RexShuttle.visitCall(RexShuttle.java:115)
	at org.apache.calcite.rex.RexShuttle.visitCall(RexShuttle.java:38)
	at org.apache.calcite.rex.RexCall.accept(RexCall.java:189)
	at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateExpr(RelDecorrelator.java:405)
	at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:807)
	at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:771)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:531)
	at org.apache.calcite.sql2rel.RelDecorrelator.getInvoke(RelDecorrelator.java:740)
	at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:1349)
	at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:1329)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:531)
	at org.apache.calcite.sql2rel.RelDecorrelator.getInvoke(RelDecorrelator.java:740)
	at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:1349)
	at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:1329)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:531)
	at org.apache.calcite.sql2rel.RelDecorrelator.getInvoke(RelDecorrelator.java:740)
	at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:1349)
	at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:1329)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:531)
	at org.apache.calcite.sql2rel.RelDecorrelator.getInvoke(RelDecorrelator.java:740)
	at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:1349)
	at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:1329)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:531)
	at org.apache.calcite.sql2rel.RelDecorrelator.getInvoke(RelDecorrelator.java:740)
	at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:1349)
	at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:1329)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:531)
	at org.apache.calcite.sql2rel.RelDecorrelator.getInvoke(RelDecorrelator.java:740)
	at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:1349)
	at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:1329)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:531)
	at org.apache.calcite.sql2rel.RelDecorrelator.getInvoke(RelDecorrelator.java:740)
	at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:1244)
	at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:1227)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:531)
	at org.apache.calcite.sql2rel.RelDecorrelator.getInvoke(RelDecorrelator.java:740)
	at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:782)
	at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:771)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:531)
	at org.apache.calcite.sql2rel.RelDecorrelator.getInvoke(RelDecorrelator.java:740)
	at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:448)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:531)
	at org.apache.calcite.sql2rel.RelDecorrelator.getInvoke(RelDecorrelator.java:740)
	at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateRel(RelDecorrelator.java:448)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:531)
	at org.apache.calcite.sql2rel.RelDecorrelator.getInvoke(RelDecorrelator.java:740)
	at org.apache.calcite.sql2rel.RelDecorrelator.decorrelate(RelDecorrelator.java:330)
	at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:216)
	at org.apache.calcite.sql2rel.RelDecorrelator.decorrelateQuery(RelDecorrelator.java:181)
	at org.apache.flink.table.planner.plan.optimize.program.FlinkDecorrelateProgram.optimize(FlinkDecorrelateProgram.scala:41)
	at org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram.$anonfun$optimize$2(FlinkGroupProgram.scala:59)
	at scala.collection.TraversableOnce.$anonfun$foldLeft$1(TraversableOnce.scala:156)
	at scala.collection.TraversableOnce.$anonfun$foldLeft$1$adapted(TraversableOnce.scala:156)
	at scala.collection.Iterator.foreach(Iterator.scala:937)
	at scala.collection.Iterator.foreach$(Iterator.scala:937)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
	at scala.collection.IterableLike.foreach(IterableLike.scala:70)
	at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
	at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:156)
	at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:154)
	at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:104)
	at org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram.$anonfun$optimize$1(FlinkGroupProgram.scala:56)
	at org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram.$anonfun$optimize$1$adapted(FlinkGroupProgram.scala:51)
	at scala.collection.TraversableOnce.$anonfun$foldLeft$1(TraversableOnce.scala:156)
	at scala.collection.TraversableOnce.$anonfun$foldLeft$1$adapted(TraversableOnce.scala:156)
	at scala.collection.immutable.Range.foreach(Range.scala:155)
	at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:156)
	at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:154)
	at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:104)
	at org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram.optimize(FlinkGroupProgram.scala:51)
	at org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.$anonfun$optimize$1(FlinkChainedProgram.scala:59)
	at scala.collection.TraversableOnce.$anonfun$foldLeft$1(TraversableOnce.scala:156)
	at scala.collection.TraversableOnce.$anonfun$foldLeft$1$adapted(TraversableOnce.scala:156)
	at scala.collection.Iterator.foreach(Iterator.scala:937)
	at scala.collection.Iterator.foreach$(Iterator.scala:937)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
	at scala.collection.IterableLike.foreach(IterableLike.scala:70)
	at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
	at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:156)
	at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:154)
	at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:104)
	at org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.optimize(FlinkChainedProgram.scala:55)
	at org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizer.optimizeTree(StreamCommonSubGraphBasedOptimizer.scala:176)
	at org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizer.optimizeBlock(StreamCommonSubGraphBasedOptimizer.scala:123)
	at org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizer.$anonfun$doOptimize$2(StreamCommonSubGraphBasedOptimizer.scala:90)
	at org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizer.$anonfun$doOptimize$2$adapted(StreamCommonSubGraphBasedOptimizer.scala:90)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:58)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:51)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
	at org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizer.doOptimize(StreamCommonSubGraphBasedOptimizer.scala:90)
	at org.apache.flink.table.planner.plan.optimize.CommonSubGraphBasedOptimizer.optimize(CommonSubGraphBasedOptimizer.scala:87)
	at org.apache.flink.table.planner.delegation.PlannerBase.optimize(PlannerBase.scala:324)
	at org.apache.flink.table.planner.delegation.PlannerBase.translate(PlannerBase.scala:182)
	at org.apache.flink.table.api.internal.TableEnvironmentImpl.translate(TableEnvironmentImpl.java:1277)
	at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:862)
	at org.apache.flink.table.api.internal.StatementSetImpl.execute(StatementSetImpl.java:109)
	at our.organization.main(SearchUpdater.java:72)
m
There’s no Elasticsearch or Kafka connector for 1.18 yet
o
And the 1.17.1 ones are not compatible?
m
But it could also be related to the Calcite upgrades that exist in Flink 1.18
Not guaranteed no
o
Ok, we might wait for those releases first and then test it out again. Thanks for the help!
m
I’ll also ping someone with more knowledge on the Calcite upgrades
🙌 1
o
Exchanging the 
execute()
to an explain statement ends up in the same exception. So i'm not really sure if we can get some insight into calcite specific errors. Perhaps if we start stripping down our statement sets until it works to see what might be the root cause.
s
it could be related, however not sure yet can you share a test case to reproduce it i guess you have some sql
seems we are still in a lack of tests since none of them detected this issue
😱 1
o
I've found a minimal setup of our code that works (single kafka upsert source 1.17.1, single ES sink and some processing in the middle). Let me add back stuff until it breaks and see if I can give you some insight
👍 1
Ok, small update. after a lot of mocking (we have around 5000 LoCs of various joined tables) I think i've narrowed it down to a 
unionAll
clause. Both tables work individually but a 
unionAll
between them gives the NULL error above. Not sure if this is helpful for you. I will continue investigating on my end
s
thanks for narrowing it down however I think there should be something else i tried different conbinations with `union`/`union all` and was not able to reproduce it so far what I tried
Copy code
CREATE TABLE t (id INT, id2 STRING) WITH ('connector' = 'datagen');
select * from t
union all
select * from t;
also
Copy code
select * from t where id > 0
union all
select * from t where id < 0;
also tried same with just 
union
was not able to reproduce it... i guess there should be something else to make it failing
o
Let me see if there's an edge case involved
8 Views
Open in Slack
PreviousNext
Powered by