Hi, In my project I have a flink-table wrapper aro...
# troubleshooting
n
Hi, In my project I have a flink-table wrapper around the functionality offered by my project. Since Flink 1.17.0 the normal Flink wrapper still works fine, yet the flink-table wrapper fails during the build with mostly:
Copy code
Error:  nl.basjes.parse.useragent.flink.table.TestTableFunctionClientHints.testMapFunctionReturnMap  Time elapsed: 5.42 s  <<< ERROR!
java.lang.IllegalArgumentException: fromIndex(2) > toIndex(0)
	at java.base/java.util.AbstractList.subListRangeCheck(AbstractList.java:509)
	at java.base/java.util.AbstractList.subList(AbstractList.java:497)
	at org.apache.calcite.rel.metadata.janino.CacheGeneratorUtil$CacheKeyStrategy$1.safeArgList(CacheGeneratorUtil.java:213)
	at org.apache.calcite.rel.metadata.janino.CacheGeneratorUtil$CacheKeyStrategy$1.cacheKeyBlock(CacheGeneratorUtil.java:205)
	at org.apache.calcite.rel.metadata.janino.CacheGeneratorUtil.cachedMethod(CacheGeneratorUtil.java:68)
	at org.apache.calcite.rel.metadata.janino.RelMetadataHandlerGeneratorUtil.generateHandler(RelMetadataHandlerGeneratorUtil.java:121)
	at org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.generateCompileAndInstantiate(JaninoRelMetadataProvider.java:138)
	at org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.lambda$static$0(JaninoRelMetadataProvider.java:73)
	at org.apache.flink.calcite.shaded.com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:165)
	at org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
	at org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
	at org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
	at org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
	at org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache.get(LocalCache.java:3951)
	at org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
	at org.apache.flink.calcite.shaded.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
	at org.apache.calcite.rel.metadata.JaninoRelMetadataProvider.revise(JaninoRelMetadataProvider.java:197)
	at org.apache.calcite.rel.metadata.RelMetadataQueryBase.revise(RelMetadataQueryBase.java:118)
	at org.apache.calcite.rel.metadata.RelMetadataQuery.getPulledUpPredicates(RelMetadataQuery.java:844)
	at org.apache.calcite.rel.rules.ReduceExpressionsRule$ProjectReduceExpressionsRule.onMatch(ReduceExpressionsRule.java:307)
	at org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:337)
...
I have put the update to Flink 1.17 in a separate branch till I figure out what is wrong https://github.com/nielsbasjes/yauaa/pull/991 Any pointers on how I can fix this are welcome.
b
I have the same problem
s
i guess it could relate to update of Calcite but sticking to the older janino version in 1.17.0 however not sure... it would be great to have something reproducible. i tried with https://github.com/nielsbasjes/yauaa however currently locally it fails
Copy code
Could not resolve dependencies for project nl.basjes.parse.useragent:yauaa-logstash:jar:7.17.1-SNAPSHOT: The following artifacts could not be resolved: org.logstash:logstash-core:jar:8.6.2, nl.basjes.parse.useragent:yauaa:jar:7.17.1-SNAPSHOT: org.logstash:logstash-core:jar:8.6.2 was not found in <https://repo.maven.apache.org/maven2> during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of central has elapsed or updates are forced
what am i doing wrong?
just a guess: not sure how difficult is to try it in 1.18.x Janino was updated to the latest available version [1] will it help? [1] https://issues.apache.org/jira/browse/FLINK-27995
n
@Sergey Nuyanzin That dependency problem has to do with maven 3.9.0 and a problem with the maven install plugin. For this case you can just remove the UDFs you are not interested in https://github.com/nielsbasjes/yauaa/blob/main/udfs/pom.xml#L79
Just delete all modules in that file that are not flink.
I have updated https://github.com/nielsbasjes/yauaa/pull/991 to have those changes so you can reproduce it.
s
Thanks, i was able to reproduce it. It looks like Calcite expects methods with 2+ args for
Metadata
and caches them. At the same time for some reason JaCoCo generates
$jacocoInit()
like e.g.
private static boolean[] org.apache.calcite.rel.metadata.BuiltInMetadata$Predicates$Handler.$jacocoInit()
Seems it is the reason, I disabled JaCoCo for flink-table and it starts passing.
n
Yes, that did the trick. Thank you !
🎉 1
r
Hi @Niels Basjes, I’m facing the same issue with calcite Metadata.
java.lang.AssertionError
at org.apache.calcite.rel.metadata.MetadataDef.<init>(MetadataDef.java:48)
at org.apache.calcite.rel.metadata.MetadataDef.of(MetadataDef.java:64)
at org.apache.calcite.rel.metadata.BuiltInMetadata$PercentageOriginalRows.<clinit>(BuiltInMetadata.java:345)
at org.apache.calcite.rel.metadata.RelMdPercentageOriginalRows$RelMdPercentageOriginalRowsHandler.getDef(RelMdPercentageOriginalRows.java:231)
at org.apache.calcite.rel.metadata.ReflectiveRelMetadataProvider.reflectiveSource(ReflectiveRelMetadataProvider.java:134)
Can you please share how you added the exclusion?