https://flink.apache.org/ logo
Join Slack
Powered by
i am a bit confused with what KDA gives out of the...
# troubleshooting
f
i am a bit confused with what KDA gives out of the box in PyFlink land. trying to continue upgrading an app we have to 1.15.2 and now I get this problem:
Copy code
2023-04-19 15:49:21
java.lang.NoSuchMethodError: 'org.apache.flink.metrics.MetricGroup org.apache.flink.api.common.functions.RuntimeContext.getMetricGroup()'
	at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:765)
we have a pyflink app that bundles with a fat jar of some extra utilities where we do stuff in java-land, e.g. deserialisation stuff, and so on. 
flink-streaming-java
is the only dependency in the POM.xml file that is marked as provided. our dependencies are: flink-streaming-java, flink-java, flink-core, flink-connector-kafka, kafka-clients (2.8.1). everything is set to be version 1.15.2 (other than the kafka-clients). any ideas what is up here? thanks!
https://stackoverflow.com/questions/72505052/java-lang-nosuchmethoderror-org-apache-flink-metrics-metricgroup-org-apache-fl this points to needing 
statefun
. is this relevant for 1.15.2? we don't use stateful functions in our submitted jobs so not sure if this is an internal thing or not
m
Is the Kafka Clients version the same as Flink Connector Kafka uses?
It’s most likely not Statefun but another artifact that’s missing
One that’s used by the Flink Kafka connector
f
Hey, thanks for the reply - they are all on 1.15.2 at the moment
d
@Felix Angell I guess the connector version isn’t consistent with the Flink version. Could you double check that? There is an API change for 
RuntimeContext.getMetricGroup
, see https://github.com/apache/flink/blob/4799111928a74f73b96292560440f92f18caf0cf/flin[…]/java/org/apache/flink/api/common/functions/RuntimeContext.java for more details
f
In our jar that we bundle in the PyFlink application lib directory we have a dependency in the pom to this: 
<artifactId>flink-connector-kafka</artifactId>
which is specified to 1.15.2. @Hong Teoh sorry to tag, is there anything you know around the runtime dependencies versions on PyFlink on KDA? Afaik all of this is out of the box so I'm wondering if these 1.15.2 dependencies conflict with something defined at runtime?
👀 1
i think i've figured something out... because we use FlinkKafkaConsumer (from the connectors lib) it was deprecated as of 1.14.x and I think is incompatible with the changes above to the RuntimeContext
alright so it's now using KafkaSource instead which i've also had to backport to 1.15. as the python mappings were only included in 1.16
and now the same problem 🤦‍♂️
Copy code
2023-04-20 15:00:07
java.lang.NoSuchMethodError: 'org.apache.flink.metrics.MetricGroup org.apache.flink.api.connector.source.SourceReaderContext.metricGroup()'
	at org.apache.flink.connector.kafka.source.KafkaSource.createReader(KafkaSource.java:138)
	at org.apache.flink.streaming.api.operators.SourceOperator.initReader(SourceOperator.java:286)
our fat jar we have in the lib/ dir i've literally decompiled and can see KafkaSource and all of the methods we're looking for
unless, i need to provide flink-metrics/flink-metrics-core as a dependency?
h
Taking a look. This is indeed curious.
unless, i need to provide flink-metrics/flink-metrics-core as a dependency? (edited)
No, this should not be needed. It seems to be complaining that the 
RuntimeContext
does not have method 
getMetricGroup()
(present in 1.15.2) and 
SourceReaderContext
does not have method 
metricGroup()
(also present in 1.15.2) My suspicion is now that either the runtime your app is on is not Flink 1.15, OR that the fatjar somehow has transitive dependencies that is <1.15 Do you mind PM-ing the pom.xml?
f
runtime is set to (copy pasted from aws) Apache Flink 1.15
will ping the pom over, cheers!
mystery solved with the help of Hong! turns out the simplest answer is the right one. somehow our build has put in the old runtime of 1.13.2 into the mix so all the flink engine stuff is being resolved to a muuuch older version 🤦‍♂️ thanks everyone for the help
🎉 3
6 Views
Open in Slack
PreviousNext
Powered by