Hi 👋 I've recently written a Flink POC (Java) to be executed in Kinesis Data Analytics. Now I'm giving PyFlink a try. I have a virtualenv with python 3.10.12 and apache-flink 1.17.x installed. I have a very simple program using the DataStream API, it has to connect to Kinesis, fetch data and print() it. When I try to run the app (from PyCharm or command line), I get a

Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.flink.kinesis.shaded.com.amazonaws.ClientConfiguration

I understand this error comes from the Java ClassLoader, where it can not find the .class file on the class path. I have several JVM's installed and have defined JAVA_HOME as part of the run configuration. The following sentences have no effect

env.add_jars("file:///<path>/flink-connector-kinesis-4.1.0-1.17.jar")
env.add_classpaths("file:///<path>/flink-connector-kinesis-4.1.0-1.17.jar")

If I examine the

flink-connector-kinesis-4.1.0-1.17.jar

file, the

<http://org.apache.flink.kinesis.shaded.com|org.apache.flink.kinesis.shaded.com>.amazonaws.ClientConfiguration.class

file it's there. I'm pretty sure I'm missing something obvious. Any hints will be very welcome. Thanks.

You should use

flink-sql-connector-kinesis-4.1.0-1.17.jar

since it is an Uber jar that contains bundled transitive dependencies.

Got it, I don't have the exception anymore, but it seems that the mini-cluster does not start (the program just hangs). Thanks!

i have working app on kda + pyflink. Dependencies works different than java. If i remember correctly, https://docs.aws.amazon.com/kinesisanalytics/latest/java/how-python.html You need to bundle app code + dependencies (as uber jar) in zip file, upload it to s3 and then use this zip as source for kda flink app Then create kda properties, something like this (in my case i have CDK code)

ka.CfnApplicationV2.PropertyGroupProperty(
    property_group_id="kinesis.analytics.flink.run.options",
    property_map={
        "jarfile": "waf-logs-dependencies-1.15.3.jar",
        "python": "main.py"
    }
)

And then in flink app, use application-properties json, if im not mistaken, dependencies are loaded via this zip file

I also assume the python example is not using KDA, since KDA does not support Flink 1.17.

BTW yes, i use latest possible version 1.15.2 https://docs.aws.amazon.com/kinesisanalytics/latest/java/flink-1-15-2.html