Hi all, we're running into issues with JobManager running out of JVM metaspace. As per docs, we're adding dependencies to

/opt/flink/lib

. No issues with postgres driver & the flink jdbc connector, but we're having issues with

flink-connector-kinesis:4.0.0-1.16

(running on Flink image

flink:1.16.0-scala_2.12-java11

). Adding this jar to the lib folder gives the following error:

jobmanager_1   | java.lang.NoClassDefFoundError: Could not initialize class org.apache.flink.kinesis.shaded.com.amazonaws.ClientConfiguration
jobmanager_1   |        at org.apache.flink.kinesis.shaded.com.amazonaws.ClientConfigurationFactory.getDefaultConfig(ClientConfigurationFactory.java:46) ~[flink-connector-kinesis-4.0.0-1.16.jar:4.0.0-1.16]
jobmanager_1   |        at org.apache.flink.kinesis.shaded.com.amazonaws.ClientConfigurationFactory.getConfig(ClientConfigurationFactory.java:36) ~[flink-connector-kinesis-4.0.0-1.16.jar:4.0.0-1.16]
jobmanager_1   |        at org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.createKinesisClient(KinesisProxy.java:268) ~[flink-connector-kinesis-4.0.0-1.16.jar:4.0.0-1.16]
jobmanager_1   |        at org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.<init>(KinesisProxy.java:152) ~[flink-connector-kinesis-4.0.0-1.16.jar:4.0.0-1.16]
jobmanager_1   |        at org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.create(KinesisProxy.java:280) ~[flink-connector-kinesis-4.0.0-1.16.jar:4.0.0-1.16]
jobmanager_1   |        at org.apache.flink.streaming.connectors.kinesis.internals.KinesisDataFetcher.<init>(KinesisDataFetcher.java:412) ~[flink-connector-kinesis-4.0.0-1.16.jar:4.0.0-1.16]

Any thoughts on what might be causing this?

this looks like a version mismatch potentially? can you inspect the jar to see if the right ClientConfiguration is getting loaded at the top level?

It works if we package flink-connector-kinesis within the jar, but it's not working when we mark flink-connector-kinesis as provided and add it to /opt/flink/lib. Version mismatch between what exactly? flink-connector-kinesis is a fat jar and is packaging in the ClientConfiguration, so not sure if that's the issue here?

Oh I see got it so when you package with the jar it works.