Tom Oram
05/17/2025, 9:48 AMmain
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
KafkaSourceString source = KafkaSource.Stringbuilder()
.setBootstrapServers("kafka.kafka.svc.cluster.local:9092")
.setProperty("sasl.mechanism", "SCRAM-SHA-256")
.setProperty("security.protocol", "SASL_PLAINTEXT")
.setProperty("sasl.jaas.config", "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"user\" password=\"pass\";")
.setTopics("input-topic")
.setGroupId("my-group")
.setStartingOffsets(OffsetsInitializer.earliest())
.setValueOnlyDeserializer(new SimpleStringSchema())
.build();
KafkaSinkString sink = KafkaSink.Stringbuilder()
.setBootstrapServers("kafka.kafka.svc.cluster.local:9092")
.setProperty("sasl.mechanism", "SCRAM-SHA-256")
.setProperty("security.protocol", "SASL_PLAINTEXT")
.setProperty("sasl.jaas.config", "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"user\" password=\"pass\";")
.setRecordSerializer(KafkaRecordSerializationSchema.builder()
.setTopic("output-topic")
.setValueSerializationSchema(new SimpleStringSchema())
.build()
)
.setDeliveryGuarantee(DeliveryGuarantee.NONE)
.build();
DataStreamString stream = env.fromSource(source, WatermarkStrategy.noWatermarks(), "Kafka Source");
stream.map(String::toUpperCase).sinkTo(sink);
env.execute("Uppercase Kafka Stream");
It compiles fine, but when I submit it to a 2.0.0 cluster running in Minikube, I get the following error:
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Failed to execute job 'Uppercase Kafka Stream'.
Caused by: org.apache.flink.util.FlinkException: Failed to execute job 'Uppercase Kafka Stream'.
Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to submit ExecutionPlan.
Caused by: org.apache.flink.runtime.rest.util.RestClientException: [Internal server error., <Exception on server side: org.apache.flink.runtime.client.JobSubmissionException: Failed to submit job.
Caused by: java.lang.RuntimeException: java.io.InvalidObjectException: ReflectiveOperationException during deserialization
Caused by: java.lang.reflect.InvocationTargetException
Caused by: java.lang.IllegalArgumentException: Invalid lambda deserialization
This only happens when I add the sinkTo, if I replace it with a print then it works.
Can anyone help me determine the cause?
Thanks.Tom Oram
05/17/2025, 9:05 PMAndreas Bube
05/22/2025, 8:21 AMReflectiveOperationException makes me think it's related to JDK modularization and JVM Reflection configuration. Have a look at how the default Java Opts are configured here: https://github.com/apache/flink/blob/4c77daaf3554446e67f6b75ac18d54da84208b9a/flink-dist/src/main/resources/config.yaml#L24. Perhaps the Bitnami Helm chart doesn't include these Java Opts?Andreas Bube
05/22/2025, 8:26 AM2025-05-22 08:04:43,847 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] JVM Options:
2025-05-22 08:04:43,848 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -Xmx1530082096
2025-05-22 08:04:43,849 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -Xms1530082096
2025-05-22 08:04:43,850 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -XX:MaxMetaspaceSize=268435456
2025-05-22 08:04:43,850 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] -XX:+IgnoreUnrecognizedVMOptions
2025-05-22 08:04:43,851 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] --add-exports=java.base/sun.net.util=ALL-UNNAMED
2025-05-22 08:04:43,852 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] --add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED
2025-05-22 08:04:43,853 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED
2025-05-22 08:04:43,854 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED
2025-05-22 08:04:43,854 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED
2025-05-22 08:04:43,855 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED
2025-05-22 08:04:43,856 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED
2025-05-22 08:04:43,857 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] --add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED
2025-05-22 08:04:43,857 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] --add-opens=java.base/java.lang=ALL-UNNAMED
2025-05-22 08:04:43,869 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] --add-opens=java.base/java.net=ALL-UNNAMED
2025-05-22 08:04:43,893 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] --add-opens=java.base/java.io=ALL-UNNAMED
2025-05-22 08:04:43,894 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] --add-opens=java.base/java.nio=ALL-UNNAMED
2025-05-22 08:04:43,894 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] --add-opens=java.base/sun.nio.ch=ALL-UNNAMED
2025-05-22 08:04:43,895 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] --add-opens=java.base/java.lang.reflect=ALL-UNNAMED
2025-05-22 08:04:43,896 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] --add-opens=java.base/java.text=ALL-UNNAMED
2025-05-22 08:04:43,896 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] --add-opens=java.base/java.time=ALL-UNNAMED
2025-05-22 08:04:43,897 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] --add-opens=java.base/java.util=ALL-UNNAMED
2025-05-22 08:04:43,898 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
2025-05-22 08:04:43,909 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
2025-05-22 08:04:43,910 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMEDTom Oram
05/26/2025, 7:59 AM