Hey Guys, I am trying to move from Standalone deployment to use Flink operator. I am using a custom Flink image where we copy over the job jar from

/opt/lib

to

/opt/flink/lib

. In the

flinkdeployment

custom resource then I refer job like below -

job:
    jarURI: local:///opt/flink/lib/myjob-all.jar
    parallelism: 4
    upgradeMode: 'savepoint'
    state: 'running'

In this case, I see the following exception and the job manager pod goes to failed state -

flink-main-container Exception in thread "main" java.lang.VerifyError: Instruction type does not match stack map
 flink-main-container Exception Details:
 flink-main-container   Location:
 flink-main-container     org/apache/flink/util/FlinkUserCodeClassLoaders.wrapWithSafetyNet(Lorg/apache/flink/util/FlinkUserCodeClassLoader;Z)Lorg/apache/flink/util/MutableURLClassLoader; @20: areturn
 flink-main-container   Reason:
 flink-main-container     Type 'org/apache/flink/util/FlinkUserCodeClassLoader' (current frame, stack[0]) is not assignable to 'org/apache/flink/util/MutableURLClassLoader' (stack map, stack[0])
 flink-main-container   Current Frame:
 flink-main-container     bci: @20
 flink-main-container     flags: { }
 flink-main-container     locals: { 'org/apache/flink/util/FlinkUserCodeClassLoader', integer }
 flink-main-container     stack: { 'org/apache/flink/util/FlinkUserCodeClassLoader' }
 flink-main-container   Stackmap Frame:
 flink-main-container     bci: @20
 flink-main-container     flags: { }
 flink-main-container     locals: { 'org/apache/flink/util/FlinkUserCodeClassLoader', integer }
 flink-main-container     stack: { 'org/apache/flink/util/MutableURLClassLoader' }
 flink-main-container   Bytecode:
 flink-main-container     0000000: 1b99 0012 bb00 1d59 2a2a b600 1eb7 001f
 flink-main-container     0000010: a700 042a b0                           
 flink-main-container   Stackmap Table:
 flink-main-container     same_frame(@19)
 flink-main-container     same_locals_1_stack_item_frame(@20,Object[#84])
 flink-main-container 
 flink-main-container    at org.apache.flink.client.ClientUtils.buildUserCodeClassLoader(ClientUtils.java:63)
 flink-main-container    at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:145)
 flink-main-container    at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:65)
 flink-main-container    at org.apache.flink.client.program.PackagedProgram$Builder.build(PackagedProgram.java:691)
 flink-main-container    at org.apache.flink.client.program.DefaultPackagedProgramRetriever.getPackagedProgram(DefaultPackagedProgramRetriever.java:213)
 flink-main-container    at org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint.getPackagedProgram(KubernetesApplicationClusterEntrypoint.java:100)
 flink-main-container    at org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint.main(KubernetesApplicationClusterEntrypoint.java:70)

However, if I refer the job from

/opt/flink

, everything works fine.

If you add your jar file to

/opt/flink/lib

you do not need to specify it under

jarURI

(just skip it).

Thanks @sap1ens I will try that.

This exact configuration works for me. I’m on the older version of the operator though (1.3).

no, I have not. is this not compatible with native-kubernetes ?

@Ravi Nishant why are you copying your user jar under flink/libs ? that should not be necessary. Just keep it outside and use jarURI

@Gyula Fóra We recently moved from standalone docker to native-k8s using operator. I intended to do exactly what you said until we saw that logger context is not being initialized. We are using logback. Logging used to work in standalone mode when we were copying the jar to

/opt/flink/lib

. I was just trying to see if that makes a difference. (was suspecting issue could be related to class loading)

flink-main-container Starting kubernetes-application as a console application on host clientperfs3job-jm-ue1a-6b9f6d4bb5-7kjfq.
flink-main-container Failed to instantiate [ch.qos.logback.classic.LoggerContext]
flink-main-container Reported exception:
flink-main-container java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/util/ByteArrayBuilder
clientperfs3job-jm-ue1a-6b9f6d4bb5-7kjfq flink-main-container   at net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder.createFormatter(LoggingEventCompositeJsonEncoder.java:27)
flink-main-container   at net.logstash.logback.encoder.CompositeJsonEncoder.<init>(CompositeJsonEncoder.java:48)
flink-main-container   at net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder.<init>(LoggingEventCompositeJsonEncoder.java:23)
flink-main-container   at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

there problem is likely with the logger jars included in your flink base image under lib. I would try to create a custom flink base image where you remove the log4j jars and include logback instead.

yes, probably need to look at customizing our flink image appropriately. Thanks for the pointer.