https://flink.apache.org/ logo
Join Slack
Powered by
I'm using Flink 1.16.1 . Seeing an error: ```2023-...
# troubleshooting
b
I'm using Flink 1.16.1 . Seeing an error:
Copy code
2023-05-23 00:26:48,863 WARN  org.apache.flink.runtime.webmonitor.handlers.JarRunHandler   [] - Configuring the job submission via query parameters is deprecated. Please migrate to submitting a JSON request instead.
2023-05-23 00:26:48,931 INFO  org.apache.flink.client.ClientUtils                          [] - Starting program (detached: true)
2023-05-23 00:26:49,101 ERROR org.apache.flink.runtime.webmonitor.handlers.JarRunHandler   [] - Exception occurred in REST handler: Could not execute application.
I don't see any stack trace in the logs or in the UI logs . This error is seen in jobmanager log. Any idea how to find the root cause for this ?
The code I'm trying to run is to read from a google pubsub topic
Copy code
val env = StreamExecutionEnvironment.getExecutionEnvironment
    env.enableCheckpointing(60000, CheckpointingMode.AT_LEAST_ONCE)
    env.getCheckpointConfig.enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION)


    val deserializer = new SimpleStringSchema()
    val pubsubSource = PubSubSource.newBuilder()
      .withDeserializationSchema(deserializer)
      .withProjectName("project")
      .withSubscriptionName("subscriber")
      .build()

    val pubsubStream = env.addSource(pubsubSource)

    pubsubStream.print()
I enabled debug logs and got some error messages. It appears the issue is:
Copy code
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_372]
Caused by: java.util.concurrent.CompletionException: java.lang.NoClassDefFoundError: com/google/protobuf/GeneratedMessageV3
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) ~[?:1.8.0_372]
I excluded all transitive dependencies for this jar 
protobuf-java
that includes this class except here:
Copy code
com.roku.mlp:flink-stream-from-gcs:jar:1.0
[INFO] \- org.apache.flink:flink-connector-gcp-pubsub:jar:1.16.1:compile
[INFO]    \- com.google.cloud:google-cloud-core:jar:1.93.7:compile
[INFO]       \- com.google.protobuf:protobuf-java:jar:3.12.2:compile
Still I'm getting same error while running the app
Update: After resolving some jars, I'm still facing similar conflicts
Copy code
switched from INITIALIZING to FAILED on 100.65.216.70:38319-553add @ 100.65.216.70 (dataPort=36005).
java.lang.LinkageError: loader constraint violation: when resolving overridden method "com.google.pubsub.v1.PullRequest.getDefaultInstanceForType()Lcom/google/protobuf/MessageLite;" the class loader (instance of org/apache/flink/util/ChildFirstClassLoader) of the current class, com/google/pubsub/v1/PullRequest, and its superclass loader (instance of sun/misc/Launcher$AppClassLoader), have different Class objects for the type com/google/protobuf/MessageLite used in the signature
Not sure if I need to do any shading in the app to use pubsub-connector ? Any help appreciated
Open in Slack
PreviousNext
Powered by