Hi, Hoping somebody could help me as I am brand ne...
# troubleshooting
y
Hi, Hoping somebody could help me as I am brand new to flink, we are trying to evaluate it to see if it can replace Spark for certain use cases. We are using pyflink table API (with the k8 operator) and we want to read data from our Kafka Cluster (Confluent). Now all our schemas are in protobuf and for some reason I keep running into an issue while defining the protobuf format.
CREATE TABLE test(
*****
) WITH (
'connector' = 'kafka',
'topic' = 'engagement-mobius-publish_ruleengine-in-log-v1',
'properties.bootstrap.servers' = '****:9092',
'properties.group.id' = 'python-demo-stream-flink',
'scan.startup.mode' = 'earliest-offset',
'format' = 'protobuf',
'protobuf.message-class-name' = 'iit.datahub.engagement.mobius.v1.PublishMessageEventOuterClass.PublishMessageEvent',
'protobuf.ignore-parse-errors' = 'true'
I have tried importing the JAR for the protobuf version in a normal java app, that works fine. With above code I get the following in k8 ( see screenshot) Our java files are compiled with protobuf v 3.20.1, while I can see that flink 1.17 uses 3.21.7, does that make a difference ?
Also forgot, the JAR with the protobuf images is copied into /opt/flink/lib/ in the docker img
Also the stack trace ends like this -->
Through Trial and error I was able to solve the above issue, but now I am getting a new error -->
Solved it by adding protobuf-java-3.20.1.jar to the docker image πŸ™‚ No more follow up needed