Hi guys, We are trying to consume data from Kafka ...
# troubleshooting
t
Hi guys, We are trying to consume data from Kafka using protobuf format, with the SQL Client. Code looks like this:
SET 'state.checkpoints.dir' = 's3://state/checkpoints';
SET 'state.backend.incremental' = 'true';
SET 'execution.checkpointing.unaligned' = 'true';
SET 'execution.checkpointing.mode' = 'EXACTLY_ONCE';
SET 'execution.checkpointing.interval' = '10s';
SET 'execution.checkpointing.min-pause' = '10s';
SET 'sql-client.execution.result-mode'='TABLEAU';
SET 'parallelism.default' = '1';
ADD JAR '/opt/sql-client/lib/flink-sql-connector-kafka-1.16.0.jar';
ADD JAR '/opt/sql-client/lib/flink-protobuf-1.16.0.jar';
ADD JAR '/opt/sql-client/lib/original-kafka-send-proto-0.1.0.jar';
-- SYNTHETIC EVENTS GEO
DROP TABLE IF EXISTS TEST;
CREATE TABLE TEST (
name STRING
) WITH (
'connector' = 'kafka',
'topic' = 'protos_topic_cards',
'properties.bootstrap.servers' = 'kafka_broker:29092',
'format' = 'protobuf',
'protobuf.message-class-name' = 'com.example.CardData',
'protobuf.ignore-parse-errors' = 'true'
);
SELECT * FROM TEST;
And the
original-kafka-send-proto-0.1.0.jar
does contain the protobuf Java class com.example.CardData. Still, the SQL script fails in the SQL Client with the error: Flink SQL> CREATE TABLE TEST (
name STRING
) WITH (
'connector' = 'kafka',
'topic' = 'protos_topic_cards',
'properties.bootstrap.servers' = 'kafka_broker:29092',
'format' = 'protobuf',
'protobuf.message-class-name' = 'com.example.CardData',
'protobuf.ignore-parse-errors' = 'true'
)[INFO] Execute statement succeed.
Flink SQL> [ERROR] Could not execute SQL statement. Reason: java.lang.ClassNotFoundException: com.example.CardData Shutting down the session... done. Anyone ever faced this issue with protobuf?
I do also pass the jar file containing my custom class to the sql client using the command: sql-client.sh -f myjob.sql -j path-to-jar