L P V
10/01/2025, 12:03 PM'format' = 'protobuf' for a Kafka source table. The CREATE TABLE succeeds, but EXPLAIN PLAN FOR SELECT * fails with ClassNotFoundException during schema inference/planning. This happens in a distributed setup with Flink SQL Gateway and a remote Flink Session Cluster. I've tried dynamic JAR loading via GCS (gs://) but the class isn't resolvable at planning time, even though GCS access works (no filesystem errors).
Setup:
• Flink version: 1.19.1
• SQL Client startup: ./bin/sql-client.sh gateway --endpoint 0.0.0.0:8083 -Drest.address=xxxxx -Drest.port=8081
• JAR: trackify_proto_file_deploy.jar (built from Protobuf-generated code via Maven/Gradle). Verified contents with `jar -tvf`: Includes vn/com/momo/mle/proto/v1/UserCountResult.class and inner classes (e.g., UserCountResult$1.class). It's a fat JAR with protobuf-java shaded in.
Steps to Reproduce:
• Start SQL Client (as above) and connect to Gateway session.
• Add JAR and set config:
ADD JAR 'gs://..../trackify_proto_file_deploy.jar';
SET 'pipeline.jars' = 'gs://ml-game-short-term-storage-sing/flink_sql/jars/trackify_proto_file_deploy.jar';
◦ Both succeed: [INFO] Execute statement succeed. / [INFO] JAR added successfully.
• Create table (parses fine, registers in catalog):
CREATE TABLE src.kafka.test (
.......
) WITH (
'connector' = 'kafka',
'topic' = '',
'properties.bootstrap.servers' = '...',
'format' = 'protobuf',
'protobuf.message-class-name' = 'vn.com.momo.mle.proto.v1.UserCountResult',
'properties.security.protocol' = 'SASL_PLAINTEXT',
'properties.sasl.mechanism' = 'SCRAM-SHA-512',
'properties.group.id' = ',
'properties.sasl.jaas.config' = '...',
'scan.startup.mode' = 'earliest-offset'
);
Succeeds: [INFO] Execute statement succeed.
• Run planning:
EXPLAIN PLAN FOR SELECT * FROM src.kafka.test;
• Fails with: [ERROR] Could not execute SQL statement. Reason: java.lang.ClassNotFoundException: vn.com.momo.mle.proto.v1.UserCountResult
What I've Tried (No Redeploy):
• SHOW JARS; lists the added JAR.
Question/Goal:
• Why does Protobuf class loading fail at planning time (EXPLAIN PLAN) in a Gateway + remote Session Cluster, even with pipeline.jars set to a resolvable URL (gs:// or HTTP via REST upload)? Is planning strictly client-side, or does the Gateway/JM need additional setup for dynamic JARs in custom formats like Protobuf?
• How to dynamically add/update Protobuf JARs (e.g., for schema evolution in Kafka topics) without redeploying the Flink image or putting in /opt/flink/lib/?