Hi all,
I'm trying to consume protobuf messages from a Confluent cluster. The confluent protobuf format has some magic bytes that make it not possible to use the existing protobuf deserializer. So I implemented my own ProtobufDeserializationSchema in Java that removes the first few bytes and deserializes the messages correctly. However, I need to make this accessible with pyflink as well.
From what I can see creating a custom DeserializationSchema
may not be possible in pyflink, so I'm wondering if I can package my ProtobufDeserializationSchema in a jar and access it using
pyflink.java_gateway
as it's done in the pyflink source.
Does anyone have experience with that?