Clemens Valiente
08/25/2025, 4:40 AMkryoRegistrations
"streams.tis.trust.fhpstream.filtered_hydra_payload.RequestSource" -> {KryoRegistration@26135}
key = "streams.tis.trust.fhpstream.filtered_hydra_payload.RequestSource"
value = {KryoRegistration@26135}
registeredClass = {Class@7694} "class streams.tis.trust.fhpstream.filtered_hydra_payload.RequestSource"
serializerClass = {Class@21986} "class com.grab.grabdefence.acorn.proto.ScalaPbEnumSerializer"
serializableSerializerInstance = null
serializerDefinitionType = {KryoRegistration$SerializerDefinitionType@26334} "CLASS"
but for some reason a different serializer is used
type = {Class@7694} "class streams.tis.trust.fhpstream.filtered_hydra_payload.RequestSource"
kryo = {KryoBase@21592} com.twitter.chill.KryoBase@7a852
objSer = {ObjectSerializer@26028} com.twitter.chill.ObjectSerializer@11d3192e
with a debug checkpoint on copy here:
try {
checkKryoInitialized();
try {
return kryo.copy(from);
} catch (KryoException ke) {Andreas Bube
08/25/2025, 5:54 AMcom.google.protobuf.ByteString is not a POJO it seems. Replace that with a regular String or byte[].Andreas Bube
08/25/2025, 5:59 AMConfiguration config = new Configuration();
// ensures that we never use Kryo serialization. Kryo serialization doesn't allow for schema evolution.
// See <https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/datastream/fault-tolerance/serialization/schema_evolution/#kryo-cannot-be-used-for-schema-evolution>
config.set(PipelineOptions.GENERIC_TYPES, false);
var env = StreamExecutionEnvironment.getExecutionEnvironment(config)Clemens Valiente
08/25/2025, 6:06 AMstring id = 1; is automatically backed by a com.google.protobuf.ByteString in the compiled java classes for that proto message.Clemens Valiente
08/25/2025, 6:07 AMAndreas Bube
08/25/2025, 6:14 AMAndreas Bube
08/25/2025, 6:15 AMClemens Valiente
08/25/2025, 6:17 AMClemens Valiente
08/25/2025, 6:21 AMAndreas Bube
08/25/2025, 6:45 AMKafkaSource.<RegularPOJO>builder().setDeserializer(new ProtobufToPojoDeserializer())Clemens Valiente
08/25/2025, 7:16 AMAndreas Bube
08/25/2025, 9:20 AMcom.esotericsoftware.kryo.Serializer then you're very much still using Kryo. From your debugger info it looks like that's what you did with the ScalaPbEnumSerializer? Since you're already relying on Kryo, why not use the existing Kryo protobuf serializer provided in the
chill-protobuf library?
If you'd want keep going down the custom serialization route, and avoid Kryo, you'd havde to implement a org.apache.flink.api.common.typeutils.TypeSerializer which is super bare bones and very low level. Here you're dealing with byte level representations of your type.
When using POJO serialization Flink already has serializers for most of the commonly used Java types.Clemens Valiente
08/25/2025, 9:47 AMchill-protobuf is for the java Message protobuf classes, our project however uses the scala GeneratedMessage protobuf classes, so I did an implementation for that.
Unfortunately out of the box Flink is not very good at handling the GeneratedMessage type so we had to write quite a bit of the serialization on our own