it seems like it is ignoring my registrations? wit...
# troubleshooting
c
it seems like it is ignoring my registrations? with the debugger I can see this entry in
kryoRegistrations
Copy code
"streams.tis.trust.fhpstream.filtered_hydra_payload.RequestSource" -> {KryoRegistration@26135} 
 key = "streams.tis.trust.fhpstream.filtered_hydra_payload.RequestSource"
 value = {KryoRegistration@26135} 
  registeredClass = {Class@7694} "class streams.tis.trust.fhpstream.filtered_hydra_payload.RequestSource"
  serializerClass = {Class@21986} "class com.grab.grabdefence.acorn.proto.ScalaPbEnumSerializer"
  serializableSerializerInstance = null
  serializerDefinitionType = {KryoRegistration$SerializerDefinitionType@26334} "CLASS"
but for some reason a different serializer is used
Copy code
type = {Class@7694} "class streams.tis.trust.fhpstream.filtered_hydra_payload.RequestSource"
kryo = {KryoBase@21592} com.twitter.chill.KryoBase@7a852
 objSer = {ObjectSerializer@26028} com.twitter.chill.ObjectSerializer@11d3192e
with a debug checkpoint on
copy
here:
Copy code
try {
            checkKryoInitialized();
            try {
                return kryo.copy(from);
            } catch (KryoException ke) {
a
If you are in control of the classes you use when deserializing the payload, I'd suggest you convert them into regular POJOs. The POJO serializer is much more performant than Kryo, and it able to handle simple schema evolution (e.g. removal/addition of fields). Look up in the docs what Flink requires of POJO: https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#pojos
com.google.protobuf.ByteString
is not a POJO it seems. Replace that with a regular
String
or
byte[]
.
Once you have converted your classes to POJOs, or another supported format, I would recommend you to disable Kryo entirely, so you won't accidentally run into this issue again. This can for instance be done like this:
Copy code
Configuration config = new Configuration();

// ensures that we never use Kryo serialization. Kryo serialization doesn't allow for schema evolution.
// See <https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/datastream/fault-tolerance/serialization/schema_evolution/#kryo-cannot-be-used-for-schema-evolution>
config.set(PipelineOptions.GENERIC_TYPES, false);

var env = StreamExecutionEnvironment.getExecutionEnvironment(config)
c
I am not in control of the classes since they are generated from the protobuf definitions. e.g. a protobuf field
string  id                          = 1;
is automatically backed by a
com.google.protobuf.ByteString
in the compiled java classes for that proto message.
I think at least for my RequestSource enum I was able to figure out what happened, I need to register the serializer for each enum value individually
a
Ok. If you're concerned about performance (I got that vibe since you posted a flame graph) then you should reconsider your approach of using Kryo serialization in your pipeline. That being said. If you don't mind using Kryo I believe there's a 3rd party serializer made for protobuf. See https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/third_party_serializers/
I've never used it, so I have no clue if it's any good or how it works.
c
Yeah I'm trying to avoid using Kryo Serialization. I wrote my own serializers (similar to the twitter chill ones from that third party page) and registered them, I'm investigating why it doesn't always seem to work
I wrote a custom serializer for `ByteString`s and registered it but I still get the aforementioned log entries so I'm wondering what that's about.
a
I reckon you'd still being used Kryo even with your custom serializer - and get poor performance and no schema evolution. Not sure what's going on the you kryo serializer config. I always stay clear of Kryo entirely. So I can't provide any guidance on that part. Another approach to consider is deserialize directly at the source to a regular POJO type by implementing your own deserializer. Then you'd avoid using Kryo entirely. E.g. when using Kafa:
Copy code
KafkaSource.<RegularPOJO>builder().setDeserializer(new ProtobufToPojoDeserializer())
c
hmm the way I understood this page https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/third_party_serializers/ is that if you register your own serializer with Kryo, it doesn't actually still use kryo serialization any more. Kryo is still used to manage the registration of serializers and that cannot be disabled (but it's also not an issue)
a
If you've implemented and registered a
com.esotericsoftware.kryo.Serializer
then you're very much still using Kryo. From your debugger info it looks like that's what you did with the
ScalaPbEnumSerializer
? Since you're already relying on Kryo, why not use the existing Kryo protobuf serializer provided in the
chill-protobuf
library? If you'd want keep going down the custom serialization route, and avoid Kryo, you'd havde to implement a
org.apache.flink.api.common.typeutils.TypeSerializer
which is super bare bones and very low level. Here you're dealing with byte level representations of your type. When using POJO serialization Flink already has serializers for most of the commonly used Java types.
c
chill-protobuf
is for the java
Message
protobuf classes, our project however uses the scala
GeneratedMessage
protobuf classes, so I did an implementation for that. Unfortunately out of the box Flink is not very good at handling the
GeneratedMessage
type so we had to write quite a bit of the serialization on our own