Hi everyone, I noticed that the classes in the flink-json package that relate to SerDe were deprecated in favor of the Table API. I was wondering, does this mean I shouldn’t be working with JSON formats in the datastream layer? I am trying to stick to the datastream layer as it feels more stable and responsive than other abstractions I’ve worked with (eg. table api, pyflink)

Which classes specifically are you referring to?

hi Chesnay, here is one example: https://github.com/apache/flink/blob/master/flink-formats/flink-json/src/main/java[…]org/apache/flink/formats/json/JsonRowDeserializationSchema.java

You were using the Rows in a datastream application?

I am still developing but I have been pushing Row’s and JsonNode’s through my pipeline

org.apache.flink.types.Row

Rows aren't really intended to be used in a pure DataStream application and are more a Table API thing. What do you hope to gain from Rows over just working with JsonNodes?

I’m still piecing that together but basically I need to clean the incoming data according to a runtime json schema. I found the class in that same package that creates a RowTypeInfo from a JSON schema and the next thing I’m trying to do is use that type info to clean data (populate missing fields with null, drop excess fields, and custom type conversion if possible

The runtime json schema is unknown upfront?

I guess so, I was only planning to convert the JsonNode to a Row because that was what I thought I had to do when I saw the class that converts a JSON Schema into a RowTypeInfo But if Rows are not really a data stream construct then I can continue to work with two JsonNode classes - one for schema and one for data. I did start doing this earlier but it’s a lot of work and I thought maybe if I moved into the Row-based approach if I could reuse existing patterns, eg the way createSerializer in RowTypeInfo creates a tree of serializers seems like a very useful pattern for what I’m building