Hi, I see that `flink-avro` deserializes to SQL Ta...
# troubleshooting
c
Hi, I see that
flink-avro
deserializes to SQL Tables using the position in the avro record. Since with avro we have the schema and the field names, is there a possibility to map by field names instead ? The current converter leads to issues when the schema evolves.
m
Are you looking at
flink-avro
or
flink-sql-avro
?
IIRC, schema evolution is only supported with the Confluent Avro implementation
c
I'm looking at
org.apache.flink.formats.avro.AvroToRowDataConverters
in flink-avro
m
I'm not sure what you're trying to achieve. There's still the Flink SQL Catalog that you need to take into account. A change in the Avro schema wouldn't automatically result in a change in the Catalog
c
It wouldn't necessarily be a problem if the mapping is done by name and the change is backward compatible.
This is the code for Avro:
Copy code
public static AvroToRowDataConverter createRowConverter(RowType rowType) {
        final AvroToRowDataConverter[] fieldConverters =
                rowType.getFields().stream()
                        .map(RowType.RowField::getType)
                        .map(AvroToRowDataConverters::createNullableConverter)
                        .toArray(AvroToRowDataConverter[]::new);
        final int arity = rowType.getFieldCount();

        return avroObject -> {
            IndexedRecord record = (IndexedRecord) avroObject;
            GenericRowData row = new GenericRowData(arity);
            for (int i = 0; i < arity; ++i) {
                // avro always deserialize successfully even though the type isn't matched
                // so no need to throw exception about which field can't be deserialized
                row.setField(i, fieldConverters[i].convert(record.get(i)));
            }
            return row;
        };
    }
It casts to IndexedRecord but I think it's actually a GenericRecord and the mapping could be made by field names
By comparison, for JSON:
Copy code
public JsonToRowDataConverter createRowConverter(RowType rowType) {
        final JsonToRowDataConverter[] fieldConverters =
                rowType.getFields().stream()
                        .map(RowType.RowField::getType)
                        .map(this::createConverter)
                        .toArray(JsonToRowDataConverter[]::new);
        final String[] fieldNames = rowType.getFieldNames().toArray(new String[0]);

        return jsonNode -> {
            ObjectNode node = (ObjectNode) jsonNode;
            int arity = fieldNames.length;
            GenericRowData row = new GenericRowData(arity);
            for (int i = 0; i < arity; i++) {
                String fieldName = fieldNames[i];
                JsonNode field = node.get(fieldName);
                try {
                    Object convertedField = convertField(fieldConverters[i], fieldName, field);
                    row.setField(i, convertedField);
                } catch (Throwable t) {
                    throw new JsonParseException(
                            String.format("Fail to deserialize at field: %s.", fieldName), t);
                }
            }
            return row;
        };
    }
IIRC, schema evolution is only supported with the Confluent Avro implementation
Can this be used outside of the Confluent registry ?
m
Yeah, if you use the Kafka Schema Registry
c
I'm not πŸ˜…
m
What are you using as a schema registry store?
c
You won't like it πŸ˜„.
I'm using the flink-pulsar connector.
Pulsar has a schema registry. But flink-pulsar doesn't use it. It reads raw bytes.
For Avro this should not be an issue as the avro schema can be used to do the mapping