Christophe Bornet
03/27/2023, 12:54 PMflink-avro deserializes to SQL Tables using the position in the avro record.
Since with avro we have the schema and the field names, is there a possibility to map by field names instead ?
The current converter leads to issues when the schema evolves.Martijn Visser
03/27/2023, 12:55 PMflink-avro or flink-sql-avro ?Martijn Visser
03/27/2023, 12:55 PMChristophe Bornet
03/27/2023, 12:59 PMorg.apache.flink.formats.avro.AvroToRowDataConverters in flink-avroMartijn Visser
03/27/2023, 1:00 PMChristophe Bornet
03/27/2023, 1:01 PMChristophe Bornet
03/27/2023, 1:03 PMpublic static AvroToRowDataConverter createRowConverter(RowType rowType) {
final AvroToRowDataConverter[] fieldConverters =
rowType.getFields().stream()
.map(RowType.RowField::getType)
.map(AvroToRowDataConverters::createNullableConverter)
.toArray(AvroToRowDataConverter[]::new);
final int arity = rowType.getFieldCount();
return avroObject -> {
IndexedRecord record = (IndexedRecord) avroObject;
GenericRowData row = new GenericRowData(arity);
for (int i = 0; i < arity; ++i) {
// avro always deserialize successfully even though the type isn't matched
// so no need to throw exception about which field can't be deserialized
row.setField(i, fieldConverters[i].convert(record.get(i)));
}
return row;
};
}
It casts to IndexedRecord but I think it's actually a GenericRecord and the mapping could be made by field namesChristophe Bornet
03/27/2023, 1:04 PMpublic JsonToRowDataConverter createRowConverter(RowType rowType) {
final JsonToRowDataConverter[] fieldConverters =
rowType.getFields().stream()
.map(RowType.RowField::getType)
.map(this::createConverter)
.toArray(JsonToRowDataConverter[]::new);
final String[] fieldNames = rowType.getFieldNames().toArray(new String[0]);
return jsonNode -> {
ObjectNode node = (ObjectNode) jsonNode;
int arity = fieldNames.length;
GenericRowData row = new GenericRowData(arity);
for (int i = 0; i < arity; i++) {
String fieldName = fieldNames[i];
JsonNode field = node.get(fieldName);
try {
Object convertedField = convertField(fieldConverters[i], fieldName, field);
row.setField(i, convertedField);
} catch (Throwable t) {
throw new JsonParseException(
String.format("Fail to deserialize at field: %s.", fieldName), t);
}
}
return row;
};
}Christophe Bornet
03/27/2023, 1:05 PMIIRC, schema evolution is only supported with the Confluent Avro implementationCan this be used outside of the Confluent registry ?
Martijn Visser
03/27/2023, 1:05 PMChristophe Bornet
03/27/2023, 1:05 PMMartijn Visser
03/27/2023, 1:06 PMChristophe Bornet
03/27/2023, 1:07 PMChristophe Bornet
03/27/2023, 1:08 PMChristophe Bornet
03/27/2023, 1:09 PMChristophe Bornet
03/27/2023, 1:10 PM