https://flink.apache.org/ logo
Join Slack
Powered by
Hi All, I’m trying with `OutputTag[AvroSpecificrec...
# troubleshooting
s
Hi All, I’m trying with 
OutputTag[AvroSpecificrecord]
, AvroSpecificRecord is deeply nested object with multiple other record type: I’ve tried passing 
TypeInformation
like this:
Copy code
1. new AvroTypeInfo[AvroSpecificRecord](classOf[AvroSpecificRecord])
2. AvroSchemaConverter.convertToTypeInfo(AvroSpecificRecord.getClassSchema.toString(true))
e.g: new OutputTag("side-output-1", AvroSchemaConverter.convertToTypeInfo(AvroSpecificRecord.getClassSchema.toString(true)))
But Flink is not using this 
TypeInformation
and using reflection based type detection.
Copy code
INFO  org.apache.flink.api.java.typeutils.TypeExtractor            [] - Field AvroSpecificRecord#field1 will be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance and schema evolution
Please help
s
s
@Sameer Chandra So If I pass
Copy code
TypeInformation.of(classOf[AvroSpecificRecord])
type information like this, it will be automatically understood by Flink and not treated as Generic?
Looks like the issue is with OUT type:
Copy code
KafkaSink[AvroSpecificRecord]
It is treating OUT type as GenericType
to be specific, the inner fields of Avro Schema are treated as GenericType
Inner field is of type 
List<String>
in the Avro Schema, that means Generic type. But I want to avoid issue by using Avro Serializer
Looks like it is not picking 
AvroSerializer
based on the object type used in the OUT type
s
I am not familiar with Scala but you need to provide a type of SpecificRecordBase
If you specify the Flink type as 
SpecificRecord
and not 
SpecificRecordBase
, Flink will not see this as an Avro type. Instead, it will use Kryo to de/serialize any objects which may be considerably slower.
s
AvroSpecificrecord
extends 
SpecificRecordBase
since it is auto generated Avro class from Maven Plugin
The Avro record inner field should not be Generic, if Generic it will be treated as GenericType
a
Hi Sunil, I am facing the same problem? Were you able to resolve it ?
3 Views
Open in Slack
PreviousNext
Powered by