Hi! I'm running into an unsafe type-cast error in ...
# ingestion
o
Hi! I'm running into an unsafe type-cast error in trying to convert a in-memory MetadataChangeEvent to a GenericRecord to be sent to Kafka using EventUtils (pegasusToAvroMCE). The MetadataChangeEvent has DataMap format for some of the fields (ex: URN fields). In DataTranslator.java from the REST.li project this is causing the issue:
Copy code
line 531-533:        
case STRING:
          result = new Utf8((String) value);
          break;
Value in this case is a DataMap representing a DatasetUrn. The Avro schema defines that as as a String type expecting something like "urnlidataset:..." but since it is in the form "{ platform: {...}, origin: ... , name: ...}" this case results in a ClassCastException. Is there a different way I can generate the GenericRecord with the format I have? NOTE: This is NOT master, this is a separate development branch that I'm working on in a forked repo. Not a bug report, looking for advice 🙂
m
@microscopic-receptionist-23548 has worked on the java ingestion pathway
o
Took a look at his code for the KafkaMetadataEventProducer. I think my issue lies in not creating the Snapshot correctly. Looks like the ModelUtils method flattens URNs down as opposed to the static method provided from Snapshot.java. Will update
Okay it doesn't flatten nested fields, but I can flatten them in my custom deserializer. Should work, thanks for pointing me towards that code! Hadn't looked through it
m
Yeah PDL supports some customer marshallers / serializers. URNs all have custom ones to flatten them to strings.
👍 1