I'm trying to generate JSON files containing MCE e...
# ingestion
o
I'm trying to generate JSON files containing MCE events from Java code. I'm modelling the MCE structure using Java classes and using Jackson to actually generate the JSON. It would be nice to be able to re-use any classes that Datahub exposes in order to create MCE objects. Does Datahub provide any Java libraries for this? Thanks.
b
Hihi! Currently we have libraries inside of the repository that act as strongly typed helper POJOs for constructing the MCEs, yes. They are not however currently published on Maven 😞 I'm assuming you are hoping to push metadata from another java service?
o
Hi John, that's right. I'm using a Java program to generate MCE files. Will probably push it through the datahub command line using a recipe file. Is there any precedence for using the Java POJOs to construct MCEs, either within the Datahub codebase or elsewhere?
b
Yes there is. After writing to our primary state store (usually MySQL), we build & emit MetadataAuditEvents in Java by converting the Rest.li (PDL) equivalent into Avro
I can point you towards that code if you'd like?
o
That'd be very useful. Please point me to the relevant code.
b
Hi Pramod - Here's where we convert Rest.li models into their Avro GenericRecord equivalent prior to emitting a change audit event to kafka: https://github.com/linkedin/datahub/blob/master/metadata-dao-impl/kafka-producer/s[…]din/metadata/dao/producer/EntityKafkaMetadataEventProducer.java
thank you 1