https://datahubproject.io logo
Join Slack
    n

    nutritious-bird-77396

    1 year ago
    Copy to Clipboard
    Hi LinkedIn Team….Can you guys help us understand the reasoning behind renaming schema namespaces from 
    com.linkedin.mxe
    to 
    com.linkedin.pegasus2avro.mxe
    ? For ex: https://github.com/linkedin/datahub/blob/54fef777ac7079e6a44aad40af55e868a5844f57/metadata-events/mxe-utils-avro-1.7/src/main/java/com/linkedin/metadata/EventUtils.java#L84
    a

    ambitious-battery-33996

    1 year ago
    Copy to Clipboard
    Kafka events are in avro schemas. But, rest of the processing in datahub gms happens in PDL schemas. Hence, we need to convert 
    com.linkedin.pegasus2avro.mxe
    to 
    com.linkedin.mxe
    m

    microscopic-receptionist-23548

    1 year ago
    Copy to Clipboard
    So we have a few different schema languages. Our models are written in Pegasus (PDL). But then we auto generate Avro from the PDL (hence 
    pegasus2avro
    in the name). If you want to use Kafka, you should use Avro. Otherwise, Pegasus is often nicer to deal with.
    Copy to Clipboard
    Basically, what @ambitious-battery-33996 said 🙂
    n

    nutritious-bird-77396

    1 year ago
    Copy to Clipboard
    Thanks!
    b

    bumpy-keyboard-50565

    1 year ago
    Copy to Clipboard
    More importantly is to avoid class clashes. There are places where both Pegasus & Avro classes are needed. Without renaming it simply won't work.
    👍 1
View count: 1
Join thread in Slack