Hi All, I am not using docker (docker commands) to...
# ingestions
silly-cat-58453
08/01/2020, 1:56 PMHi All, I am not using docker (docker commands) to ingest data, so i have produced the data to "MetadataChangeEvent" topic, which is persisted to MYSQL through GMS, so Can anyone suggest me how to use the "MetadataAuditEvent" ?
b
bumpy-keyboard-50565
08/01/2020, 1:59 PMIs this about https://github.com/linkedin/datahub/issues/1766?
s
silly-cat-58453
08/01/2020, 1:59 PMyes sir
b
bumpy-keyboard-50565
08/01/2020, 2:00 PMGot it. So what do you mean by "using MAE"? Are you trying to update the search index?
bumpy-keyboard-50565
08/01/2020, 2:00 PMOr just to verify that MAE was emitted correctly?
s
silly-cat-58453
08/01/2020, 2:00 PMyes actually, my data is persisted to mysql but is not searchable in datahub UI
silly-cat-58453
08/01/2020, 2:01 PMi want to make my data searchable
b
bumpy-keyboard-50565
08/01/2020, 2:01 PMGot it. I assume you're using the quickstart docker images?
s
silly-cat-58453
08/01/2020, 2:02 PMyes, but i am not using docker commands to ingest metadata
b
bumpy-keyboard-50565
08/01/2020, 2:03 PMSure. Could you run
Copy code
docker logs datahub-mae-consumers
silly-cat-58453
08/01/2020, 2:05 PMi've installed kafka and mysql locally
silly-cat-58453
08/01/2020, 2:06 PMso, the docker command won't work
b
bumpy-keyboard-50565
08/01/2020, 2:06 PMsure. but you need to have the
datahub-mae-consumerbumpy-keyboard-50565
08/01/2020, 2:07 PMSee https://github.com/linkedin/datahub/blob/master/docs/architecture/architecture.md for an overview of the architecture
s
silly-cat-58453
08/01/2020, 2:07 PMokay, i will check and get back to you.
silly-cat-58453
08/01/2020, 2:07 PMSure, thanks
b
bumpy-keyboard-50565
08/01/2020, 2:08 PMYou can start only the mae-consumer-job using this docker file: https://github.com/linkedin/datahub/tree/master/docker/mae-consumer
s
silly-cat-58453
08/01/2020, 2:09 PMis "mae-consumer-job" a topic in kafka?
silly-cat-58453
08/01/2020, 2:10 PMtopic with the name "MetadataAuditEvent"
b
bumpy-keyboard-50565
08/01/2020, 2:10 PMIt's a kafka stream job. Just like the "mce-consumer-job" that consumes MetadataChangeEvent and injects that into the MySQL DB
bumpy-keyboard-50565
08/01/2020, 2:11 PMPlease take a look at the architecture diagram I posted above to get a better understanding how things are hooked together
s
silly-cat-58453
08/01/2020, 2:12 PMyes, and i already have created a topic "MetadataAuditEvent"
b
bumpy-keyboard-50565
08/01/2020, 2:16 PMYou still need the process running in order to consume the messages from the topic
m
microscopic-receptionist-23548
08/01/2020, 8:00 PMMaybe also worth mentioning: you also need to have elasticsearch running as well