I have a question: for example, mysql, MCE events are generated by monitoring logs or by rotation？

you generate these events by periodic jobs such as cron or airflow, the job will query the data source and generate the MCE

@better-orange-49102 After installing mysql plug-in, do you collect metadata by monitoring log or directly query? Is MCE also generated by this plug-in?

I'm not sure what logs mySQL generates. the datahub python library queries the target database for the tables inside and generates metadata for them.

@better-orange-49102 Ok, thanks! Can I take it this way? Datahub is the metadata generated by installing a Python plug-in that periodically queries tables in the database to monitor changes. I also want to ask how dataHub is handled if I change the table name.

I'm afraid you may have had the wrong impression of datahub 😅. Datahub is the name of the open source product actually. It consist of 1. UI for users to view the metadata 2. Backend processes to store, process the metadata 3. Default Data stores (MySQL, elastic search) that stores the metadata. 4. A python library that queries other data bases for metadata on the tables stored there. It also sends the generated MCE to the backend via REST API or kafka

@better-orange-49102 thanks，In fact, I did understand, but my leader asked me whether DataHub monitors metadata changes by listening to logs or polling the database. I will continue to learn, thank you again👍

I'm curious, but the database you have in mind, doesn't refer to Kafka, does it? For relational databases, it queries the db, but for Kafka, I'm rather fuzzy on how it generates the mce

Hi @cuddly-family-62352: the current plugin queries the db, but we have plans of supporting monitoring metadata changes by listening to the log (similar to cdc systems). Which db were you planning to connect to?

@mammoth-bear-12532 We are primarily relational databases