Hi team 🙂 is there a way, or is there anyone that tried to detect metadata changes in data source "in real-time", and ingest them automatically (if possible, only the changed part) to datahub? I'm looking at the mce builder and emitters, but i think i'm completely lost. 😢

Morning @nice-planet-17111, I am not aware if someone is auto detecting the changes in the source and then ingesting in real time but I'll let the community also answer this. @helpful-optician-78938, is it possible to ingest only the changed part, perhaps leveraging the stateful ingestion?

Currently there are 2 types of ingestion sources: 1. Batch: Pull everything (or many things) by polling periodically 2. Real time: Push metadata in real time Today, the only real time connectors are integrations with Airflow, wherein on each Airflow task run some metadata is emitted. Batch connectors include most other things, like MySQL, Snowflake, Kafka, etc. Ideally, more things would be real time. The limitation we face is really with the 3rd party systems we integrate with, which don't typically offer first-class event APIs

@big-carpet-38439 Hello, thank you for the reply 🙂 About

integrations with Airflow

- i didn't quite understand. How will it automatically detect changes in metadata (every time change occurs)? Is there any sample script that i can refer to?

@nice-planet-17111 theoritically if you can poll for metadata changes on source, you can implement a near realtime mce emitter to achieve this.. this will depend a lot on what your source platform is and how frequently do you want to poll for changes (how expensive is it to detect those changes based on the platform).