@here I have a question regarding pull based meta data ingestion. Do we pull the entire schema of the data source? If so, how is it applied to the DataHub store, is the old schema completely replaced or does it perform upsert (update or insert) operations?

We maintain historical versions in DataHub. So a new version is created

@rapid-piano-43271 for things that don’t change it is a no op. So essentially an upsert.

@mammoth-bear-12532 Thanks for the quick reply. Do we pull the entire schema of the data source?

Currently we support some filtering capabilities at schema and table level, but those are typically applied post extraction as each sql database has their own quirks for regexes etc.

@mammoth-bear-12532 Yes, I am. In one of the articles you mentioned that it would put lot of the load the DB and the DBAs might stop the crawler.

Which db are you connecting to in this case?

mostly relational databases

@rapid-piano-43271: yes, there are a few approaches we are taking to reduce the load on upstream dbs: Stateful ingestion + Using the audit logs of these dbs will allow us to only inspect tables that have changed since the last time we ingested metadata.