i know the data quality RFC is still in developmen...
# getting-startedb
better-orange-49102
07/15/2021, 6:21 AMi know the data quality RFC is still in development, but just wondering, is the integration of tools like Great Expectations supposed to work with the existing python ingest framework? meaning, the ingest script will run DQ scripts as part of its metadata scraping process. Just want to know if it will change the way we do metadata ingestion.
b
big-carpet-38439
07/15/2021, 12:17 PMThat’s the running idea - yes! However this will be highly configurable, ie you won’t have to profile everything
👍 1
l
loud-island-88694
07/15/2021, 1:46 PM
If the question is about running dq checks, we are still designing that. We may want to run the checks in response to certain events
loud-island-88694
07/15/2021, 1:59 PM
for profiling, we are doing what John mentioned above. if you have ideas for how checks should be run please suggest in #design-data-quality