A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

hi there, I’m just looking at adding profiling stages to some of our metadata ingestion and some of the calcualtion are pretty expensive and might not add too much value for us. Is there a way to configure which profiling steps will be applied?

for  reference I’m using the athena recipe triggered by airflow, and some of the profiling on some of our smaller (~1Million rows) tables can take upto 30 minutes

specifically it looks like th its the quartile range measurement which takes ages

Thanks for reporting Chris! We will look into supporting switches to selectively turn off expensive stages