Greetings Team! Today I come with questions about Stats. Not Usage stats, but Data profiling. We are trying to implement DataHub in our company, and we find that the current profiling is very slow for our purposes (this is properly warned in the documentation 🙂). And we are wondering a couple of things: • What metrics are included in the turn_off_expensive_profiling_metrics parameter? • Can we find an extensive list of parameters for the profiling recipe? I think its absent from the docu. We found some Youtube guides, but we are not sure if the parameters covered there are all the available ones. • Can columns included in the profiling process be selected by name? I know there is an option to limit the number of columns included, but I suppose this works by taking the first n columns found in the table. • Does DataHub support external profiling? We have implemented different quality assertions through Great Expectations or dbt test, but I couldn't find references about ingesting other tool's profiling info to be shown in Stats, we assume this means that right now this is not an option. In our setup, profiling the full tables each time we ingest is quite expensive. We thought that a plausible solution would be to profile only the increments of the tables in each iteration of our pipelines, but at the same time keep this partial profiles under the full table's entity in DataHub. In this way, the dataset list in DataHub would only list the tables once, but within each one, we would have the list of incremental profilings (in a similar way to that of Validations, in which you have a timelined list). We think this is not achievable right now, is it? Is there any option that we achieve a reduced or preliminar version of this in the current DataHub? Thanks for your time and for your wonderful tool!! Dani

Hey Daniel, Docs for profiling specific config is documented in each connector's config list. e.g. snippet attached from snowflake config page here. This should give you all the answers including incremental profiling options. • Can columns included in the profiling process be selected by name? I know there is an option to limit the number of columns included, but I suppose this works by taking the first n columns found in the table. Not yet. There is an old open PR for this, but we've not been able to refresh and merge this in yet. It will probably get prioritized in this month.

• Does DataHub support external profiling? We have implemented different quality assertions through Great Expectations or dbt test, but I couldn't find references about ingesting other tool's profiling info to be shown in Stats, we assume this means that right now this is not an option.

Yes it does! We also support assertions directly with great expectations and dbt test. Check out those links.