I’m running into this weird issue where, when I ingest table metadata from BigQuery with profiling enabled, only one or two of the numeric columns actually get

Min

,

Max

,

Mean

,

Median

and

Standard Deviation

, while all the others just show “unknown”.

Null

and

Distinct

stats and

sample values

are all working fine. Anyone know what might be causing this? The columns with missing stats have a high % of null (99.9x%), but there are still some valid values in there.

Hi @agreeable-hamburger-38305, are you supplying the profiling.limit parameter ?

I’m not. The only other config I have is

profile_pattern

, which shouldn’t be affecting this?

It should not be affecting this.

I have the same problem

Hi @agreeable-hamburger-38305, it is an artifact of how the default great expectations profiler that we use behaves underneath currently . If you have more than 50% nulls, stats computation will be skipped ( https://github.com/great-expectations/great_expectations/blob/develop/great_expectations/profile/basic_dataset_profiler.py#L157). We are moving towards a customizable profiler instead of the default one, and we will explore the possibility of configuring this behavior.

@helpful-optician-78938 Ah that makes sense. Thanks for the reply! Look forward to the updates on this