Is it possible to manually turn off some of the co...
# ingestion
e
Is it possible to manually turn off some of the columns stats (i.e. min, max, median, null count etc) from Profiling tab to lessen the stress load for ingestion runs?
s
yes. You can see the config has options. e.g. for bigquery docs https://datahubproject.io/docs/generated/ingestion/sources/bigquery/ if you search for
profiling.include_field_mean_value
that is control for mean value. Similar docs exist for other sources
👍 1
d
yes, you can control it:
Copy code
profiling:
      enabled: true
      #max_workers: 5
      #turn_off_expensive_profiling_metrics: true
      #limit: 10
      query_combiner_enabled: false
      max_number_of_fields_to_profile: 8
      #include_field_stddev_value: false
      #profile_table_level_only: true
      include_field_null_count: true
      include_field_min_value: true
      include_field_max_value: true
      include_field_mean_value: true
      include_field_median_value: true
      include_field_stddev_value: true
      include_field_quantiles: false
      include_field_distinct_value_frequencies: false
      include_field_histogram: false
      include_field_sample_values: false
👍 1
e
@square-activity-64562 @dazzling-judge-80093 great, thank you!
b
hi, how to add these profilings to recipe yaml file? I got some error like this:
Is there any examples?
b
ok, thanks