i am trying to profile tables using allow_deny_pat...
# ingestion
l
i am trying to profile tables using allow_deny_pattern but somehow it is profiling on the basis of table_pattern
Copy code
source:
    type: mysql
    config:
        host_port: 'X.X.X.X:3306'
        username: x
        password: x
        platform: test-pattern
        include_tables: true
        include_views: true
        schema_pattern:
            ignoreCase: true
            allow:
                - dp_datahub
        table_pattern:
            ignoreCase: true
            allow:
                - 'dp_datahub.stocks_bse*'
        profiling:
            enabled: true
            bigquery_temp_table_schema: abc.datahub
            turn_off_expensive_profiling_metrics: false
            query_combiner_enabled: false
            max_number_of_fields_to_profile: 2
            profile_table_level_only: false
            include_field_null_count: true
            include_field_min_value: true
            include_field_max_value: true
            include_field_mean_value: true
            include_field_median_value: true
            include_field_stddev_value: false
            include_field_quantiles: false
            include_field_distinct_value_frequencies: false
            include_field_histogram: false
            include_field_sample_values: false
            allow_deny_patterns:
              allow:
                  - dp_datahub.stocks_bse_ci_test.date
sink:
    type: datahub-rest
    config:
        server: '<http://X.X.X.X:8080>'
s
Please use
profile_pattern.allow
and
profile_pattern.deny
I will update the docs
l
I tried profile_pattern its working to filter tables but how do i apply pattern on columns to profile??