Hi all, I'm giving a try to the tool again and it looks amazing now, congrats to the team. I was trying the new profiling feature for BigQuery, but I detected a couple of errors: 1. I'm getting this error for many of the table profile runs

Profiling exception No BigQuery dataset specified. Use bigquery_temp_table batch_kwarg or a specify a default dataset in engine url

. For some others it works. I'm of course passing the

bigquery_temp_table_schema

setting. I've tried with

max_workers: 1

but it still fails in some cases 2. The other one has been reported already. Looks like it is failing for columns with a REPEATED (ARRAY) type 3. A third one looks like it comes from GE itself. Looks like GE is not working with the GEOGRAPHY type

For the 1st one you should set the bigquery_temp_table_schema property -> https://datahubproject.io/docs/metadata-ingestion/source_docs/bigquery/#profiling

Hi Tamas, thanks for your answer. Regarding point 1. I've already set up the

bigquery_temp_table_schema

property. Looks, this is a extract of the log without filling the property:

{'records_written': 75,
 'warnings': [],
 'failures': [],
 'downstream_start_time': datetime.datetime(2022, 2, 16, 10, 17, 43, 834333),
 'downstream_end_time': datetime.datetime(2022, 2, 16, 10, 17, 56, 14536),
 'downstream_total_latency_in_seconds': 12.180203}

'bigquery-urn:li:dataset:(urn:li:dataPlatform:bigquery,my-project.my_dataset.some_table,DEV)-upstreamLineage',
                  'container-urn:li:container:ffcf59fce96af99d615e757879630918-to-urn:li:dataset:(urn:li:dataPlatform:bigquery,my-project.my_dataset.some_table_1,DEV)',
                  'my-project.my_dataset.some_table_1',
                  'container-urn:li:container:ffcf59fce96af99d615e757879630918-to-urn:li:dataset:(urn:li:dataPlatform:bigquery,my-project.my_dataset.some_table_2,DEV)',
                  'my-project.my_dataset.some_table_2',
                  'container-urn:li:container:ffcf59fce96af99d615e757879630918-to-urn:li:dataset:(urn:li:dataPlatform:bigquery,my-project.my_dataset.some_table_3,DEV)',
                  'my-project.my_dataset.some_table_3',
                  'container-urn:li:container:ffcf59fce96af99d615e757879630918-to-urn:li:dataset:(urn:li:dataPlatform:bigquery,my-project.my_dataset.some_table_4,DEV)',
                  'my-project.my_dataset.some_table_4'],
 'warnings': {},
 'failures': {'my-project.my_dataset.some_table_5': ['Profiling exception No BigQuery dataset specified. Use bigquery_temp_table batch_kwarg or a '
                                                     'specify a default dataset in engine url'],
              'my-project.my_dataset.some_table_6': ['Profiling exception No BigQuery dataset specified. Use bigquery_temp_table batch_kwarg or a '
                                                        'specify a default dataset in engine url'],
              'my-project.my_dataset.some_table_7': ['Profiling exception No BigQuery dataset specified. Use bigquery_temp_table batch_kwarg or a '
                                                   'specify a default dataset in engine url'],

This an extract of the log with the property set up:

{'records_written': 89,
 'warnings': [],
 'failures': [],
 'downstream_start_time': datetime.datetime(2022, 2, 16, 10, 21, 1, 731805),
 'downstream_end_time': datetime.datetime(2022, 2, 16, 10, 23, 0, 463661),
 'downstream_total_latency_in_seconds': 118.731856}

'profile-my-project.my_dataset.some_table_1',
                  'profile-my-project.my_dataset.some_table_2',
                  'profile-my-project.my_dataset.some_table_3',
                  'profile-my-project.my_dataset.some_table_4',
                  'profile-my-project.my_dataset.some_table_5'],
 'warnings': {},
 'failures': {'my-project.my_dataset.some_table_6': ['Profiling exception No BigQuery dataset specified. Use bigquery_temp_table '
                                                                'batch_kwarg or a specify a default dataset in engine url'],
              'my-project.my_dataset.some_table_7': ['Profiling exception No BigQuery dataset specified. Use bigquery_temp_table batch_kwarg or a '
                                                 'specify a default dataset in engine url'],
              'my-project.my_dataset.some_table_8': ['Profiling exception No BigQuery dataset specified. Use bigquery_temp_table '
                                                                  'batch_kwarg or a specify a default dataset in engine url'],

Can you share you config?

Config

source:
  type: bigquery
  config:
    project_id: my-project # optional - can autodetect from environment
    env: DEV
    use_exported_bigquery_audit_metadata: True
    bigquery_audit_metadata_datasets:
      - bigquery_audit_logs
    start_time: 2022-02-14T00:00:00Z
    schema_pattern:
      allow:
        - dlv.*
    profiling:
      enabled: true
      bigquery_temp_table_schema: <http://my-project.ge|my-project.ge>
      limit: 10  # Max number of documents to profile
      max_workers: 10
      max_number_of_fields_to_profile: 3

Can you also run the profiling in debug and share the outputs?

datahub --debug ingest

…

I think I've found where the problem is

the profiling should work for non-partitioned tables but the partitioned empty tables might cause issues. Let me dig a bit deeper to understand.

Hey Tamas, it worked smoothly when removing the limit. Thanks a lot for your support!

awesome, the limit issue will be fixed soon