Hi Team, At multiple stages I am getting below error when ingesting the metadata for bigquery from CLI:

Does your service account has bigquery.tables.list, bigquery.routines.get, bigquery.routines.list permission, bigquery.tables.getData permission? The error was: 'type'
[2023-01-23, 21:55:49 UTC] {process_utils.py:168} INFO - Traceback (most recent call last):
[2023-01-23, 21:55:49 UTC] {process_utils.py:168} INFO -   File "/tmp/venv45wzxte5/lib/python3.8/site-packages/datahub/ingestion/source/bigquery_v2/bigquery.py", line 587, in _process_project
[2023-01-23, 21:55:49 UTC] {process_utils.py:168} INFO -     yield from self._process_schema(conn, project_id, bigquery_dataset)
[2023-01-23, 21:55:49 UTC] {process_utils.py:168} INFO -   File "/tmp/venv45wzxte5/lib/python3.8/site-packages/datahub/ingestion/source/bigquery_v2/bigquery.py", line 702, in _process_schema
[2023-01-23, 21:55:49 UTC] {process_utils.py:168} INFO -     yield from self._process_table(conn, table, project_id, dataset_name)
[2023-01-23, 21:55:49 UTC] {process_utils.py:168} INFO -   File "/tmp/venv45wzxte5/lib/python3.8/site-packages/datahub/ingestion/source/bigquery_v2/bigquery.py", line 735, in _process_table
[2023-01-23, 21:55:49 UTC] {process_utils.py:168} INFO -     yield from self.gen_table_dataset_workunits(table, project_id, schema_name)
[2023-01-23, 21:55:49 UTC] {process_utils.py:168} INFO -   File "/tmp/venv45wzxte5/lib/python3.8/site-packages/datahub/ingestion/source/bigquery_v2/bigquery.py", line 774, in gen_table_dataset_workunits
[2023-01-23, 21:55:49 UTC] {process_utils.py:168} INFO -     custom_properties["time_partitioning"] = str(table.time_partitioning)
[2023-01-23, 21:55:49 UTC] {process_utils.py:168} INFO -   File "/tmp/venv45wzxte5/lib/python3.8/site-packages/google/cloud/bigquery/table.py", line 2689, in __repr__
[2023-01-23, 21:55:49 UTC] {process_utils.py:168} INFO -     key_vals = ["{}={}".format(key, val) for key, val in self._key()]
[2023-01-23, 21:55:49 UTC] {process_utils.py:168} INFO -   File "/tmp/venv45wzxte5/lib/python3.8/site-packages/google/cloud/bigquery/table.py", line 2665, in _key
[2023-01-23, 21:55:49 UTC] {process_utils.py:168} INFO -     properties["type_"] = repr(properties.pop("type"))
[2023-01-23, 21:55:49 UTC] {process_utils.py:168} INFO - KeyError: 'type'

The service account has access based on https://datahubproject.io/docs/quick-ingestion-guides/bigquery/setup/ and I am at v0.9.6.1

Hi @quaint-barista-82836, could you double-check that your service account also has the list of roles defined in that article?

Hi @astonishing-answer-96712, I gave below permissions, as basic role was not working added biqquery admin permission. Still after that its not able to resolve the issue. Any thoughts on it ?

Hmm interesting- we’re going to be having a workshop on this in a few weeks, for now lets see if @dazzling-judge-80093 might be able to offer some insight

Hi @dazzling-judge-80093, any suggestions for the fix ?

You’ll need to bring in each project separately I believe since the ingestion targets by project See Tamas’s comment below

@quaint-barista-82836 if the service account can list all the projects then it can ingest all your project with one ingestion recipe.

@dazzling-judge-80093, I can fetch data from all projects however, the keyError type issue still persist, with all permissions assigned on the table: Unable to get tables for dataset PRICING_TEMP in project project_name, skipping. Does your service account has bigquery.tables.list, bigquery.routines.get, bigquery.routines.list permission, bigquery.tables.getData permission? The error was: ‘type' Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/datahub/ingestion/source/bigquery_v2/bigquery.py", line 582, in _process_project yield from self._process_schema(conn, project_id, bigquery_dataset) File "/usr/local/lib/python3.7/site-packages/datahub/ingestion/source/bigquery_v2/bigquery.py", line 697, in _process_schema yield from self._process_table(conn, table, project_id, dataset_name) File "/usr/local/lib/python3.7/site-packages/datahub/ingestion/source/bigquery_v2/bigquery.py", line 730, in _process_table yield from self.gen_table_dataset_workunits(table, project_id, schema_name) File "/usr/local/lib/python3.7/site-packages/datahub/ingestion/source/bigquery_v2/bigquery.py", line 769, in gen_table_dataset_workunits custom_properties["time_partitioning"] = str(table.time_partitioning) File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery/table.py", line 2659, in repr key_vals = ["{}={}".format(key, val) for key, val in self._key()] File "/usr/local/lib/python3.7/site-packages/google/cloud/bigquery/table.py", line 2635, in _key properties["type_"] = repr(properties.pop("type")) KeyError: 'type'

@quaint-barista-82836 thanks it really helps, I will create fix for this soon

Hi @dazzling-judge-80093, can you help with tentative fix date for the issue, as because of this its not reading metadata for multiple tables ?

@quaint-barista-82836 please, can you try with the latest python version 0.10.0.3, that should contain a fix for the above issue

@dazzling-judge-80093 Are you referring to https://datahubproject.io/docs/generated/ingestion/sources/bigquery/#install-the-plugin with pip install 'acryl-datahub[bigquery]==0.10.0.3' and not the datahub upgrade ?

yes, currently it gets tables in batch and not getting it in parallel/distributed way

I ran with the new lib version and seems like that issue is fixed however getting a new issue with "poorly formatted table name, contains {invalid_chars_in_table_name}" ValueError: Cannot handle project_a_name.[project_b_name:market_basket_daily.pos_rolling_sales_daily$20180413] - poorly formatted table name, contains ['$'] To resolve it gave : "match_fully_qualified_names" : True, dataset_pattern": { "deny" :["project_a_name.[project_b_name:market_basket_daily.pos_rolling_sales_daily$20180413]"]}, in recipe, however getting below error: File "/usr/local/lib/python3.7/site-packages/datahub/ingestion/run/pipeline.py", line 341, in run self.preview_workunits if self.preview_mode else None, File "/usr/local/lib/python3.7/site-packages/datahub/utilities/source_helpers.py", line 85, in auto_stale_entity_removal for wu in stream: File "/usr/local/lib/python3.7/site-packages/datahub/utilities/source_helpers.py", line 109, in auto_workunit_reporter for wu in stream: File "/usr/local/lib/python3.7/site-packages/datahub/utilities/source_helpers.py", line 36, in auto_status_aspect for wu in stream: File "/usr/local/lib/python3.7/site-packages/datahub/ingestion/source/bigquery_v2/bigquery.py", line 514, in get_workunits_internal yield from self._process_project(conn, project_id) File "/usr/local/lib/python3.7/site-packages/datahub/ingestion/source/bigquery_v2/bigquery.py", line 612, in _process_project self.config.match_fully_qualified_names, File "/usr/local/lib/python3.7/site-packages/datahub/configuration/pattern_utils.py", line 11, in is_schema_allowed return schema_pattern.allowed(f"{db_name}.{schema_name}") File "/usr/local/lib/python3.7/site-packages/datahub/configuration/common.py", line 248, in allowed if re.match(deny_pattern, string, self.regex_flags): File "/usr/local/lib/python3.7/re.py", line 175, in match return _compile(pattern, flags).match(string) File "/usr/local/lib/python3.7/re.py", line 288, in _compile p = sre_compile.compile(pattern, flags) File "/usr/local/lib/python3.7/sre_compile.py", line 764, in compile p = sre_parse.parse(p, flags) File "/usr/local/lib/python3.7/sre_parse.py", line 924, in parse p = _parse_sub(source, pattern, flags & SRE_FLAG_VERBOSE, 0) File "/usr/local/lib/python3.7/sre_parse.py", line 420, in _parse_sub not nested and not items)) File "/usr/local/lib/python3.7/sre_parse.py", line 574, in _parse raise source.error(msg, len(this) + 1 + len(that)) re.error: bad character range r-p at position 25 [while running 'push to datahub-ptransform-32']