Hello everyone, I'm having problems with BigQuery ...
# troubleshootg
gentle-camera-33498
12/12/2022, 6:41 PMHello everyone, I'm having problems with BigQuery lineage metadata ingestion. Can someone help me?
In the company where I work, we have patterns for tables and view names: tables are named following the 'snake case' pattern, and views are named following the 'upper camel case' pattern. The problem is that the lineage builder returns the tables and views name in lowercase. With that, we have a consistency problem in our lineage visualization (as we have tables with the same name as views but in different case patterns)
The possible location of the problem: https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/bigquery_v2/lineage.py#L437
The 'get_table' method correctly returns the tables and views involved, but all with the names converted to lower case.
gentle-camera-33498
12/12/2022, 6:53 PMExample: In the same BigQuery dataset, we have a table called 'campaign' and a view called 'Campaign'. There is another table in this same dataset which is a materialization of a query involving the 'Campaign' view, but in its lineage, the 'campaign' table appears. This does not match the SQLX definitions defined in the DataForm(Google service that works like DBT) which is responsible for managing the materialization of this table.
gentle-camera-33498
12/12/2022, 8:21 PMThis PR may solve the problem: https://github.com/datahub-project/datahub/pull/6739
gentle-camera-33498
12/12/2022, 8:23 PMThe problem above is also the same in the lineage builder in Metabase integration.
a
astonishing-answer-96712
12/13/2022, 6:29 PMHi @gentle-camera-33498, thank you for submitting! We’ll get it looked at 🙂
2 Views