Hey DataHub team. We're looking to possibly introduce a new optional config to the Looker ingestion plugin to allow us to handle "symbolic links" in BigQuery. For further context, for the purposes of Looker consumption, we recently introduced linked datasets in our BigQuery prod instances, which are basically read-only datasets that serve as a symbolic link to a shared dataset. Thus, right now, the tables in our shared datasets aren't displaying their downstream Looker dashboards properly. We were thinking of introducing an optional config called, say

bigquery_project_map

, which will just have project mappings between the symbolic link and the shared datasets. As an example:

bigquery_project_map:
    looker-1: prd-1
    looker-2: prd-2
    looker-3: prd-3

Is this something that you would be open to us adding? cc: @plain-farmer-27314

Hey @refined-lizard-83096: could you elaborate a bit more what these linked datasets are? I'm wondering if these linkages can be automatically discovered, so it doesn't have to be manually specified.

@mammoth-bear-12532 thanks for the response! By "linked dataset", I just mean this functionality in BigQuery (https://cloud.google.com/bigquery/docs/analytics-hub-introduction#linked_datasets). We have this enabled on a set of tables just so Looker doesn't need to access them directly (for both performance and privacy/security reasons). We keep these BigQuery project mappings internally, but I'm not sure if it's stored anywhere where it can be automatically be discovered. I can try asking around.