A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

image.png

Hi all, when I tried to ingest bigquery metadata, how do I put table info of bigquery(ex, Last modified) into “Properties” of the dataset? I would like to use this information to tag datasets which are ‘out-of-date’.

A custom transformer that queries Bigquery and populate the properties tab for each dataset would work.
Or you could try submitting a PR that modifies the existing ingestion behavior for Bigquery

last modified is automatically ingested when you use the <https://datahubproject.io/docs/generated/ingestion/sources/bigquery#module-bigquery-usage|bigquery-usage> connector, and have the default setting for `include_operational_stats` which is True

<@UV0M2EB8Q> I tried to use `bigquery-usage` to ingest data. but I still can not have any properties in the properties tab… Do you know the reason? Thanks.
```source:
    type: bigquery-usage
    config:
        projects:
            - xxx
        credential:
            project_id: xxx
            private_key_id: '${BQ_KEY_ID}'
            private_key: '${BQ_KEY}'
            client_email: '${BQ_EMAIL}'
            client_id: '${BQ_CLIENT_ID}'
        include_operational_stats: true```