A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hello! I'm noticing that when ingesting data from BigQuery, if the active service account doesn't actually have the necessary permissions on the GCP project being ingested, then there will be no error message, just a "Pipeline finished successfully" message but no data ingested. Is that expected behavior?

Relatedly, I think it would be helpful to add the exact Google scopes needed to the BQ ingestion guide: <https://datahubproject.io/docs/metadata-ingestion/source_docs/bigquery>

I can see that the required scopes are specified for `bigquery-usage` but not plain `bigquery`. My team has had success with a svc account that had `BigQuery Metadata Viewer` permissions for the projects in question, but I don't know if that is the only permission that's needed

Thanks for letting us know <@U01UBVC5TM3>  we'll update the docs once we figure out the exact scope

From my experience `BigQuery Metadata Viewer` permissions are enough.
<@U01UBVC5TM3> One thing to mention here is that you need to be aware that you can only fetch native tables. External tables are not visible for ingest :confused: