A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

What role should I give to ingestion-cron’s service account to ingest from bigquery? I gave ‘BigQuery Metadata Viewer’ but got authentication error :disappointed:

Error:
```
Message: 'Error computing lineage information using GCP logs.'
Arguments: (Forbidden('POST <https://logging.googleapis.com/v2/entries:list?prettyPrint=false>: The caller does not have permission'),)```

Also got this warning.
```/usr/local/lib/python3.8/site-packages/google/cloud/bigquery/client.py:513: UserWarning: Cannot create BigQuery Storage client, the dependency google-cloud-bigquery-storage is not installed.```

are you using the usage and lineage feature as well? then see <https://datahubproject.io/docs/metadata-ingestion/source_docs/bigquery/#capabilities-1>

and if you install `acryl-datahub[bigquery-usage]` the right packages should be installed. Can you verify this again? :slightly_smiling_face:

I’m using the official image though,
<https://github.com/acryldata/datahub-helm/tree/master/charts/datahub/subcharts/datahub-ingestion-cron>

do I have to install the bigquery-usage plugin, and make a custom image?

This is our recipe’s manifest.
```apiVersion: v1
kind: ConfigMap
metadata:
  name: datahub-recipes
  namespace: datahub
data:
  bigquery.yaml: |-
    source:
      type: bigquery
      config:
        project_id: &lt;OUR_PROJECT_ID&gt;

    sink:
      type: "datahub-rest"
      config:
        server: "<http://datahub-datahub-gms:8080>"```

ah okay. sorry I wasn’t aware of this chart. I think it should have installed all dependencies :thinking_face:

but yes this one package is missing. Can you point me to the DataHub code which is using this module?

In datahub github repo’s code, or the helm chart’s value code?

my helm chart value is:
```  datahub-ingestion-cron:
    enabled: true
    image:
      repository: linkedin/datahub-ingestion
      tag: "head"
    crons:
      bigquery:
        schedule: "*/30 * * * *" # Every 30 minutes
        recipe:
          configmapName: datahub-recipes
          fileName: bigquery.yaml
        serviceAccountName: datahub-service-agent```

I mean in datahub repo code. So from the stacktrace there it says the module is missing

No stacktrace logged in my cronjob’s log.
Full log is:
```[2021-11-05 06:00:11,582] INFO     {datahub.cli.ingest_cli:57} - Starting metadata ingestion
/usr/local/lib/python3.8/site-packages/google/cloud/bigquery/client.py:513: UserWarning: Cannot create BigQuery Storage client, the dependency google-cloud-bigquery-storage is not installed.
  warnings.warn(```

hm okay. So not sure where/when this dependency was introduced. Maybe because of profiling/lineage feature. You could try disabling them. Otherwise I guess this needs to be fixed anyway. <@U02AYK6RSPP> you know this source best I guess :slightly_smiling_face:

<@U02JYEBTR2Q>, the bigquery source also computes dataset level lineage for which it requires access to google audit logs. Does your service account have the permissions to access logs? If not, you may need to add logging admin permission.

Thanks! I think we need a documentation that which role needed to ingest from big query. :slightly_smiling_face:

Yes, thanks for calling out <@U02JYEBTR2Q>