Hi team! I’ve recently been meddling with the BigQuery ingestion module — I’m mostly interested in being able to parse lineage info from GCP audit logs and tables. In most cases I’m really happy with DataHub overall, but I have a couple of issues regarding the current state of lineage parsing. 1. Parsing lineage from a huge GCP project fails due to 503 errors on https://logging.googleapis.com/v2/entries:list, due to an internal timeout issue according to Google Support. It seems like fetching logs and aggregating them takes too long internally within GCP. I’ve tried to circumvent these issues by limiting page size, but with no success. -> What I think could be a good workaround would be to ingest audit logs from exported audit logs, which would in many cases would be stored in the form of BigQuery tables. 2. It is required that audit logs are in the same project as the dataset/tables to be ingested. I think separating dataset projects and query execution projects is a pretty common usecase, and maybe adding the option to define projects from which audit logs are to be extracted from would be pretty useful. Let me know what you think! I would be happy to contribute if possible 🙏 . Thanks!

Hi @wooden-gpu-7761, these are great questions. For 1, @miniature-tiger-96062 and I are currently exploring how to speed up getting the audit log entries (list_entries API). If you have any solution that already works, it would be great if you could share it with us. Regarding (2), bigquery-usage source already supports this. Any contributions here are very much welcome! Please feel free to reach out if you would like to discuss this further.

Hi @helpful-optician-78938 thanks for the info. Regarding 1, I’ve had some success in using BigQuery log sinks (https://cloud.google.com/bigquery/docs/reference/auditlogs#defining_a_bigquery_log_sink_using_gcloud). A project’s audit logs would be stored in a user-specified dataset under the table name

cloudaudit_googleapis_com_data_access

which has largely the same schema as

AuditLogEntry

as used in

bigquery

and

bigquery-usage

modules. On 2, although

bigquery-usage

supports ingesting from multiple projects, it doesn’t seem to be support ingesting lineage metadata (please let me know if otherwise!). That said, the

bigquery

ingestion module sort of requires that the audit logs are in the same project as their referenced tables. I do have a working solution that does resolve issues in both 1 and 2 and I’d be happy to shoot a PR if you guys would like to check it out.