Hi everyone, wondering if anyone’s had issues ingesting lineage from BigQuery due to 503s when querying GCP logs. The logs are:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/logging/__init__.py", line 1083, in emit
    msg = self.format(record)
  File "/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/logging/__init__.py", line 927, in format
    return fmt.format(record)
  File "/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/logging/__init__.py", line 663, in format
    record.message = record.getMessage()
  File "/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/logging/__init__.py", line 367, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
  File "/Users/hyunmin/datahub-recipes/env/bin/datahub", line 8, in <module>
    sys.exit(main())
  File "/Users/hyunmin/datahub-recipes/env/lib/python3.9/site-packages/datahub/entrypoints.py", line 93, in main
    sys.exit(datahub(standalone_mode=False, **kwargs))
  File "/Users/hyunmin/datahub-recipes/env/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/hyunmin/datahub-recipes/env/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/hyunmin/datahub-recipes/env/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/hyunmin/datahub-recipes/env/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/hyunmin/datahub-recipes/env/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/hyunmin/datahub-recipes/env/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/hyunmin/datahub-recipes/env/lib/python3.9/site-packages/datahub/cli/ingest_cli.py", line 58, in run
    pipeline.run()
  File "/Users/hyunmin/datahub-recipes/env/lib/python3.9/site-packages/datahub/ingestion/run/pipeline.py", line 141, in run
    for wu in self.source.get_workunits():
  File "/Users/hyunmin/datahub-recipes/env/lib/python3.9/site-packages/datahub/ingestion/source/sql/bigquery.py", line 207, in get_workunits
    self._compute_big_query_lineage()
  File "/Users/hyunmin/datahub-recipes/env/lib/python3.9/site-packages/datahub/ingestion/source/sql/bigquery.py", line 121, in _compute_big_query_lineage
    logger.error(
Message: 'Error computing lineage information using GCP logs.'
Arguments: (ServiceUnavailable('POST <https://logging.googleapis.com/v2/entries:list?prettyPrint=false>: The service is currently unavailable.'),)

I’ve tried to relax start_time, end_time, and max_query_duration constraints (to almost 10 second intervals) but unfortunately still haven’t seen good results. It seems like the project DataHub is querying against is too big in terms of log size and GCP’s API seems to timeout when returning the logs internally (this was confirmed by GCP’s support team). Would there be any options I could tweak or anything I’m missing? FYI I’ve tried to call the API manually via curl with smaller page sizes of about 10 and seen better results, but it seems like DataHub’s bigquery ingestion module uses a fixed page size of 1000. Any ideas would be much appreciated!

Hey @wooden-gpu-7761! Thank you so much for the report. This PR makes the page size configurable: https://github.com/linkedin/datahub/pull/3556

Wow thanks @big-carpet-38439, what a lightning fast response! I left a comment on your PR, basically about applying the same changes to

bigquery.py

— let me know how it sounds to you! Thanks again!

Of course!