Hey all, after updating our ingestion plugins to 0...
# ingestion
p
Hey all, after updating our ingestion plugins to 0.8.26.3, I'm seeing the following error:
Copy code
[2022-02-18, 14:00:15 UTC] {pod_launcher.py:149} INFO -   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
[2022-02-18, 14:00:15 UTC] {pod_launcher.py:149} INFO -     return self.main(*args, **kwargs)
[2022-02-18, 14:00:15 UTC] {pod_launcher.py:149} INFO -   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
[2022-02-18, 14:00:15 UTC] {pod_launcher.py:149} INFO -     rv = self.invoke(ctx)
[2022-02-18, 14:00:15 UTC] {pod_launcher.py:149} INFO -   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
[2022-02-18, 14:00:15 UTC] {pod_launcher.py:149} INFO -     return ctx.invoke(self.callback, **ctx.params)
[2022-02-18, 14:00:15 UTC] {pod_launcher.py:149} INFO -   File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
[2022-02-18, 14:00:15 UTC] {pod_launcher.py:149} INFO -     return callback(*args, **kwargs)
[2022-02-18, 14:00:15 UTC] {pod_launcher.py:149} INFO -   File "discord_data/python/bin/datahub/datahub_looker_ingest", line 33, in datahub_looker_ingest
[2022-02-18, 14:00:15 UTC] {pod_launcher.py:149} INFO -     'sink': {'type': 'datahub-rest', 'config': {'server': f'{server_url}'}},
[2022-02-18, 14:00:15 UTC] {pod_launcher.py:149} INFO -   File "/usr/local/lib/python3.7/site-packages/datahub/ingestion/run/pipeline.py", line 175, in create
[2022-02-18, 14:00:15 UTC] {pod_launcher.py:149} INFO -     return cls(config, dry_run=dry_run, preview_mode=preview_mode)
[2022-02-18, 14:00:15 UTC] {pod_launcher.py:149} INFO -   File "/usr/local/lib/python3.7/site-packages/datahub/ingestion/run/pipeline.py", line 116, in __init__
[2022-02-18, 14:00:15 UTC] {pod_launcher.py:149} INFO -     preview_mode=preview_mode,
[2022-02-18, 14:00:15 UTC] {pod_launcher.py:149} INFO -   File "/usr/local/lib/python3.7/site-packages/datahub/ingestion/api/common.py", line 41, in __init__
[2022-02-18, 14:00:15 UTC] {pod_launcher.py:149} INFO -     self.graph = DataHubGraph(datahub_api) if datahub_api is not None else None
[2022-02-18, 14:00:15 UTC] {pod_launcher.py:149} INFO -   File "/usr/local/lib/python3.7/site-packages/datahub/ingestion/graph/client.py", line 47, in __init__
[2022-02-18, 14:00:15 UTC] {pod_launcher.py:149} INFO -     ca_certificate_path=self.config.ca_certificate_path,
[2022-02-18, 14:00:15 UTC] {pod_launcher.py:149} INFO -   File "/usr/local/lib/python3.7/site-packages/datahub/emitter/rest_emitter.py", line 121, in __init__
[2022-02-18, 14:00:15 UTC] {pod_launcher.py:149} INFO -     allowed_methods=self._retry_methods,
[2022-02-18, 14:00:15 UTC] {pod_launcher.py:149} INFO - TypeError: __init__() got an unexpected keyword argument 'allowed_methods'
Any thoughts on what could be causing this? Could also be that another dependency needs to be updated
s
Maybe try updating to the latest
0.8.26.7
https://github.com/acryldata/datahub/releases and retrying this?
p
I think it might be because we are using urlib 1.25 my local has 1.26 and its working fine
but ill update the datahub version as well while im at it
@square-activity-64562 Fixed our urlib issue, but after updating to .7 i'm now seeing:
Copy code
[2022-02-22, 16:21:23 UTC] {pod_launcher.py:149} INFO -               'dataset-name': ['Profiling exception No BigQuery dataset specified. Use '
[2022-02-22, 16:21:23 UTC] {pod_launcher.py:149} INFO -                                                                                        'bigquery_temp_table batch_kwarg or a specify a default '
[2022-02-22, 16:21:23 UTC] {pod_launcher.py:149} INFO -                                                                                        'dataset in engine url'],