Hi Team, Got below message for the BQ ingestion pi...
# troubleshoot
q
Hi Team, Got below message for the BQ ingestion pipeline run, I ran this with standard parameter with table profiling enabled:
Copy code
'[2023-01-23 17:42:42,108] WARNING  {py.warnings:109} - '
           '/tmp/datahub/ingest/venv-bigquery-0.9.6/lib/python3.10/site-packages/datahub/ingestion/source/bigquery_v2/bigquery.py:937: '
           'DeprecationWarning: Call to deprecated function (or staticmethod) wrap_aspect_as_workunit. (use '
           'MetadataChangeProposalWrapper(...).as_workunit() instead)\n'
           '  wu = wrap_aspect_as_workunit(\n'
           '\n'
           '[2023-01-23 17:42:42,110] WARNING  {py.warnings:109} - '
           '/tmp/datahub/ingest/venv-bigquery-0.9.6/lib/python3.10/site-packages/datahub/ingestion/source/bigquery_v2/bigquery.py:957: '
           'DeprecationWarning: Call to deprecated function (or staticmethod) wrap_aspect_as_workunit. (use '
           'MetadataChangeProposalWrapper(...).as_workunit() instead)\n'
           '  wu = wrap_aspect_as_workunit("dataset", dataset_urn, "subTypes", subTypes)\n'
           '\n'
           '[2023-01-23 17:42:42,190] DEBUG    {datahub.emitter.rest_emitter:250} - Attempting to emit to DataHub GMS; using curl equivalent to:\n',
           '2023-01-23 17:42:42.336687 [exec_id=96401624-f6b0-46e7-98c9-836345181165] INFO: Caught exception EXECUTING '
           'task_id=96401624-f6b0-46e7-98c9-836345181165, name=RUN_INGEST, stacktrace=Traceback (most recent call last):\n'
           '  File "/usr/local/lib/python3.10/asyncio/streams.py", line 525, in readline\n'
           '    line = await self.readuntil(sep)\n'
           '  File "/usr/local/lib/python3.10/asyncio/streams.py", line 620, in readuntil\n'
           '    raise exceptions.LimitOverrunError(\n'
           'asyncio.exceptions.LimitOverrunError: Separator is found, but chunk is longer than limit\n'
           '\n'
           'During handling of the above exception, another exception occurred:\n'
           '\n'
           'Traceback (most recent call last):\n'
           '  File "/usr/local/lib/python3.10/site-packages/acryl/executor/execution/default_executor.py", line 123, in execute_task\n'
           '    task_event_loop.run_until_complete(task_future)\n'
           '  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete\n'
           '    return future.result()\n'
           '  File "/usr/local/lib/python3.10/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 147, in execute\n'
           '    await tasks.gather(_read_output_lines(), _report_progress(), _process_waiter())\n'
           '  File "/usr/local/lib/python3.10/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 99, in _read_output_lines\n'
           '    line_bytes = await ingest_process.stdout.readline()\n'
           '  File "/usr/local/lib/python3.10/asyncio/streams.py", line 534, in readline\n'
           '    raise ValueError(e.args[0])\n'
           'ValueError: Separator is found, but chunk is longer than limit\n']}
Execution finished with errors.
b
Hi there! This appears to be an issue wherein the StdOut line is super super long and out executor is not expecting that. I'll take a look on how to fix this, but for now using the DataHub CLI itself (instead of UI) to perform ingestion should be a workaround!
q
Somewhere on the channel I read it is fixed with 0.0.8 actions version but as per the latest version of datahub it has 0.0.7 version
can we schedule it using the airflow DAGs ? if possible do you have sample for reference ?
Also in continuation to the issue, we upgraded the actions to 0.0.8 version, now getting the issue:
Copy code
skipping. Does your service account has bigquery.tables.list, bigquery.routines.get, bigquery.routines.list permission, bigquery.tables.getData permission? The error was: 'type'"]},
The SA have all these permissions and still throwing the error
n
Hi @quaint-barista-82836 @big-carpet-38439, I have encountered the same problem in both cases where I run ingestion via CLI or UI. Is there any workaround available?
q
@nice-match-35259, Looks like this is a bug,needs to be fixed