Hello, I'm trying to use UI ingestion and got foll...
# ingestion
s
Hello, I'm trying to use UI ingestion and got following error:
Copy code
~~~~ Execution Summary ~~~~

RUN_INGEST - {'errors': [],
 'exec_id': 'fe59f987-686e-4078-8f83-eb1ddf63fc2f',
 'infos': ['2022-09-15 12:22:12.253433 [exec_id=fe59f987-686e-4078-8f83-eb1ddf63fc2f] INFO: Starting execution for task with name=RUN_INGEST',
           '2022-09-15 12:22:48.478154 [exec_id=fe59f987-686e-4078-8f83-eb1ddf63fc2f] INFO: stdout=venv setup time = 0\n'
           'This version of datahub supports report-to functionality\n'
           'datahub  ingest run -c /tmp/datahub/ingest/fe59f987-686e-4078-8f83-eb1ddf63fc2f/recipe.yml --report-to '
           '/tmp/datahub/ingest/fe59f987-686e-4078-8f83-eb1ddf63fc2f/ingestion_report.json\n'
           '[2022-09-15 12:22:34,221] INFO     {datahub.cli.ingest_cli:182} - DataHub CLI version: 0.8.44.2\n'
           '[2022-09-15 12:22:34,243] INFO     {datahub.ingestion.run.pipeline:175} - Sink configured successfully. DataHubRestEmitter: configured '
           'to talk to <http://datahub-datahub-gms:8080>\n'
           '[2022-09-15 12:22:46,903] ERROR    {datahub.entrypoints:192} - \n'
           'Traceback (most recent call last):\n'
           '  File "/usr/local/lib/python3.9/site-packages/datahub/ingestion/run/pipeline.py", line 196, in __init__\n'
           '    self.source: Source = source_class.create(\n'
           '  File "/usr/local/lib/python3.9/site-packages/datahub/ingestion/source/sql/bigquery.py", line 989, in create\n'
           '    config = BigQueryConfig.parse_obj(config_dict)\n'
           '  File "pydantic/main.py", line 521, in pydantic.main.BaseModel.parse_obj\n'
           '  File "/usr/local/lib/python3.9/site-packages/datahub/ingestion/source_config/sql/bigquery.py", line 69, in __init__\n'
           '    super().__init__(**data)\n'
           '  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__\n'
           'pydantic.error_wrappers.ValidationError: 1 validation error for BigQueryConfig\n'
           'include_view_lineage\n'
           '  extra fields not permitted (type=value_error.extra)\n'
           '\n'
           'The above exception was the direct cause of the following exception:\n'
           '\n'
           'Traceback (most recent call last):\n'
           '  File "/usr/local/lib/python3.9/site-packages/datahub/cli/ingest_cli.py", line 197, in run\n'
           '    pipeline = Pipeline.create(\n'
           '  File "/usr/local/lib/python3.9/site-packages/datahub/ingestion/run/pipeline.py", line 317, in create\n'
           '    return cls(\n'
           '  File "/usr/local/lib/python3.9/site-packages/datahub/ingestion/run/pipeline.py", line 202, in __init__\n'
           '    self._record_initialization_failure(\n'
           '  File "/usr/local/lib/python3.9/site-packages/datahub/ingestion/run/pipeline.py", line 129, in _record_initialization_failure\n'
           '    raise PipelineInitError(msg) from e\n'
           'datahub.ingestion.run.pipeline.PipelineInitError: Failed to configure source (bigquery)\n'
           '[2022-09-15 12:22:46,903] ERROR    {datahub.entrypoints:195} - Command failed: \n'
           '\tFailed to configure source (bigquery) due to \n'
           "\t\t'1 validation error for BigQueryConfig\n"
           'include_view_lineage\n'
           "  extra fields not permitted (type=value_error.extra)'.\n"
           '\tRun with --debug to get full stacktrace.\n'
           "\te.g. 'datahub --debug ingest run -c /tmp/datahub/ingest/fe59f987-686e-4078-8f83-eb1ddf63fc2f/recipe.yml --report-to "
           "/tmp/datahub/ingest/fe59f987-686e-4078-8f83-eb1ddf63fc2f/ingestion_report.json'\n",
           "2022-09-15 12:22:48.478380 [exec_id=fe59f987-686e-4078-8f83-eb1ddf63fc2f] INFO: Failed to execute 'datahub ingest'",
           '2022-09-15 12:22:48.478596 [exec_id=fe59f987-686e-4078-8f83-eb1ddf63fc2f] INFO: Caught exception EXECUTING '
           'task_id=fe59f987-686e-4078-8f83-eb1ddf63fc2f, name=RUN_INGEST, stacktrace=Traceback (most recent call last):\n'
           '  File "/usr/local/lib/python3.9/site-packages/acryl/executor/execution/default_executor.py", line 123, in execute_task\n'
           '    task_event_loop.run_until_complete(task_future)\n'
           '  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete\n'
           '    return future.result()\n'
           '  File "/usr/local/lib/python3.9/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 168, in execute\n'
           '    raise TaskError("Failed to execute \'datahub ingest\'")\n'
           "acryl.executor.execution.task.TaskError: Failed to execute 'datahub ingest'\n"]}
Execution finished with errors.
This is recipe generated:
Copy code
source:
    type: bigquery
    config:
        include_table_lineage: false
        credential:
            private_key_id: '${private-key-id}'
            project_id: '${project_id}'
            client_email: '${client_email}'
            private_key: '${private-key}'
            client_id: '${client_id}'
        profiling:
            enabled: false
        project_id: '${project_id}'
        include_view_lineage: false
        stateful_ingestion:
            enabled: false
pipeline_name: 'urn:li:dataHubIngestionSource:75869f85-d290-476a-a359-a7a6e6015ebc'
v0.8.44
b
Hi @square-bird-94136, i think include_view_lineage is not present in config option for bigquery
s
But this was generated by DataHub
g
This is a bug - could you click the “yaml” button the source config panel and remove the “include_view_lineage” line?