Hello everyone, how to avoid this kind of alarm? A...
# ingestionl
lemon-zoo-63387
07/06/2022, 6:08 AMHello everyone, how to avoid this kind of alarm? Although the UI is still alarming,
metadata has been ingested,Thanks in advance for your help!
Copy code
" 'CRM.crmqas_FE.systemuser': ['unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type BIT() to metadata schema',\n"
" 'unable to map type BIT() to metadata schema',\n"
" 'unable to map type BIT() to metadata schema',\n"
" 'unable to map type BIT() to metadata schema',\n"
" 'unable to map type BIT() to metadata schema',\n"
" 'unable to map type BIT() to metadata schema',\n"
" 'unable to map type BIT() to metadata schema',\n"
" 'unable to map type BIT() to metadata schema',\n"
" 'unable to map type BIT() to metadata schema',\n"
" 'unable to map type BIT() to metadata schema',\n"
" 'unable to map type BIT() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema'],\n"
" 'CRM.crmqas_FE.transactioncurrency': ['unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema',\n"
" 'unable to map type UNIQUEIDENTIFIER() to metadata schema']},\n"
' \'failures\': {\'DELTA\\\\EDITH.CE.CHANG\': ["Tables error: (pytds.tds_base.OperationalError) Database \'DELTA\\\\EDITH.CE\' does not '
'exist. Make sure that "\n'
" 'the name is entered correctly.\\n'\n"
" '[SQL: use [DELTA\\\\EDITH.CE]]\\n'\n"
" '(Background on this error at: <http://sqlalche.me/e/13/e3q8)>',\n"
' "Views error: (pytds.tds_base.OperationalError) Database \'DELTA\\\\EDITH.CE\' does not exist. '
'Make sure that "\n'
" 'the name is entered correctly.\\n'\n"
" '[SQL: use [DELTA\\\\EDITH.CE]]\\n'\n"
" '(Background on this error at: <http://sqlalche.me/e/13/e3q8)']>},\n"
" 'cli_version': '0.8.38',\n"
" 'cli_entry_location': '/tmp/datahub/ingest/venv-73071ee2-6365-4acf-b3e5-8fcaa08684dd/lib/python3.9/site-packages/datahub/__init__.py',\n"
" 'py_version': '3.9.9 (main, Dec 21 2021, 10:03:34) \\n[GCC 10.2.1 20210110]',\n"
" 'py_exec_path': '/tmp/datahub/ingest/venv-73071ee2-6365-4acf-b3e5-8fcaa08684dd/bin/python3',\n"
" 'os_details': 'Linux-3.10.0-1160.62.1.el7.x86_64-x86_64-with-glibc2.31',\n"
" 'tables_scanned': 4,\n"
" 'views_scanned': 167,\n"
" 'entities_profiled': 0,\n"
" 'filtered': [],\n"
" 'soft_deleted_stale_entities': [],\n"
" 'query_combiner': None}\n"
'Sink (datahub-rest) report:\n'
"{'records_written': 755,\n"
" 'warnings': [],\n"
" 'failures': [],\n"
" 'downstream_start_time': datetime.datetime(2022, 6, 25, 4, 0, 7, 991097),\n"
" 'downstream_end_time': datetime.datetime(2022, 6, 25, 4, 0, 32, 45487),\n"
" 'downstream_total_latency_in_seconds': 24.05439,\n"
" 'gms_version': 'v0.8.38'}\n"
'\n'
'Pipeline finished with 2 failures in source producing 755 workunits\n',
"2022-06-25 04:00:55.833939 [exec_id=73071ee2-6365-4acf-b3e5-8fcaa08684dd] INFO: Failed to execute 'datahub ingest'",
'2022-06-25 04:00:55.834347 [exec_id=73071ee2-6365-4acf-b3e5-8fcaa08684dd] INFO: Caught exception EXECUTING '
'task_id=73071ee2-6365-4acf-b3e5-8fcaa08684dd, name=RUN_INGEST, stacktrace=Traceback (most recent call last):\n'
' File "/usr/local/lib/python3.9/site-packages/acryl/executor/execution/default_executor.py", line 121, in execute_task\n'
' self.event_loop.run_until_complete(task_future)\n'
' File "/usr/local/lib/python3.9/site-packages/nest_asyncio.py", line 89, in run_until_complete\n'
' return f.result()\n'
' File "/usr/local/lib/python3.9/asyncio/futures.py", line 201, in result\n'
' raise self._exception\n'
' File "/usr/local/lib/python3.9/asyncio/tasks.py", line 256, in __step\n'
' result = coro.send(None)\n'
' File "/usr/local/lib/python3.9/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 115, in execute\n'
' raise TaskError("Failed to execute \'datahub ingest\'")\n'
"acryl.executor.execution.task.TaskError: Failed to execute 'datahub ingest'\n"]}
Execution finished with errors.b
better-orange-49102
07/06/2022, 6:15 AMwhat 'unable to map type BIT() to metadata schema',\n" is saying is that the ingestion logic has encountered an unaccounted field type in the source, and does not have a equivalent field, so they fallback to use a default fieldtype (typically NULL type)
l
lemon-zoo-63387
07/06/2022, 7:03 AMThanks,Can we modify the config to avoid this alarm? Let the UI display succeeded. The alarm is not very good-looking.
s
square-activity-64562
07/06/2022, 7:56 AMthat error comes from this file https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/sql/sql_common.py#L324. If you would like to remove that error then a contribution to this file with the correct mapping would be needed
Currently there isn't a way to remove that error. If you would like to make a fix please see https://datahubproject.io/docs/metadata-ingestion/developing/
l
little-megabyte-1074
07/06/2022, 6:50 PM
Hi @lemon-zoo-63387! Gentle reminder to please post large blocks of code/logs in message threads - it’s a HUGE help for us to keep track of open/unanswered questions across our support channels teamwork
l
little-spring-72943
11/24/2022, 9:43 PMThese type are there in SQLAlchemy - https://docs.sqlalchemy.org/en/14/dialects/mssql.html#sql-server-data-types
34 Views