Hi! I am new here, testing possibilities with datahub.
With ingestion and profiling, I have a problem.
My recipe looks like this
source:
type: mssql
config:
env: dev
username: datahubproject
password: supersecret
database: ShopfloorMgmt
host_port: 'host:1433'
profiling:
enabled: true
The ingestion works, but I dont see any stats.
In the logs, there is the following - it says, profiling done for 99 tables, but there is a bunch of error messages right before:
"AttributeError: 'CreateColumn' object has no attribute 'name'\n"
'[2022-07-14 07
4242,478] ERROR {datahub.utilities.sqlalchemy_query_combiner:250} - Failed to execute query normally, using fallback: '
'INSERT INTO [#ge_temp_95eb8c63] (condition) SELECT CASE WHEN (1 = 1 AND [BillingDocumentCategory] IS NOT NULL) THEN %(param_1)s ELSE '
'%(param_2)s END AS condition \n'
'FROM dbo.[SalesDocumentItems]\n'
'Traceback (most recent call last):\n'
' File '
'"/tmp/datahub/ingest/venv-a4978ec6-eceb-460e-94e9-7068140d0b35/lib/python3.9/site-packages/datahub/utilities/sqlalchemy_query_combiner.py", '
'line 111, in get_query_columns\n'
' inner_columns = list(query.inner_columns)\n'
"AttributeError: 'Insert' object has no attribute 'inner_columns'\n"
'\n'
'During handling of the above exception, another exception occurred:\n'
'\n'
'Traceback (most recent call last):\n'
' File '
'"/tmp/datahub/ingest/venv-a4978ec6-eceb-460e-94e9-7068140d0b35/lib/python3.9/site-packages/datahub/utilities/sqlalchemy_query_combiner.py", '
'line 246, in _sa_execute_fake\n'
' handled, result = self._handle_execute(conn, query, args, kwargs)\n'
' File '
'"/tmp/datahub/ingest/venv-a4978ec6-eceb-460e-94e9-7068140d0b35/lib/python3.9/site-packages/datahub/utilities/sqlalchemy_query_combiner.py", '
'line 211, in _handle_execute\n'
' if not self.is_single_row_query_method(query):\n'
' File '
'"/tmp/datahub/ingest/venv-a4978ec6-eceb-460e-94e9-7068140d0b35/lib/python3.9/site-packages/datahub/ingestion/source/ge_data_profiler.py", '
'line 220, in _is_single_row_query_method\n'
' query_columns = get_query_columns(query)\n'
' File '
'"/tmp/datahub/ingest/venv-a4978ec6-eceb-460e-94e9-7068140d0b35/lib/python3.9/site-packages/datahub/utilities/sqlalchemy_query_combiner.py", '
'line 114, in get_query_columns\n'
' return list(query.columns)\n'
"AttributeError: 'Insert' object has no attribute 'columns'\n"
'[2022-07-14 07
4300,878] INFO {
datahub.ingestion.source.ge_data_profiler:930} - Finished profiling '
'WMITShopfloorMgmt.dbo.SalesDocumentItems; took 71.096 seconds\n'
'[2022-07-14 07
4300,971] INFO {
datahub.ingestion.source.ge_data_profiler:776} - Profiling 99 table(s) finished in 117.176 seconds\n'