Hi! I am new here, testing possibilities with data...
# ingestion
m
Hi! I am new here, testing possibilities with datahub. With ingestion and profiling, I have a problem. My recipe looks like this source: type: mssql config: env: dev username: datahubproject password: supersecret database: ShopfloorMgmt host_port: 'host:1433' profiling: enabled: true The ingestion works, but I dont see any stats. In the logs, there is the following - it says, profiling done for 99 tables, but there is a bunch of error messages right before: "AttributeError: 'CreateColumn' object has no attribute 'name'\n" '[2022-07-14 074242,478] ERROR {datahub.utilities.sqlalchemy_query_combiner:250} - Failed to execute query normally, using fallback: ' 'INSERT INTO [#ge_temp_95eb8c63] (condition) SELECT CASE WHEN (1 = 1 AND [BillingDocumentCategory] IS NOT NULL) THEN %(param_1)s ELSE ' '%(param_2)s END AS condition \n' 'FROM dbo.[SalesDocumentItems]\n' 'Traceback (most recent call last):\n' ' File ' '"/tmp/datahub/ingest/venv-a4978ec6-eceb-460e-94e9-7068140d0b35/lib/python3.9/site-packages/datahub/utilities/sqlalchemy_query_combiner.py", ' 'line 111, in get_query_columns\n' ' inner_columns = list(query.inner_columns)\n' "AttributeError: 'Insert' object has no attribute 'inner_columns'\n" '\n' 'During handling of the above exception, another exception occurred:\n' '\n' 'Traceback (most recent call last):\n' ' File ' '"/tmp/datahub/ingest/venv-a4978ec6-eceb-460e-94e9-7068140d0b35/lib/python3.9/site-packages/datahub/utilities/sqlalchemy_query_combiner.py", ' 'line 246, in _sa_execute_fake\n' ' handled, result = self._handle_execute(conn, query, args, kwargs)\n' ' File ' '"/tmp/datahub/ingest/venv-a4978ec6-eceb-460e-94e9-7068140d0b35/lib/python3.9/site-packages/datahub/utilities/sqlalchemy_query_combiner.py", ' 'line 211, in _handle_execute\n' ' if not self.is_single_row_query_method(query):\n' ' File ' '"/tmp/datahub/ingest/venv-a4978ec6-eceb-460e-94e9-7068140d0b35/lib/python3.9/site-packages/datahub/ingestion/source/ge_data_profiler.py", ' 'line 220, in _is_single_row_query_method\n' ' query_columns = get_query_columns(query)\n' ' File ' '"/tmp/datahub/ingest/venv-a4978ec6-eceb-460e-94e9-7068140d0b35/lib/python3.9/site-packages/datahub/utilities/sqlalchemy_query_combiner.py", ' 'line 114, in get_query_columns\n' ' return list(query.columns)\n' "AttributeError: 'Insert' object has no attribute 'columns'\n" '[2022-07-14 074300,878] INFO {datahub.ingestion.source.ge_data_profiler:930} - Finished profiling ' 'WMITShopfloorMgmt.dbo.SalesDocumentItems; took 71.096 seconds\n' '[2022-07-14 074300,971] INFO {datahub.ingestion.source.ge_data_profiler:776} - Profiling 99 table(s) finished in 117.176 seconds\n'
c
Which version of datahub are you using? This seems to be some issue with installation?
e
Hi there! See if one of the recommendations below helps you answer your question.
m
Hi, thank you! It is version v0.8.40, running in docker containers on ubuntu. I`ll check the recomm
endations, I assume it is a spark issue.
c
@magnificent-kangaroo-91705 are you still stuck here? Please tag me if you need help. Thanks
m
Hello! I faced the same issue. Please tell me the solution to the problem. 🙏Can't create temporary table.
Copy code
"AttributeError: 'Insert' object has no attribute 'columns'\n"
           '[2022-08-31 09:46:52,291] ERROR    {datahub.utilities.sqlalchemy_query_combiner:249} - Failed to execute query normally, using '
           'fallback: \n'
           'CREATE TABLE "#ge_temp_dbf5dfdd" (\n'
           '\tcondition INTEGER NOT NULL\n'
           ')\n'
           '\n'
@magnificent-kangaroo-91705 Have you solved this problem?