hello , i have a problem when trying ingest MySQL ...
# troubleshoot
n
hello , i have a problem when trying ingest MySQL
TypeError: 21:30:00 is of type timedelta which cannot be serialized
does anyone has experience how to solve it ?
d
can you provide a full stacktrace if it is available?
n
Traceback (most recent call last):
File "/root/datahub/datahub/lib64/python3.7/site-packages/datahub/ingestion/source/ge_data_profiler.py", line 313, in _get_column_cardinality
unique_count = self.dataset.get_column_unique_count(column)
File "/root/datahub/datahub/lib64/python3.7/site-packages/datahub/ingestion/source/ge_data_profiler.py", line 134, in get_column_unique_count_patch
self._table
File "/root/datahub/datahub/lib64/python3.7/site-packages/datahub/utilities/sqlalchemy_query_combiner.py", line 259, in _sa_execute_fake
return _sa_execute_underlying_method(conn, query, *args, **kwargs)
File "/root/datahub/datahub/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/root/datahub/datahub/lib64/python3.7/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/root/datahub/datahub/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement
distilled_params,
File "/root/datahub/datahub/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
e, statement, parameters, cursor, context
File "/root/datahub/datahub/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1514, in _handle_dbapi_exception
util.raise_(exc_info[1], with_traceback=exc_info[2])
File "/root/datahub/datahub/lib64/python3.7/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/root/datahub/datahub/lib64/python3.7/site-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
cursor, statement, parameters, context
File "/root/datahub/datahub/lib64/python3.7/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
cursor.execute(statement, parameters)
File "/root/datahub/datahub/lib64/python3.7/site-packages/pymysql/cursors.py", line 148, in execute
result = self._query(query)
File "/root/datahub/datahub/lib64/python3.7/site-packages/pymysql/cursors.py", line 310, in _query
conn.query(q)
File "/root/datahub/datahub/lib64/python3.7/site-packages/pymysql/connections.py", line 548, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/root/datahub/datahub/lib64/python3.7/site-packages/pymysql/connections.py", line 775, in _read_query_result
result.read()
File "/root/datahub/datahub/lib64/python3.7/site-packages/pymysql/connections.py", line 1163, in read
self._read_result_packet(first_packet)
File "/root/datahub/datahub/lib64/python3.7/site-packages/pymysql/connections.py", line 1236, in _read_result_packet
self._read_rowdata_packet()
File "/root/datahub/datahub/lib64/python3.7/site-packages/pymysql/connections.py", line 1274, in _read_rowdata_packet
rows.append(self._read_row_from_packet(packet))
File "/root/datahub/datahub/lib64/python3.7/site-packages/pymysql/connections.py", line 1294, in _read_row_from_packet
data = converter(data)
ValueError: invalid literal for int() with base 10: ''
@dazzling-judge-80093
d
Is this the stacktrace which belongs to the initial error?
Copy code
TypeError: 21:30:00 is of type timedelta which cannot be serialized
Here it seems like it fails on an int field while previously it was complaining about a timedelta field
it seems like pymysql fails on certain values during profiling when getting unique count
As quick fix you can disable the columns from profiling where it fails