How do i ingest data queries/Stats
# ingestion
d
How do i ingest data queries/Stats
r
you’ll have to turn on profiling to ingest that - https://datahubproject.io/docs/metadata-ingestion/source_docs/sql_profiles
plus1 2
d
Got some error, and cannot see any Stats in my dataset
h
Hi @damp-ambulance-34232, could you share more details about your mysql instance, such as the the version and the storage engine being used?
Also, do these columns have very high cardinality?
Could you try setting the
profiling.limit
to 1000 and try to run ingestion again?
d
@helpful-optician-78938 When set limit, i saw it create temp table, but my account doesn't have access create temporary table
OperationalError: (pymysql.err.OperationalError) (1044, "Access denied for user 'etl'@'%' to database 'cs_thunder'") [SQL: CREATE TEMPORARY TABLE ge_temp_befa87a5 AS SELECT * FROM cs_thunder.csdb_bigsize_products LIMIT 10000]
pymysql.err.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 't AS \n(SELECT shop_order, CAST(percent_rank() OVER (ORDER BY shop_order ASC) AS ' at line 1") The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/great_expectations/dataset/sqlalchemy_dataset.py", line 1016, in _get_column_quantiles_mysql quantiles_results: Row = self.engine.execute(quantiles_query).fetchone() File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1011, in execute return meth(self, multiparams, params) File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement distilled_params, File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context e, statement, parameters, cursor, context File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception sqlalchemy_exception, with_traceback=exc_info[2], from_=e File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line 182, in raise_ raise exception File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context cursor, statement, parameters, context File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 608, in do_execute cursor.execute(statement, parameters) File "/usr/local/lib/python3.6/dist-packages/pymysql/cursors.py", line 148, in execute result = self._query(query) File "/usr/local/lib/python3.6/dist-packages/pymysql/cursors.py", line 310, in _query conn.query(q) File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 548, in query self._affected_rows = self._read_query_result(unbuffered=unbuffered) File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 775, in _read_query_result result.read() File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 1156, in read first_packet = self.connection._read_packet() File "/usr/local/lib/python3.6/dist-packages/pymysql/connections.py", line 725, in _read_packet packet.raise_for_error() File "/usr/local/lib/python3.6/dist-packages/pymysql/protocol.py", line 221, in raise_for_error err.raise_mysql_exception(self._data) File "/usr/local/lib/python3.6/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception raise errorclass(errno, errval) sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 't AS \n(SELECT shop_order, CAST(percent_rank() OVER (ORDER BY shop_order ASC) AS ' at line 1") [SQL: WITH t AS (SELECT shop_order, CAST(percent_rank() OVER (ORDER BY shop_order ASC) AS DECIMAL(18, 15)) AS p FROM cs_thunder.csdb_bigsize_packages ORDER BY p ASC) SELECT DISTINCT first_value(shop_order) OVER (ORDER BY CASE WHEN (t.p <= CAST(%(param_1)s AS DECIMAL(18, 15))) THEN t.p END DESC) AS q_0, first_value(shop_order) OVER (ORDER BY CASE WHEN (t.p <= CAST(%(param_2)s AS DECIMAL(18, 15))) THEN t.p END DESC) AS q_1, first_value(shop_order) OVER (ORDER BY CASE WHEN (t.p <= CAST(%(param_3)s AS DECIMAL(18, 15))) THEN t.p END DESC) AS q_2, first_value(shop_order) OVER (ORDER BY CASE WHEN (t.p <= CAST(%(param_4)s AS DECIMAL(18, 15))) THEN t.p END DESC) AS q_3 FROM t ORDER BY t.p DESC] [parameters: {'param_1': 0.0, 'param_2': 0.25, 'param_3': 0.75, 'param_4': 1.0}] (Background on this error at: http://sqlalche.me/e/13/f405)
@helpful-optician-78938 Db: Mysql Version: 5.7.35-38-log
Copy code
innodb_version	5.7.35-38
protocol_version	10
slave_type_conversions	
tls_version	TLSv1,TLSv1.1,TLSv1.2
version	5.7.35-38-log
version_comment	Percona Server (GPL), Release '38', Revision '3692a61'
version_compile_machine	x86_64
version_compile_os	debian-linux-gnu
version_suffix	-log
h
Hi @damp-ambulance-34232, (1) looks like the error related to quantiles is related to how to sql alchemy is configured. I'll dig further into this. To unblock yourself for now, please exclude quantile stats from your profiling (set profile.include_field_quantiles to False). (2) The way GE profiling works is by creating some temporary tables. Please make sure that the user has sufficient privileges to create table in each of the databases being profiled.
plus1 1