Hi, What's the reason for this:(v0.8.41), The prob...
# all-things-deployment
f
Hi, What's the reason for this:(v0.8.41), The problem of my positioning is caused by "table_pattern,schema_pattern"
Copy code
from datahub.ingestion.run import pipeline
from datahub.configuration.common import DynamicTypedConfig

config = pipeline.PipelineConfig(
    source=pipeline.SourceConfig(
        type="mysql",
        config={
            "username": "datahub",
            "password": "datahub",
            "database": "action_airflow",
            "host_port": "10.196.48.76:3306",
            "include_views": False,
            "include_tables": True,
            "table_pattern": {
                        "allow": ["action_airflow.*"]
            },
            "schema_pattern": {
                "allow": ["action_airflow.*"]
            },
            "profiling": {
                "enabled": True
            }
        }
    ),
    sink=DynamicTypedConfig(
        type="datahub-rest",
        config={
            "server": "<http://xxxx>",
            "token": "xxxx"
        }
    )
)

pip = pipeline.Pipeline(config=config)
pip.run()
Copy code
/home/shdzh/.local/lib/python3.9/site-packages/datahub/utilities/sqlalchemy_query_combiner.py:321: SADeprecationWarning: The Select.append_from() method is deprecated and will be removed in a future release.  Use the generative method Select.select_from(). (deprecated since: 1.4)
  combined_query.append_from(cte)
Failed to execute queue using combiner: (pymysql.err.ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'xpbuxnmrasxaqeaz AS \n(SELECT count(*) AS count_1 \nFROM action_airflow.creator_ch' at line 1")
[SQL: WITH xpbuxnmrasxaqeaz AS 
(SELECT count(*) AS count_1 
FROM action_airflow.creator_channels)
 SELECT xpbuxnmrasxaqeaz.count_1 
FROM xpbuxnmrasxaqeaz]
(Background on this error at: <https://sqlalche.me/e/14/f405>)
Traceback (most recent call last):
  File "/home/shdzh/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
    self.dialect.do_execute(
  File "/home/shdzh/.local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
  File "/home/shdzh/.local/lib/python3.9/site-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
  File "/home/shdzh/.local/lib/python3.9/site-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "/home/shdzh/.local/lib/python3.9/site-packages/pymysql/connections.py", line 548, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/home/shdzh/.local/lib/python3.9/site-packages/pymysql/connections.py", line 775, in _read_query_result
    result.read()
  File "/home/shdzh/.local/lib/python3.9/site-packages/pymysql/connections.py", line 1156, in read
    first_packet = self.connection._read_packet()
  File "/home/shdzh/.local/lib/python3.9/site-packages/pymysql/connections.py", line 725, in _read_packet
    packet.raise_for_error()
  File "/home/shdzh/.local/lib/python3.9/site-packages/pymysql/protocol.py", line 221, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/home/shdzh/.local/lib/python3.9/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'xpbuxnmrasxaqeaz AS \n(SELECT count(*) AS count_1 \nFROM action_airflow.creator_ch' at line 1")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/shdzh/.local/lib/python3.9/site-packages/datahub/utilities/sqlalchemy_query_combiner.py", line 383, in flush
    self._execute_queue(main_greenlet)
  File "/home/shdzh/.local/lib/python3.9/site-packages/datahub/utilities/sqlalchemy_query_combiner.py", line 325, in _execute_queue
    sa_res = _sa_execute_underlying_method(queue_item.conn, combined_query)
  File "/home/shdzh/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1380, in execute
    return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
  File "/home/shdzh/.local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/home/shdzh/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
    ret = self._execute_context(
  File "/home/shdzh/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
    self._handle_dbapi_exception(
  File "/home/shdzh/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception
    util.raise_(
  File "/home/shdzh/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
  File "/home/shdzh/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
    self.dialect.do_execute(
  File "/home/shdzh/.local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
    cursor.execute(statement, parameters)
  File "/home/shdzh/.local/lib/python3.9/site-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
  File "/home/shdzh/.local/lib/python3.9/site-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "/home/shdzh/.local/lib/python3.9/site-packages/pymysql/connections.py", line 548, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/home/shdzh/.local/lib/python3.9/site-packages/pymysql/connections.py", line 775, in _read_query_result
    result.read()
  File "/home/shdzh/.local/lib/python3.9/site-packages/pymysql/connections.py", line 1156, in read
    first_packet = self.connection._read_packet()
  File "/home/shdzh/.local/lib/python3.9/site-packages/pymysql/connections.py", line 725, in _read_packet
    packet.raise_for_error()
  File "/home/shdzh/.local/lib/python3.9/site-packages/pymysql/protocol.py", line 221, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/home/shdzh/.local/lib/python3.9/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
    raise errorclass(errno, errval)
d
This is more because of the profiling. It seems like GE generates an invalid sql somehow
f
How to solve it,The same data structure, all obtained, seems to have no problem
@dazzling-judge-80093 I use airflow to ingestion the same data source and the same configuration. There is no problem