Hi, during import with profiling from Oracle we fa...
# troubleshoot
a
Hi, during import with profiling from Oracle we face an run failure. Would anyone help us to solve this? Details bellow
``` 'sqlalchemy.exc.DatabaseError: (cx_Oracle.DatabaseError) ORA-00936: missing expression\n'
'[SQL: SELECT FROM DUAL \n'
'WHERE ROWNUM <= :param_1]\n'
"[parameters: {'param_1': 1}]\n"
'(Background on this error at: http://sqlalche.me/e/13/4xp6)\n'
'[2022-11-29 120449,030] INFO {datahub.ingestion.source.ge_data_profiler:909} - Profiling '
'scheme_name.table_name\n'
'[2022-11-29 120449,037] ERROR {datahub.ingestion.source.ge_data_profiler:939} - Encountered exception while profiling '```
Without enabled profiling it runs fine without error. Here is our ingestion recipe:
Copy code
source:
    type: oracle
    config:
        env: TEST
        password: xxxx
        host_port: 'ip:port'
        service_name: test
        username: xxxx
        schema_pattern:
            allow:
scheme_name
        include_views: true
        include_tables: true
        profiling:
            enabled: true
pipeline_name: 'xxx'
Probably it causes query SELECT FROM DUAL which can be seen in log.
d
Does this stop all the profiling or only this table fails to get profiled?
a
@dazzling-judge-80093, the whole ingestion fails
a
Hello @ancient-wire-3767, @dazzling-judge-80093 I have encountered exactly same error as you have with SELECT FROM DUAL. How did you manage to overcome it? I'm using version 0.12.0 and here is my log:
2023-11-21 10:57:52,867] ERROR    {<http://datahub.ingestion.source.ge|datahub.ingestion.source.ge>_data_profiler:1094} - Encountered exception while profiling scheme.table_name
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
cx_Oracle.DatabaseError: ORA-00936: missing expression
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/source/ge_data_profiler.py", line 1065, in _generate_single_profile
batch = self._get_ge_dataset(
File "/usr/local/lib/python3.10/site-packages/datahub/ingestion/source/ge_data_profiler.py", line 1128, in _get_ge_dataset
batch = ge_context.data_context.get_batch(
File "/usr/local/lib/python3.10/site-packages/great_expectations/data_context/data_context/abstract_data_context.py", line 1076, in get_batch
return self._get_batch_v2(
File "/usr/local/lib/python3.10/site-packages/great_expectations/data_context/data_context/abstract_data_context.py", line 1204, in _get_batch_v2
return validator.get_dataset()
File "/usr/local/lib/python3.10/site-packages/great_expectations/validator/validator.py", line 2122, in get_dataset
return self.expectation_engine(
File "/usr/local/lib/python3.10/site-packages/great_expectations/dataset/sqlalchemy_dataset.py", line 909, in __init__
self.columns = self.column_reflection_fallback()
File "/usr/local/lib/python3.10/site-packages/great_expectations/dataset/sqlalchemy_dataset.py", line 1769, in column_reflection_fallback
col_names: list = self.engine.execute(query).keys()
File "/usr/local/lib/python3.10/site-packages/datahub/utilities/sqlalchemy_query_combiner.py", line 273, in _sa_execute_fake
return _sa_execute_underlying_method(conn, query, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1380, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection
return connection._execute_clauseelement(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
ret = self._execute_context(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
self._handle_dbapi_exception(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception
util.raise_(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
raise exception
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
self.dialect.do_execute(
File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.DatabaseError: (cx_Oracle.DatabaseError) ORA-00936: missing expression
[SQL: SELECT  FROM DUAL
WHERE ROWNUM <= 1]
(Background on this error at: <https://sqlalche.me/e/14/4xp6>)