Hello, I have question regarding enable profiling ...
# ingestion
n
Hello, I have question regarding enable profiling status when ingestion from Trino (only one db2 table, 56 columns, 200rows ) to datahub. Below is warning I consistent got after execute ingestion. BTW, It works fine and ingestion successfully when I disabled profiling status. Any thoughts or suggestion please let me know.
Copy code
[2021-10-26 14:41:41,449] WARNING  {great_expectations.dataset.sqlalchemy_dataset:1577} - No sqlalchemy dialect found; relying in top-level sqlalchemy types.
[2021-10-26 14:41:50,462] WARNING  {great_expectations.dataset.sqlalchemy_dataset:2023} - Regex is not supported for dialect <sqlalchemy_trino.dialect.TrinoDialect object at 0x13f1fb880>
[2021-10-26 14:41:54,881] WARNING  {great_expectations.dataset.sqlalchemy_dataset:1577} - No sqlalchemy dialect found; relying in top-level sqlalchemy types.
e
@helpful-optician-78938 any ideas on this one?
h
Hi @narrow-kitchen-1309, great expectations (the library our profiling is built on top of) currently does not seem to support column-level profiling for Trino (https://github.com/great-expectations/great_expectations/blob/develop/great_expectations/dataset/sqlalchemy_dataset.py#L2011). You will soon be able to further customize profiling and skip the columns that the support is still lacking for.
n
Thanks @helpful-optician-78938 for explanation. This make sense to me. It works fine for PostgreSQL, I did try that. Could you give me estimation when this could be available?
Just realized even skip columns. I still be able to see profiling status and history . This is good enough for my POC project. Thank you for your help. @helpful-optician-78938 and @early-lamp-41924
h
The PR is currently being reviewed. It should get merged soon. :-)
👍 1