Hi everyone, about the data profiling feature, I n...
# ingestion
m
Hi everyone, about the data profiling feature, I noticed we use Great Expectations for SQL data stores and Deequ for data lake. Can I ask if the considerations behind this (despite Deequ lacks SQLAlchemy support)? If possible, I would also like to learn your comparison between those two tools, such as which tool is better for which scenarios, etc.
👍 1
s
If we also consider that: • deequ need pyspark installed • it’s not really maintained (last update on September 2021) maybe it’s best to uniform the data profiling, and use GE also for data lakes
thank you 1