Hi friends. Is Elementary could be used as a Data Profiling Tool. I have been asking ChatGPT, getting different responses every time.

Hi @Cristián Munizaga! that's a very cool usage of ChatGPT ! Currently we collect metrics on the underlying tables / columns if you define an Elementary anomaly test on your tables / columns. Then you can use the collected metrics to learn how your data behaves. We do want to expose these metrics in the UI and collect metrics regardless of the tests but that's something we are still scoping and thinking about. In any case, I'd love to learn more about your use case!

Hi @Or Avidov, nice to meet you, and thanks for your reply. I delayed answering because I was trying to better understand how elementary works. It looks like Column Level Monitors provides the basic stats but in the context of anomalies.

Nice to meet you too @Cristián Munizaga! I wouldn't necessarily compare these two tools as they serve different needs. Pandas profiling is mainly used to profile the data to get a better understanding how the data behaves. For example, if you want to see the amount of null values in a specific column you can profile your table and learn how many nulls you have in your columns. Elementary is more a data observability solution that enables you to monitor the data and validate that there are no issues in the data over time. So now when you have an idea how many nulls you have in a column, you can decide if you want to monitor the amount of null values over time using Elementary. A common practice that we see across our users is that they profile the data using tools like pandas profiling and then they know what tests are important for them to define with Elementary on their data sets. What do you think about this approach? does it make sense? Down the road and in order to simplify this process we might add data metrics and profiling of the data to Elementary as well but in general these are complementary tools which work nicely together.

Thanks, @Or Avidov. You are absolutely right. There is an overlap between once-time and tracking data profiling, but every side has exclusive use cases. Is it possible to add custom metrics to Elementary to extend Table and Column monitors?

That's correct @Cristián Munizaga! we don't support custom metrics yet but we already have two open issues about it here and here and we are definitely going to prioritize it soon!

Hi @Or Avidov do you have any updates for this? just because both those tickets are closed with not much info available