Good morning you guys, I am a working student and I'm currently in my first Data Science project. For most of us, its our first 'real' Project, so most of my team lacks experience in some areas. 🙂 We're working with Azure and are storing our data in a Parquet and CSV format. I am tasked with researching and implementing the possibilities of a data catalog and have been recommended the Datahub project. I was wondering if it's possible to integrate datahub into our Pipeline, given our infrastructure. When going through the documentation I sadly couldn't find much about how to work with parquet data or integrating it with Azure. I would be super thankful if anyone could link me a resource for our use case so that I can get started. Thank you very much in advance ! :)

@gray-shoe-75895 would love your help here!

The delta lake connector might work. I think our s3 data lake connector, which confusingly also works for local file systems (which you’d have via the mount), also supports schema inference from parquet/csv