Hi all! I'm building an open source no-code data transformation tool to enable people to manipulate their data without writing SQL. The idea is to use PRQL to build a query step by step. Each step being a statement in PRQL (eg. "filter" or "group"), and you would chain steps one after another. I was thinking Datahub could be used as a backend to store those steps. There would be a

Transform

entity, which would have a number of

TransformSteps

entities as children. Datahub is efficient at ingesting new steps being added, and retrieving them efficiently. It has the added bonus of being able to provide lineage with the rest of the stack. Does this make sense to you, or am I completely out of my mind? Have a great day, PS: Thank you for building Datahub, that's a really great contribution to the community!!

This seems like a pretty cool idea.. a transformation system built on top of a metadata store... seems logical when you think about it 🙂

Thank you Shirshanka for your feedback! I'll let you know if that works out 🙂

I don’t have much to contribute regarding using datahub as a backend for your system, but maybe you could use supabase as an open-source alternative to Firebase

My two cents, integrating two systems from the get go is going to be a large overhead. Storing your metadata into a DB for fast access & ability to customize how you store metadata is going to be much more efficient as you're starting out a project. You also avoid "vendor" lock in, and could ingest all the info into datahub once the original product is tested and has adoption.