hey guys, here is the need we have: I have hand rolled a connector framework in elixir that pulls from various APIs and normalizes data into a postgres db (data warehouse). so we have all this raw data sitting in this db but what we need to do is to create new tables based on the raw tables, i.e. blending data from various tables into a single table which will be pulled into domo for analytical purposes. so we need postgres db -> transformed via sql query or similar -> new table in postgres db is airbyte a match for this use case?

We have tooling to create connectors to different APIs (and have some existing sources). We also perform normalization on that data, but I believe some of what you’re referring to as normalization is baked into our connector definitions.

yeah we were looking at DBT here too

Are you looking to replace the hand rolled framework or just the transformation piece?

it’s complicated 🙂 the breakneck pace that we need to add new connectors and the lack of dev resources to do it might force us to use an external tool that’s easier for non-devs to use, but we are pretty happy with our connector framework for now technically and operationally, just need that piece to blend tables together, like DBT

Most of our work right now for connector development is actually focused at devs to provide state management, different replication methods, improving the speed of development, etc. Currently we don’t have tooling specifically for reverse ELT (like the domo -> hubspot case) but our framework is flexible enough to support.

thanks for that insight Jared

thanks!