Hi! Im new to Airbyte. My aim is to create a connector to a system with a REST API. This API has for some objects lists of related objects, ie hierarchy. My first approach was to break these lists out as bridge tables inside the python connector code. Now I am wondering if that maybe is not the best approach. I think I read somewhere that there is a way to create dbt-models inside Airbyte or integrate the two. Maybe replicating the data just as is in Airbyte and then do the flattening of the data in dbt instead of in the Airbyte connector. Is my plan sound and if so how do I integrate Airbyte and dbt?

hiya Martin 😄 Airbyte is releasing the CDK (Connector Dev Kit) for Python maybe can be useful to you read about and help how organize your connector building (break into brigde tables or consume directly), this tool simplify the process to build new connectors. About dbt, there this documentation explaining how to run dbt after consume data from Airbyte. At this moment is not possible to customize the normalization process that Airbyte is doing (there some issues related to this). Hope this help you, but any question you are welcome to post here

Thanks! The CDK is where I started and have come a bit along with my project. So the normalization stuff in Airbyte would allow me to pass on data with hierarchy in it and let Airbyte flatten it? I am planning on storing the data in a relational database (SQL Server).

i should send to you this link about how the normalization is processed https://docs.airbyte.io/understanding-airbyte/basic-normalization

This looks really promesing!

😄 just to let you know, sql server as destination will be release in a near future

I thought it already was released. Is that a community version that I’ve seen and you are referring to the official destination?