Hey Airbyte team, I’ve used in my previous projects: Stitchdata, Airflow, Meltano, Singer and DBT, and now I’m trying to grasp all the details about Airbyte to decide if I use it for a new project I have.  I read all articles from Michel and John (about the differences between Singer, Airbyte and Meltano, also about the unified Data Infrastructure Architecture, etc) they make a lot of sense and I really enjoyed them :) I agree with all the problems you guys mentioned about Singer (and also recognize their importance in laying a foundation for the EL protocol). But I still have some questions that I wanted to double check with you: 1. When we use Airbyte and I want to run in production, I should install it on a GCP compute engine (in my case), access the UI and do all the connector’s setup via UI, similar to what I would do if I was using Stitchdata. But when you guys deploy a new version with new connectors, or change any connector, how do I deploy them to my server? Do I just pull from the github master branch? If I do that, what happens to: a) My configuration and state files? (I don’t lose it because Airbyte is using named volumes? There is no database for that like Airflow usually uses, right?) b) Could there be any compatibility issue if you deploy changes to a connector that I’m using? 2. If I want to develop a new connector that is not in your roadmap (for example a niche brazilian platform that we use here), a) Should I follow your guidelines and submit a pull request? (I read the documentation that mentions I don’t need to maintain anything). b) I could use a separate branch without sending a pull request, but this is not the desired scenario, correct? 3. As Airbyte’s goal is to handle only EL and basic normalization (equivalent to what Stitchdata does), for the ‘T’ part I should use something else, for example, DBT. a) And to run DBT I’ll need to use a separate orchestrator: it can be a simple cronjob, or airflow or their own platform, correct? So, differently from Meltano, Airbyte's goal is not to unify ELT frameworks under the same roof, but to be the best open source tool for EL correct? b) Do you plan using Kubernetes to parallelize processing and decrease server costs?

1. Yes, using compute engine works. While we are in alpha as you move from version to version, you need to start from scratch on your configuration. We are planning to be out of alpha in mid january at which time we will make sure to provide an upgrade path that lets you retain all of your configuration and state.

Thank you for your answers @charles 😄

I don't have a good answer for you on that one yet. We are still figuring that out. If I'm understanding correctly the thing that is important to you is making sure that you can run your DBT jobs after several EL pipelines have completed. Is that right?

As Meltano is fully customizable I can create my own DAG to have a different behavior, but then I start to migrate outside Meltano's UI and going to Airflow UI.

And how do you feel about jumping between those two UIs? is that a concern or is it fine to do so?

Hi @Chris (deprecated profile), sorry I didn't see your question. I decided to use Dataform as well (google acquired and released it for free), so I'm ok using Airbyte and Dataform UIs, but I decide not to use Airflow. Airflow could be useful for some DS pipelines but then I can try to use something else like MLflow.