had a quick question! (more out of curiosity) any reason why airbyte decided to go with the “deploy using docker” model rather than “deploy on your VPC” model?

@Ju Lee In general we really like Docker because encapsulates the environment needed to run an application really well. This is very user friendly as it means a user only needs to setup docker not specific dependencies like the implementation languages and tooling etc.. This is especially helpful when running connectors because each connector has different requirements. Some may be implemented in e.g python or ruby or go, and even within the same language there are differences in dependencies or the required language version. Managing all this without using an encapsulation layer like docker becomes pretty tricky

@Ju Lee but ultimately we can say that if you spin up a instance in your VPC and run airbyte on this instance it is a "deploy on your VPC" model. WDYT?

Presumably something like how Databricks manages clusters by handling things in a private VPC in your AWS? I think for a tool like Airbyte, I like the "smallness" of it. It doesn't spread into everything and hook into a bunch of things. I like that as a user.

^ yup it’s what Roshan mentioned. I’ve spoken with a couple of the people at posthog (open source analytics) and it seems they are moving from docker to vpc after moving to clickhouse as their data store. They came across some upgrade issues I believe when using the docker method here

We know posthog well we did YC together 🙂

I do really like the container approach. I am not a java programmer but if I want to create a source connector with Python or Golang or Rust then there is nothing preventing that. Making it scale with K8s via GCP GKE or AWS EKS would be a good way to standardize the deployment model. Running both the source & destination connector containers on the same K8s Pod would be a really good way to scale up and down and keep the pipes co-located.