Has anyone had any luck getting Airbyte to run on a Docker (non-docker-compose) environment? I'm interested in getting it running on Render, but it seems like it's non-trivial to do (as per https://github.com/airbytehq/airbyte/issues/1415)

Unfortunately this isn’t supported (and likely won’t be supported). Render (and similar platforms) only allow you to run a single container at a time. To support something like Render we would either have to package connectors and our core platform into a single image (which would lose a lot of our isolation guarantees that our architecture depends on) or they would have to allow containers to spawn other containers.

Thanks for the context @Jared Rhizor (Airbyte)! Render has their own docker-compose-like format for spinning up multiple containers with disk images. Given that Render is still relatively small I wouldn't expect you to formally support them, but I'd love to take a crack at making one myself. A few questions on your

docker-compose

that would help me go down that path: • Why is there the

init

instance /

create_mount_directories

script? It looks like

init

and

seed

are solely for seeding data. Do you imagine it'd be relatively simple to leave them out? • Is

airbyte/db

special in any way, or will any Postgres instance be fine? If I can get something working happy to host a guide and/or contribute it back. Big fan of what you're doing!

So the initial setup of the docker-compose-like components isn’t the problem. The problem is the way we launch jobs on docker directly from

airbyte-scheduler

. In

DockerProcessBuilderFactory

you can see we are directly interacting with Docker, and this class is used to handle any operations involving connectors. This is the piece that is incompatible with Render out of the box.