I want to create a kind of mega Dockerfile that contains airbyte and friends to showcase a few tools to run with a data lake on mybinder.org. It's ideal if I keep the Docker image that runs on binder as small as possible to load faster. So I'm hoping to find a way to install Airbyte that wouldn't require installing Docker in Docker. Any options here?
04/21/2022, 12:14 AM
Also interested in this
04/22/2022, 6:02 PM
@Alexander Butler, just got a reply from @abhi in a DM. He thinks there's no way around the Docker dependency but he's checking in with the team. Abhi, feel free to answer me here so that Alex gets the final word as well!
Augustin Lafanechere (Airbyte)
04/25/2022, 8:34 AM
Hi @Brian Olsen, Airbyte scheduler must have access to the host docker socket (docker in docker) to schedule jobs.
04/25/2022, 2:15 PM
Thanks @Augustin Lafanechere (Airbyte), so then there's no way to install Airbyte without also installing Docker.
I know Docker is very ubiquitous in most environments and this is probably a more niche use case, but has there been any discussion about making a custom build/deployment an option? My use case is more around some getting started stuff, but I'm sure there are some companies that may need this level of customization in order to adopt Airbyte into their architecture.