Anyone here tired to install pyairbyte in docker, ...
# connector-contributions
i
Anyone here tired to install pyairbyte in docker, we facing alot of issue cz it download docker for the source and docker in docker makes a lot of issues. Someone have experience and can advice
a
Hi, @Idan Moradov. The database sources and destinations are generally written in Java, which is why they default to being run via docker when invoked from PyAirbyte. The API sources (a few hundred in total) are generally written using our no-code (yaml-based) Builder and/or our Python CDK - and those are all runnable directly without needing docker.
In theory, one could pre-install Java in your docker container and then let PyAirbyte invoke those via command line or a shell script wrapper, but it's not trivial to create a wrapper like this and I haven't seen any examples of this from other community members as of yet.
i
Thanks for you help Do you know where can I found pyairbyte docker ready to use?
t
Hi @Idan Moradov I recently ran into a similar issue when trying to get Bigquery to run in my docker setup. Here are the things I learned when trying that will hopefully help you. 1. Make sure you attach the host docker.sock when running the initial docker container. a.
--volume /var/run/docker.sock:/var/run/docker.sock
2. Make sure to give your docker containers user access to the docker.sock a. You can find the user or group needed by running
ls -l /var/run/docker.sock
b. You can give the docker user permission with
RUN usermod -aG systemd-network app
(In my case
systemd-network
was the group needed and
app
was the docker user. This was set in my
Dockerfile
. Yours may be different) 3. Make sure to set the environment variables
AIRBYTE_TEMP_DIR
and
AIRBYTE_LOGGING_ROOT
to directories that live on the original host and in your docker container. This is needed so that the docker image ran by pyaribyte has access to the directories on the host machine.