Also, where does airbyte store the sources/connect...
# contributing-to-airbytes
Samuel Gordalina
03/04/2021, 10:21 PMAlso, where does airbyte store the sources/connection data? I looked into the postgres database and couldn't find it
u
user
03/04/2021, 10:21 PMthey are in the docker volume
u
user
03/04/2021, 10:21 PMwe store these configurations on disk
u
user
03/04/2021, 10:22 PMI think chis wrote about it. Let me find the tutorial
u
user
03/04/2021, 10:23 PM Copy code
docker run -it --rm --volume airbyte_data:/data busyboxu
user
03/04/2021, 10:24 PMu
user
03/04/2021, 10:24 PMthis one is about the workspace but the logic is the same
u
user
03/04/2021, 10:24 PMif you mount airbyte_data, then you can look at the configs and export them
u
user
03/04/2021, 10:24 PMalso one thing
u
user
03/04/2021, 10:24 PMin the UI you can actually export everything about airbyte
u
user
03/04/2021, 10:27 PMgot it, i understand now
u
user
03/04/2021, 10:28 PMso what i'm missing is the volumes when running gradlew
u
user
03/04/2021, 10:29 PMit wont launch because there's no data for it to load, in this case, the default workspace
u
user
03/04/2021, 10:30 PMohhh I get it!
u
user
03/04/2021, 10:30 PMhum, let me think
u
user
03/04/2021, 10:30 PMi'm running gradlew in my host OS, not the container - is that what's expected when developing locally?
u
user
03/04/2021, 10:31 PMIt was at some point. I wonder if we just killed it at some point
u
user
03/04/2021, 10:31 PM@Jared Rhizor (Airbyte) I think when we removed the dev dockercompose, we broke that right?
u
user
03/04/2021, 10:32 PMit’s still looking at a non-docker location, right?
u
user
03/04/2021, 10:32 PMyes
u
user
03/04/2021, 10:32 PMto fix we’d need to have the initialization run
u
user
03/04/2021, 10:34 PM@Samuel Gordalina FYI, we used to not use volumes in dev mode. but now we do. That is why it is breaking
u
user
03/04/2021, 10:34 PMthat's fine
u
user
03/04/2021, 10:34 PMdo you mount the volumes before running the gradlew locally?
u
user
03/04/2021, 10:34 PMOne thing you can do right now is
u
user
03/04/2021, 10:35 PM• copy the content of the volume config directory locally
• in the `dev.env`: modify CONFIG_ROOT to where you copied the config
• Re-run the server with gradle
u
user
03/04/2021, 10:37 PMOr
u
user
03/04/2021, 10:38 PMyou can just re-build the image everytime you want to test and stop/start the container (docker-compose stop server && docker-compose start server or smth like that)
u
user
03/04/2021, 10:39 PMok - testing now
u
user
03/04/2021, 10:42 PMdoes gradlew read the
.envu
user
03/04/2021, 10:42 PMno
u
user
03/04/2021, 10:42 PMthe .env.dev
u
user
03/04/2021, 10:42 PMbut I think you should change it to be .env
u
user
03/04/2021, 10:42 PMDamn my bad we broke that piece!
u
user
03/04/2021, 10:43 PMno worries about it
u
user
03/04/2021, 10:48 PMi'm still having the same issue
u
user
03/04/2021, 10:48 PMi think somehow the server doesn't read CONFIG_ROOT?
u
user
03/04/2021, 10:48 PMthe error says null in config root
u
user
03/04/2021, 10:48 PM2021-03-04 224755 INFO i.a.s.ServerApp(main):171 - {workspace_app_root=null/workspace/server/logs} - configRoot = null/data
u
user
03/04/2021, 10:49 PMlooks like it’s looking for
DEV_ROOTCONFIG_ROOTu
user
03/04/2021, 10:49 PMso actually can you do the following:
Copy code
DEV_ROOT=/tmp/sam
mkdir -p $DEV_ROOT/workspace
mv path_to_data $DEV_ROOT
# add DEV_ROOT=/tmp/sam into env.devu
user
03/04/2021, 10:54 PMits working
u
user
03/04/2021, 10:55 PMit complained about
AIRBYTE_VERSIONVERSIONu
user
03/04/2021, 10:55 PMthank you 🙏
u
user
03/04/2021, 10:55 PM Copy code
# setup
DEV_ROOT=/tmp/sam
rm -fr $DEV_ROOT
mkdir -p $DEV_ROOT/workspace
docker cp airbyte-server:/data $DEV_ROOT
# run
AIRBYTE_VERSION=dev DEV_ROOT=/tmp/sam ./gradlew :airbyte-server:runu
user
03/04/2021, 10:56 PMwonderful!!!
6 Views