Are there any docs available for running the connectors directly rather than via the web panel Airbyte comes with?
✅ 1
c
charles
01/27/2021, 5:10 PM
can you talk a bit more about what you're trying to achieve?
charles
01/27/2021, 5:10 PM
some non UI things are documented but really only a sliver of it.
charles
01/27/2021, 5:10 PM
i don't want to send you on a wild goose chase.
h
Hassan // hotglue
01/27/2021, 5:11 PM
Yeah, I’m basically trying to see if I can use Airbyte over Singer for my product. RIght now we’re running Singer taps in docker containers to sync data, and then we run some Python transformation scripts on the data that comes out. I caught one of Airbyte’s articles about how Singer’s taps tend to be really non-standardized so I was looking at the connectors that you guys have written as an alternative
yeah. so each connector has a read me which describes how it can run as a standalone docker container.
charles
01/27/2021, 5:17 PM
most of our testing is assuming that source and destination are connected by the airbyte scheduler / worker. so if you try to just pipe an airbyte source into an airbyte destination, i am not entirely sure what will happen. it is something that we intend to support but has not really been our focus right now.
👍 1
h
Hassan // hotglue
01/27/2021, 5:24 PM
Yeah seems like the sources rely on your scheduler to put stuff in the right spot “Note: The system running the container will handle mounting the appropriate paths so that the config files are available to the container. This code snippet does not include that logic.”
c
charles
01/27/2021, 5:26 PM
so that's just referring to making sure that all configuration files are mounted in the docker container. you would need to do that part no matter what. that's just cost of doing business with docker.
h
Hassan // hotglue
01/27/2021, 5:26 PM
ah okay makes sense. I’ll play around with it locally and see how it works then