https://linen.dev logo
h

Huib

02/23/2022, 1:02 PM
Hi there, I have. A few question around MongoDB connector. So I’m trying to use MongoDB connector to integrate with Amazon DocumentDB (it supports MongoDB compatibility), however Amazon only allows connection from within the same VPC to do that over TLS.  To accomplish this, I set up Airbyte dev instance on the same VPC but the connection still wasn’t going though.  After some research I found out that in order to connect, the default connection string needs a bit more information in addition to
tls=true
and the full connection string needs to have this format:
mongodb://{username}:{password}@{host}:{port}/{database}?tls=true&tlsAllowInvalidHostnames=true&tlsAllowInvalidCertificates=true
I’ve tried to test my theory by creating JDBC source, but in the end I could dockerize the container. Seems like the dependency on
airbyte/integration-base-java:dev
image is not permitted. So I guess question/request, is this possible to add to the existing MongoDB connector; and if yes, what would be the expected time frame for this? It would also be great to add ability to use SSH with this connecotr.
y

yevgen

02/23/2022, 1:56 PM
Yeah, I think some additional flexibility for SSL/TLS/Certificate verification behavior would help solve this issue. I filed this as a catch-all: https://github.com/airbytehq/airbyte/issues/9871 But it would probably be faster if you filed a MongoDB specific issue for this connector (and if you're willing, put up a PR!) to make this supported. Also linking the doc you found referring to Amazon only allowing the same VPC to connect would be nice
a

Aaron Williams

02/23/2022, 1:58 PM
I ran into the same issue and created https://github.com/airbytehq/airbyte/issues/10388 a week ago. I am working on a solution. I have one option working locally and successfully connected to DocumentDB
I was just checking on your github issue 😄
d

David Hadaller

02/23/2022, 3:12 PM
@Aaron Williams thanks for info, I feel like this is exactly what the issue with DocDb connection is. How have you managed to deploy this locally meanwhile. Whenever I try to build mongo docker image i receive
pull access denied error
or when trying to build with gradle i have an issue with building JVM (Sorry quite a noob when it comes to Java)
Hi @yevgen, you can follow this guide to have a working local development environment. If you tweaked the mongodb connector you need to build it again:
Copy code
./gradlew :airbyte-integrations:connectors:source-mongodb-v2:build
And rebuild the docker image:
Copy code
docker build -f ./airbyte-integrations/connectors/source-mongodb-v2/Dockerfile -t airbyte/source-mongodb-v2:dev
Then you need to set the version to
dev
in the settings page.
I mark this conversation as resolved because, as Aaron mentioned, an issue exists on our repo for your problem and our team will look into it asap.