https://linen.dev logo
Join Slack
Powered by
Since the "dev" and "contributing to" channels hav...
# ask-community-for-troubleshooting
i
Since the "dev" and "contributing to" channels have been archived, I'm here 🙂 I put quite a bit of time investing in Meltano, getting frustrated by poorly maintained sources before trying out Airbyte and just love it. And now I've encountered my first wrinkle - nothing big though, assuming I can jump through all the hoops of contributing to the code. I'm looking to add a simple feature to the MongoDb source connector (skipping SSL certificate validation). More details in the thread for those who are interested.
Our MongoDB connections are secured with SSL, but the host certificate isn't a "trusted" one, i.e. it's not found in the local java KeyStore. This isn't an issue for us, and it's behavior that's also supported by the "official" MongoDB Compass GUI.
So, what I'm looking to do is add a boolean value to the replication step, something like "Allow Invalid Certificates", which would default to False, but could be set to True in my case.
So the first question is: is everything I need to touch in airbyte-integrations/connectors/source-mongodb-v2? The bits that control the UI etc? found it. So yeah, looks like it.
Is the easiest way to test this • modifying the code • building and running the whole airbyte instance locally • testing that it runs fine ?
No... I guess since I'm only interested in contributing to a connector, I shouldn't waste CPU time building the core and all the other connectors?
c
No... I guess since I’m only interested in contributing to a connector, I shouldn’t waste CPU time building the core and all the other connectors?
This sounds like a good idea :)
i
yeah.. 🙂 I'm reading through here, trying to get the hang of everything. I'm sure I'll eventually arrive at what I need to do, but any quick pointers would help 🙂
c
modifying the code
of the connector only
building and running the whole airbyte instance locally
• No, you just need to use airbyte instance (from local deployment) and switch to your connector version or a custom connector instead to test that it runs fine
i
That makes total sense. I think I found the documentation for this bit too now, under the "Build a connector" section. Thanks!
👍 1
So have I understood this right: • modify the bits I need in 
source-mongodb-v2/src/
namely 
resources/spec.json
and 
main/java/io.airbyte.integrations.source.mongodb/MongoDBSource.java
• and I should be able to build running
Copy code
docker build . -t airbyte/source-mongodb-v2:dev
inside the 
source-mongodb-v2
folder where the 
Dockerfile
is? Is there something I'm missing, since I'm getting an error
Copy code
[+] Building 1.0s (3/3) FINISHED
 => [internal] load build definition from Dockerfile                                                                        0.0s
 => => transferring dockerfile: 32B                                                                                         0.0s
 => [internal] load .dockerignore                                                                                           0.0s
 => => transferring context: 34B                                                                                            0.0s
 => ERROR [internal] load metadata for <http://docker.io/airbyte/integration-base-java:dev|docker.io/airbyte/integration-base-java:dev>                                          0.9s
------
 > [internal] load metadata for <http://docker.io/airbyte/integration-base-java:dev|docker.io/airbyte/integration-base-java:dev>:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Ah... I need to be logged into docker right?
No.. still no.
c
do you have 
docker images airbyte/integration-base-java
?
i
since you ask, I'm pretty sure I don't 🙂 So I pull that in first then
or wait, do I need to build that first locally?
c
it doesn’t seem to be published on our dockerhub, so maybe you need to build it locally
it should be something like 
SUB_BUILD=CONNECTORS_BASE ./gradlew build
i
I'll give that a go.
c
but the source you are working on is in python? seems to be java, so it makes sense… just the image is not published
i
No, actually, just realized I do need to build that one too. 🙂 It's Java.
c
to build your connector, you can do
Copy code
./gradlew :airbyte-integrations:connectors:source-mongodb-v2:build
it should in theory build everything it needs as dependency
i
okay I'll tackle that first. Getting an UnsupportedClassVersionError so I think I need to update my java runtime. just a bit. I don't develop Java so all this general dev stuff around it is not familiar to me either.
a
Hey @Ilkka Peltola, thank you considering contributing to our MongoDB connector. If you need additional assistance from our team, do you mind posting on our forum in the connector development section?
i
thanks @Augustin Lafanechere (Airbyte) I will.
so okay, once I have successfully been able to build stuff I need, including the connector, how do I run that version locally?
a
@Ilkka Peltola could you please ask your Question on the forum, it'll be helpful for others too.
i
6 Views
Open in Slack
PreviousNext
Powered by