Damien Boué
09/11/2024, 3:02 PMdestination-mongodb . I made a little improvement in the connector that fail for our use case. This improvement is related to https://github.com/airbytehq/airbyte/issues/45118 and https://github.com/airbytehq/airbyte/issues/45119 . The connector where inserting in the destination line by line. I just made simple batch insert in the tmp table and a merge in the destination table. The performance has increased of x100 in the first step (inserting in tmp) and the second step (copying in the final table) now finish fast without causing a java head of memory error. I test it in our environnement and all is fine.
I want to create a pull request for this improvement, but i am struggling with the tests that never pass, but the docker image is correctly build. The connector feel pretty old comparing to other connectors and i am not sure if it's a "normal" test failure or me that juste don't arrive to launch them. I am simply launching ./gradlew :airbyte-integrations:connectors:destination-mongodb:build and i get the following fail result :
102 tests completed, 77 failed, 1 skipped
> Task :airbyte-integrations:connectors:destination-mongodb:integrationTestJava FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':airbyte-integrations:connectors:destination-mongodb:integrationTestJava'.
> There were failing tests. See the report at: file:///Users/damienboue/workspaces/airbyte/airbyte-integrations/connectors/destination-mongodb/build/reports/tests/integrationTestJava/index.html
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.6/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 11m 41s
29 actionable tasks: 4 executed, 25 up-to-date
Due to
2024-09-11 14:56:02 destination > INFO c.m.i.d.l.SLF4JLogger(info):76 Exception in monitor thread while connecting to server localhost:51822 com.mongodb.MongoSocketOpenException: Exception opening socket
at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:73) ~[mongodb-driver-core-4.10.2.jar:?]
at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:209) ~[mongodb-driver-core-4.10.2.jar:?]
at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.lookupServerDescription(DefaultServerMonitor.java:199) ~[mongodb-driver-core-4.10.2.jar:?]
at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:159) ~[mongodb-driver-core-4.10.2.jar:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682) ~[?:?]
at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:542) ~[?:?]
at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:592) ~[?:?]
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]
at java.base/java.net.Socket.connect(Socket.java:751) ~[?:?]
at com.mongodb.internal.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:107) ~[mongodb-driver-core-4.10.2.jar:?]
at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:82) ~[mongodb-driver-core-4.10.2.jar:?]
at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:68) ~[mongodb-driver-core-4.10.2.jar:?]
... 4 more[DEPRECATED] Marcos Marx
Evan Tahler (Airbyte)
09/11/2024, 3:50 PMEvan Tahler (Airbyte)
09/11/2024, 3:51 PMmonitor thread while connecting to server localhost:51822 Can you confirm that mongo is up and running on that port? If you are running from docker on Mac (I think you are), try host.docker.internal rather than localhost