Hello everyone,
We are seriously considering swit...
# ask-community-for-troubleshootingc
Calvin Lam
03/10/2022, 8:13 AMHello everyone,
We are seriously considering switching to Airbyte after running into issues with the script we built ourself to move data from MongoDB to Bigquery for reporting. The main question I have before I can commit to using Airbyte is, does it support multidocument transactions in Mongo? We used to use Fivetran until we ran into this issue (which is why we had to build our own script).
👀 1
a
Augustin Lafanechere (Airbyte)
03/10/2022, 2:05 PMHi @Calvin Lam do you mean you'd like to know if our MongoConnector uses multi-documents transactions to perform atomic reads?
Augustin Lafanechere (Airbyte)
03/10/2022, 2:12 PMIf I understand your question correctly and my interpretation of the
readread Copy code
public Stream<JsonNode> read(final String collectionName, final List<String> columnNames, final Optional<Bson> filter) {
try {
final MongoCollection<Document> collection = database.getCollection(collectionName);
final MongoCursor<Document> cursor = collection
.find(filter.orElse(new BsonDocument()))
.batchSize(BATCH_SIZE)
.cursor();
return getStream(cursor, (document) -> MongoUtils.toJsonNode(document, columnNames))
.onClose(() -> {
try {
cursor.close();
} catch (final Exception e) {
throw new RuntimeException();
}
});
} catch (final Exception e) {
LOGGER.error("Exception attempting to read data from collection: ", collectionName, e.getMessage());
throw new RuntimeException(e);
}Augustin Lafanechere (Airbyte)
03/10/2022, 2:13 PMI think you might have an interesting use case that Airbyte does not yet cover. Could you please fill an issue on our repo sharing your context for a need of multi-document read?
c
Calvin Lam
03/11/2022, 9:41 AMHi Augustin, I believe your understanding is correct. I am not that familiar with MongoDB, I am just trying to pull data out of it, but essentially when multiple documents in MongoDB are updated in the same transaction, it causes issues in the way people typically connect MongoDB to an SQL based database.
Thanks for your help Augustin,
I just filled an issue here: https://github.com/airbytehq/airbyte/issues/11063
m
Marcos Marx (Airbyte)
03/11/2022, 7:46 PMI’d answered your issue @Calvin Lam hope my comments help you
2 Views