https://linen.dev logo
Channels
Powered by
#ask-community-for-troubleshooting
Title
# ask-community-for-troubleshooting
c

Calvin Lam

03/10/2022, 8:13 AM
Hello everyone, We are seriously considering switching to Airbyte after running into issues with the script we built ourself to move data from MongoDB to Bigquery for reporting. The main question I have before I can commit to using Airbyte is, does it support multidocument transactions in Mongo? We used to use Fivetran until we ran into this issue (which is why we had to build our own script).
👀 1
a

Augustin Lafanechere (Airbyte)

03/10/2022, 2:05 PM
Hi @Calvin Lam do you mean you'd like to know if our MongoConnector uses multi-documents transactions to perform atomic reads?
If I understand your question correctly and my interpretation of the 
read
method is correct: I'm afraid that my answer is "no, our MongoDB connector does not support multi-document transactions". Here's the 
read
implementation :
Copy code
public Stream<JsonNode> read(final String collectionName, final List<String> columnNames, final Optional<Bson> filter) {
    try {
      final MongoCollection<Document> collection = database.getCollection(collectionName);
      final MongoCursor<Document> cursor = collection
          .find(filter.orElse(new BsonDocument()))
          .batchSize(BATCH_SIZE)
          .cursor();

      return getStream(cursor, (document) -> MongoUtils.toJsonNode(document, columnNames))
          .onClose(() -> {
            try {
              cursor.close();
            } catch (final Exception e) {
              throw new RuntimeException();
            }
          });

    } catch (final Exception e) {
      LOGGER.error("Exception attempting to read data from collection: ", collectionName, e.getMessage());
      throw new RuntimeException(e);
    }
I think you might have an interesting use case that Airbyte does not yet cover. Could you please fill an issue on our repo sharing your context for a need of multi-document read?
c

Calvin Lam

03/11/2022, 9:41 AM
Hi Augustin, I believe your understanding is correct. I am not that familiar with MongoDB, I am just trying to pull data out of it, but essentially when multiple documents in MongoDB are updated in the same transaction, it causes issues in the way people typically connect MongoDB to an SQL based database. Thanks for your help Augustin, I just filled an issue here: https://github.com/airbytehq/airbyte/issues/11063
m

Marcos Marx (Airbyte)

03/11/2022, 7:46 PM
I’d answered your issue @Calvin Lam hope my comments help you
2 Views
Powered by