We are currently under a multi tenant architecture with a da | Office-hour

We are currently under a multi-tenant architecture with a database server per tenant, we have a lot of tenants with large amounts of data ( from 1.2 to 10T). We want to do a full load from all tenants to our data lake, all our sources are MySQL and I tested with Postgresql destination but as I mention here before -> https://airbytehq.slack.com/archives/C021JANJ6TY/p1665381596030749 , we have some performance issues and the sync process just hangs and we are not even testing with the largest tables. 1. Which destination do you recommend in this case, to use S3 ( I've tested this, and the process finishes but is still slow, 15hrs to move 67GB) or any other destination that can speed up the process? 2. Is airbyte meant to deal with this multi-tenancy approach with a lot of data or just for a small amount of data? 3. Do I need to configure airbyte in some way to increase performance, is there a basic configuration for this scenario?

Marcos Marx (Airbyte)

10/12/2022, 3:03 PM

Great question @Eduardo Aviles we’re going to discuss this in the office hour today!

View count: 1

Join thread in Slack