Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

Hi Team,

How can I scale my self hosted Airbyte instance?

Hi, please read <https://docs.airbyte.io/operator-guides/scaling-airbyte>

<@U01HYQYV1BQ> Can we use common ( shared ) volume for multiple Airbyte instance?

I am trying to develop a micro-SaaS using Airbyte.

For each user we will create separate workspace ( Airbyte workspace ). So we can not run 1000 user's request on single Airbyte instance. Right?

So to manage the load we want to horizontally scale the system. ( Mutltiple Airbyte Instances ). In this case can we setup Common volume ( data store ) which will be used by all Airbyte instances to store all workspace &amp; user config data?

as the above document states, the workers and the core airbyte systems are disjoint

if you are running on Kube, and scale the workers, it should scale

to answer your original question - in theory that's possible. we haven't done it in practice, so I'm not sure it'll be 100% stable

one thing - we've deprecated all of the shared volumes in our latest version. so the shared piece here would be the database. the thing to watch out for when sharing the database is the migrations the server runs whenever Airbyte is upgraded.

today the server runs migrations when it starts up. if you do decide to go with multiple airbyte instances operating on the same database, you'll have to make sure there is only one server trying to run the migration on updates, otherwise the db can get into a weird state

my recommendation is to start with one, and see how far scaling it up will take ya before trying to do multiple instances :slightly_smiling_face:

&gt; you’ll have to make sure there is only one server trying to run the migration on updates, otherwise the db can get into a weird state
It actually should be safer than that. The migration is done in a transaction, and the migrations are tracked in a table.
If one server has run the migration, it is safe for the other servers to try to run the migration. The migration process is idempotent.
The only thing to watch out for is that as long as one server has been upgraded, other servers should be upgrade to the same version as quickly as possible, so that they are all assuming the same database schema.