How many GB of the data the airbyte support transfer or move from mssql to postgres. Tell me the a use case.

Airbyte can handle large data transfers between MSSQL and Postgres, but the performance and feasibility depend on various factors such as the size of the data, the configuration of the deployment, and the sync mode used. ### Use Case: Transferring 300GB from MSSQL to Postgres #### Scenario A user needs to transfer 300GB of data from MSSQL to Postgres using incremental synchronization. The table is rebuilt every 24 hours, which complicates the process. #### Challenges 1. Time Consumption: The estimated time for the transfer is around 100 hours. 2. Table Rebuild: The table is rebuilt every 24 hours, which can interrupt the sync process. 3. Cursor Setting: The user wants to set the cursor value to select only fresh data. #### Solution 1. Incremental Sync: Use incremental synchronization to transfer only the new or updated data. 2. Cursor Setting: Manually set the cursor value to select only fresh data. The user attempted to update the cursor value directly in the Airbyte database but faced issues with connections. Here is an example of how the user tried to set the cursor value:

sql UPDATE state SET state = jsonb_set(state, '{state, streams, 0, cursor}', '"2022-01-25T00:00:00.000000Z', false) WHERE connection_id = '592fd59f-2aa9-4852-9d2b-8d12e269faa1';

However, the user did not have a connection when running

select * from state

. 3. Alternative Approaches: - Octavia: The user tried using Octavia but did not find the cursor setting option. - CSV Upload: The user attempted to upload CSV to the database and then perform incremental synchronization on the view of the table with a date condition. However, Airbyte deleted all old data when the sync started. #### Recommendations - Interface Improvement: The user requested adding cursor setting to the Airbyte interface for easier management. - Resource Management: Ensure that the deployment has sufficient resources to handle large data transfers. For example, using a machine with higher vCPUs and memory can help speed up the process. For more details, you can refer to the original forum post [here](https://discuss.airbyte.io/t/3733). ### Additional Considerations - Primary and Foreign Keys: When migrating data, primary and foreign key relationships are not preserved by default. This can be a limitation if the integrity of the database schema is crucial. More information on this issue can be found [here](https://github.com/airbytehq/airbyte/issues/20876). By understanding these challenges and solutions, you can better plan and execute large data transfers using Airbyte.