Sarath Manoj
07/17/2024, 1:17 PMkapa.ai
07/17/2024, 1:18 PMMICRONAUT_SERVER_MAX_REQUEST_SIZE
to a value higher than your catalog's size. If you're using helm charts to deploy, you can add the following: yaml - name: MICRONAUT_SERVER_MAX_REQUEST_SIZE value: "10601372" # or a higher value
However, please note that the Airbyte platform may not perform optimally with a very large catalog. Another approach is to specify the schema in the connector configuration or create a specific user with limited access to split tables. This can help reduce the size of the schema being fetched. Please refer to the discussion in this [Github issue](https://github.com/airbytehq/airbyte/issues/37463) for more details.Sarath Manoj
07/17/2024, 1:21 PMkapa.ai
07/17/2024, 1:21 PMkapa.ai
07/17/2024, 1:21 PMMICRONAUT_SERVER_MAX_REQUEST_SIZE
is an environment variable that you can set in your deployment configuration. If you're using Docker, you can add it to your .env
file: bash MICRONAUT_SERVER_MAX_REQUEST_SIZE=10601372
And then in your docker-compose.yaml
file, you can add it to the environment section of the appropriate service: yaml services: airbyte-server: environment: - MICRONAUT_SERVER_MAX_REQUEST_SIZE=${MICRONAUT_SERVER_MAX_REQUEST_SIZE}
If you're using Helm charts for Kubernetes deployment, you can add it to your values.yaml file or directly in your helm command with --set
flag. Please refer to this [Github issue](https://github.com/airbytehq/airbyte/issues/4564) for more details.Sarath Manoj
07/18/2024, 6:21 AMkapa.ai
07/18/2024, 6:21 AMkapa.ai
07/18/2024, 6:21 AMSarath Manoj
07/18/2024, 6:22 AMkapa.ai
07/18/2024, 6:22 AMkapa.ai
07/18/2024, 6:22 AMSarath Manoj
07/18/2024, 6:47 AMkapa.ai
07/18/2024, 6:47 AMkapa.ai
07/18/2024, 6:48 AMSarath Manoj
07/18/2024, 6:49 AMkapa.ai
07/18/2024, 6:49 AMkapa.ai
07/18/2024, 6:49 AMdocker-compose.yaml
file is located and running the following command: bash docker-compose down
2. Remove Airbyte Docker images. This will ensure that all old versions of Airbyte are removed from your system. You can do this with the following command: bash docker rmi $(docker images -q airbyte/*)
3. Remove the Airbyte volumes. This will delete all data associated with your Airbyte instance. Be careful with this step as it will delete all your data and cannot be undone. You can do this with the following command: bash docker volume rm $(docker volume ls -q | grep airbyte)
4. Install a specific version of Airbyte. You can do this by first cloning the Airbyte repository and then checking out the specific version you want to install. For example, if you want to install version 0.30.0-alpha
, you can do this with the following commands: bash git clone <https://github.com/airbytehq/airbyte.git> cd airbyte git checkout v0.30.0-alpha
5. Start Airbyte. You can now start your specific version of Airbyte with the following command: bash docker-compose up
Please note that these steps will completely remove your old Airbyte instance, including all data, and replace it with a new instance of a specific version. Make sure to backup any important data before proceeding. Sources: - [Github Issue 30287](https://github.com/airbytehq/airbyte/issues/30287) - [Airbyte Documentation](https://docs.airbyte.com/operator-guides/upgrading-airbyte#resetting-your-configuration)Sarath Manoj
07/18/2024, 6:53 AMkapa.ai
07/18/2024, 6:53 AMkapa.ai
07/18/2024, 6:56 AMbash ssh <admin username>@<IP address>
2. Create and navigate to a new directory. This is where you'll download Airbyte. bash mkdir airbyte cd airbyte
3. Download the Airbyte install script. You can do this using the wget
command. However, the script you download from the master branch will always fetch the latest version of Airbyte. To get a specific version, you'll need to modify the script. bash wget <https://raw.githubusercontent.com/airbytehq/airbyte/master/run-ab-platform.sh>
4. Modify the script to fetch a specific version. Open the run-ab-platform.sh
file in a text editor and change the VERSION
variable to the version you want. For example, if you want version 0.30.0-alpha
, change the line to VERSION=0.30.0-alpha
. 5. Make the script executable and run it. This will download and start Airbyte. bash chmod +x run-ab-platform.sh ./run-ab-platform.sh -b
Please note that these steps assume you have Docker and Docker Compose installed on your VM. If not, you'll need to install those first. Sources: - [Deploy Airbyte on Azure (Cloud Shell)](https://docs.airbyte.com/deploying-airbyte/on-azure-vm-cloud-shell#install-and-start-airbyte) - [Deploy Airbyte on Oracle Cloud](https://docs.airbyte.com/deploying-airbyte/on-oci-vm#install-and-start-airbyte) - [Deploy Airbyte on GCP (Compute Engine)](https://docs.airbyte.com/deploying-airbyte/on-gcp-compute-engine#install-and-launch-airbyte) - [Deploy Airbyte on AWS (Amazon EC2)](https://docs.airbyte.com/deploying-airbyte/on-aws-ec2#install-and-start-airbyte) - [Deploy Airbyte on DigitalOcean](https://docs.airbyte.com/deploying-airbyte/on-digitalocean-droplet#install-airbyte)Sarath Manoj
07/18/2024, 7:54 AMkapa.ai
07/18/2024, 7:54 AMkapa.ai
07/18/2024, 7:55 AM_AIRBYTE_TMP
. Here is a simple SQL command to delete these tables: sql DO $$ DECLARE table_name text; BEGIN FOR table_name IN (SELECT table_name FROM information_schema.tables WHERE table_name LIKE '_AIRBYTE_TMP%') LOOP EXECUTE 'DROP TABLE IF EXISTS ' || table_name; END LOOP; END $$;
Please be careful while running this command as it will delete all tables starting with _AIRBYTE_TMP
. Also, note that according to a [forum post](https://discuss.airbyte.io/t/1388), it's safe to delete these tables and it won't cause any issues. However, it's recommended to investigate why the sync failed in the first place, as repeated failures could lead to an accumulation of these temporary tables. You can check the logs for more details on the failure. Airbyte has acknowledged this issue and plans to fix it as per this [Github issue](https://github.com/airbytehq/airbyte/issues/4059).Sarath Manoj
07/18/2024, 7:55 AMkapa.ai
07/18/2024, 7:55 AMkapa.ai
07/18/2024, 7:56 AMbash docker volume ls
The Airbyte volumes usually have names starting with airbyte_
. The volume for the Airbyte database is typically named airbyte_db_data
. To inspect the details of this volume, including where it is stored on your VM, you can use the docker volume inspect
command: bash docker volume inspect airbyte_db_data
This will return a JSON object with details about the volume. The Mountpoint
field in this JSON object indicates the location on your VM where the volume's data is stored. Please note that directly accessing or modifying the data in this volume is not recommended as it can lead to data loss or corruption. Always use the Airbyte interface or API to interact with your Airbyte data. Sources: - [Configuring the Airbyte Database](https://docs.airbyte.com/operator-guides/configuring-airbyte-db) - [Accessing the default database located in docker airbyte-db](https://docs.airbyte.com/operator-guides/configuring-airbyte-db#accessing-the-default-database-located-in-docker-airbyte-db)