<@U04S8PLJWMT> can you explain me how airbyte is w...
# ask-ai
s
@kapa.ai can you explain me how airbyte is working when i installed on a ec2 linux instance when its ingesting data from postgres to s3 whats happening underlying process
k
When you install Airbyte on an EC2 Linux instance (such as Amazon Linux) and use it to ingest data from Postgres to S3, here’s what happens in the underlying process: 1. Deployment & Setup - Airbyte is typically installed using Docker and the
abctl
tool. You set up Docker, add your user to the Docker group, and then use
abctl
to install Airbyte. By default, Airbyte listens on port 8000, and you can access its UI via your browser if you’ve configured the security group and host settings correctly [Using an EC2 Instance with abctl](https://docs.airbyte.com/platform/next/deploying-airbyte/abctl-ec2). 2. Connection Configuration - In the Airbyte UI, you configure a source (Postgres) and a destination (S3). You provide credentials and connection details for both. 3. Data Extraction (ELT: Extract) - Airbyte connects to your Postgres database and extracts data. The data is read as records (usually as JSON objects) from the source tables [Explore Airbyte's full refresh data synchronization](https://airbyte.com/tutorials/full-data-synchronization#step-1). 4. Data Loading (ELT: Load) - The extracted data is then loaded into the destination—in this case, S3. Airbyte serializes the data into the chosen file format (CSV, JSON, Parquet, or Avro) and writes it to your specified S3 bucket. The files are typically named and organized by schema, table, and timestamp [Export Postgres data to CSV, JSON, Parquet and Avro files in S3](https://airbyte.com/tutorials/postgres-to-csv-json-parquet-avro-in-s3). 5. File Structure in S3 - For example, if you export as CSV, the files in S3 will follow a pattern like:
bucket_name/path/schema/table_name/YYYY_MM_DD_Milliseconds_since_epoch.csv
- For Parquet or Avro, Airbyte handles the conversion and writes the files accordingly. 6. Transformation (Optional, ELT: Transform) - If you enable normalization, Airbyte uses dbt to transform the raw JSON data into a more structured format in the destination. However, for S3, the main transformation is the serialization into the chosen file format. 7. Underlying Process - Airbyte orchestrates the extraction and loading using Docker containers. Each sync job runs in its own container, ensuring isolation and scalability. - Temporary data may be stored during the sync process, but for S3 destinations, the final output is the files written to your S3 bucket. Summary: Airbyte, running on your EC2 instance, extracts data from Postgres, serializes it into your chosen format, and uploads it to S3. The process is managed via Docker containers, and you control everything through the Airbyte UI or API. The raw data is preserved in the destination, and optional normalization can be applied if needed. If you need more details about the exact file formats or want to see sample outputs, you can refer to the [Airbyte S3 destination documentation](https://docs.airbyte.com/integrations/destinations/s3/#parquet) and the [tutorials](https://airbyte.com/tutorials/postgres-to-csv-json-parquet-avro-in-s3).