Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

Hi Guys!

I have a question, how is going the integrations with S3(Parquet) - Destination and Airflow?

I would like to check which topic it's interesting to start to contribute.

We will be working on the S3 destination in late january or early february.

When you say integration with Airflow, would you mind describing what specifically you're looking for here? We've found that people have a lot of different needs when they talk about Airbyte integrating with Airflow and we want to make sure we build the right thing.

Also, could you provide any context about the S3 Parquet integration? Some specific questions I have if you would be willing to share the context:
• How are you leveraging the Parquet file on S3? 
• Which source would you be using it with
• Would the parquet file need to be chunked/partitioned, or is a single destination file ok? 
• is Parquet a requirement, or would you be able to work with other file formats? 
• What file size would you write to S3? (ballpark of 10/100/1k/10k/100k MB is sufficient)

Thanks <@U01AS8LGX41> I think would be nice to have an Airbyte operator where I can trigger jobs from airflow. In some cases will be really interesting to have this option. Also if we can have more stats like (last execution, status ...)

<@U01A6TEQXRU>
I see some points about the S3 integration:
1. Options to partitioned the data based an specific column.
2. I think snappy as default compression would be nice
3. All databases as source will be interesting, mainly because the average volume (I'm thinking more in a datalake strategy)
4. Would be nice to have a columnar file format as Parquet or ORC, but text files as JSON and CSV also works.
5. For `full-refresh` single file or files splitted for each "X" MB will be nice. For `incremental` mode could be something similar.

With these option the files can consumed for tools like Presto(Trino), Athena, Redshift spectrum, Spark and others.