Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

Hello folks, I am from a startup <https://www.instill.tech|Instill AI> doing visual data preparation. We are keen to adopt Airbyte’s standard for the data source and destination of our open-source visual data pipeline. However, I just realised that Airbyte currently doesn’t support unstructured data at all (both source and destination). I am wondering what’s the most efficient way to make this happen? Is there any technical obstacle blocking you from handling unstructured data? or is unstructured data even on the road map? We are keen to contribute to this feature too. Thanks.

Can you give an example of data or connector your looking for?

For example, to read a bunch of image files in an S3 or GCS bucket.

Yep, our connectors doesn’t handle this type of data well. But something easy to achieve developing a custom connector to your application

Got it. I will take a deeper look at how to develop a custom connector for this, and come back to you if any questions. Thanks.