Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

Hi!
I'm new to Airbyte - I took a quick look if it will suit my use case. It seems not?
Details in the thread.

Airbyte seems to be for "full DB replication", right?
I have client mssql/mysql databases with tables in this format:
```DateTime,Tag,Value
2022-01-01 00:00:00,TagA,10
...
2022-01-01 00:00:00,TagZ,10```
I do not want to replicate the full table - it has way too much data.
I only want to replicate data for certain Tags (I have a list of them).

This does not really seem to be a good fit for Airbyte, right?
Currently we run scheduled SQL queries to extract, orchestrated via Prefect. This seems to be the way?

I don’t think this is supported out of the box for the MSSQL and MySQL source connector but I guess you could write your own source connector which does the additional filtering/run the SQL query the way you want. Should not be very difficult to do with the CDK but you will of course have to write &amp; maintain it yourself.

I guess at that point the question would be what advantage do you get over using Prefect + SQL query. Main one I could think of is if you are looking to use Airbyte for other EL(T) parts and you would have all pipelines in one platform.

Hey <@U03D87PLR6K>, following ELT principles Airbyte intent is to replicate all the source data as raw data to a destination (*EL)* and assumes that the filtering is a transformation (*T)* that happens downstream. However there are workaround for your use case:
• Create a view on your source table which only has the set of tags you want to replicate
• Implement a <https://docs.airbyte.com/operator-guides/transformation-and-normalization/transformations-with-airbyte/|custom transformation with DBT,> this will still replicate all the tags in a *raw* table but you can filter out the records you want to exclude from the final table.

Thanks for your insights
I'll investigate the options