https://linen.dev logo
#ask-community-for-troubleshooting
Title
# ask-community-for-troubleshooting
s

Stéphan Taljaard

05/04/2022, 8:26 AM
Hi! I'm new to Airbyte - I took a quick look if it will suit my use case. It seems not? Details in the thread.
1
Airbyte seems to be for "full DB replication", right? I have client mssql/mysql databases with tables in this format:
Copy code
DateTime,Tag,Value
2022-01-01 00:00:00,TagA,10
...
2022-01-01 00:00:00,TagZ,10
I do not want to replicate the full table - it has way too much data. I only want to replicate data for certain Tags (I have a list of them). This does not really seem to be a good fit for Airbyte, right? Currently we run scheduled SQL queries to extract, orchestrated via Prefect. This seems to be the way?
r

Robin

05/04/2022, 9:13 AM
I don’t think this is supported out of the box for the MSSQL and MySQL source connector but I guess you could write your own source connector which does the additional filtering/run the SQL query the way you want. Should not be very difficult to do with the CDK but you will of course have to write & maintain it yourself. I guess at that point the question would be what advantage do you get over using Prefect + SQL query. Main one I could think of is if you are looking to use Airbyte for other EL(T) parts and you would have all pipelines in one platform.
a

Augustin Lafanechere (Airbyte)

05/04/2022, 2:15 PM
Hey @Stéphan Taljaard, following ELT principles Airbyte intent is to replicate all the source data as raw data to a destination (EL) and assumes that the filtering is a transformation (T) that happens downstream. However there are workaround for your use case: • Create a view on your source table which only has the set of tags you want to replicate • Implement a custom transformation with DBT, this will still replicate all the tags in a raw table but you can filter out the records you want to exclude from the final table.
s

Stéphan Taljaard

05/04/2022, 7:09 PM
Thanks for your insights I'll investigate the options
4 Views